The San Jose Mercury News coverage (registration) and SJMN's weblog seems to be in the center of the surfacing dispute between free classifieds database Craigslist and classifieds search engine Oodle. The story seems to be that Craigslist asked Oodle in an undetermined communication, presumably with legal implications, that Oodle was pushing too hard to scrape links from its site for Oodle's cross-source classifieds index and search engine. But as of yet no specifics on any legal charges have been mentioned, only concerns that the posted terms of use have been violated. As Danny Sullivan on Search Engine Watch notes, though, "Looking over at Google, I see Craigslist seems to have no problem of Google indexing 12 million of its pages. Of course, there's a difference between indexing a page and scraping the content to be included in a more vertical service."

But is there really a difference? Oodle does in fact provide more intelligent indexing than a straight search engine would, using faceted navigation to peruse listings that can only be gained from databasing an index of full content. But at the end of the day such a function is no different from any other search engine that helps to contextualize links to full content. As such this is a potentially chilling development for any content provider deriving links and indexing to other content on the Web. But the apparently discriminatory manner in which Craigslist is enforcing its browse-wrap terms and conditions would seem to open up as many issues for plaintiffs in such cases as defendants.

There are many conflicting and hazy legal threads hanging over the crawling and mining of Web sites, but a lot of the issues seem to boil down to one key question: is building an index or other services derived from crawling open Web content an act that can be restricted on a discriminatory basis or not? Are links to "browse-wrap" legal terms found at the bottom of many Web pages hidden policies that can be changed and applied prejudicially by a Web site proprietor or are they public notices equivalent to "jackets required for gentlemen" signs posted in the lobbies of fancy but public restaurants, unmistakable to all who enter? To say that some can use access to public content for commercial purposes while others cannot would seem to run up against such basic issues of discrimination in public venues.

The twists and turns of legal precedents are highly unpredictable today but with meta-searching and federated searching services such as Oodle becoming increasingly common the law is likely to move towards supporting most uses of publicly available content without explicit, up-front usage contracts or clearly consensual notices, especially when usage falls short of full replication for commercial purposes or documentable abuse that restricts its use by others or its commercial exploitation by its owners. That's going to be a pretty tall bar for most content originators to cross before they can afford to complain about what has been made available to the public. In the Content 2.X era, working collaboratively with a user base that is increasingly a publishing audience is going to take us further away from hostile confrontations between content users and originators and towards an era in which all parties creating and using content benefit fairly and mutually from both its creation and its use.