According to Reuters and numerous other sources report on portal provider Yahoo!'s plans to provide access to "Deep Web" sources not previously crawled by their search engine. Non-commercial sites may be crawled by Yahoo! without paying a fee, with plans already out for audio made available via Northwestern University from National Public Radio, the U.S. Library of Congress, the New York Public Library and the U.S.Supreme Court to be made available in Yahoo! search results. But commercial sources of content can be expected to pay a fee for crawling their deep content, based on a proportional formula that includes database size and click-through rates. Many content sources stored in databases are not crawled by public Web search engines, so this represents a great potential gain of high-value content that could position Yahoo! more favorably for searchers who are seeking more valuable sources. The implication from some reports, though, is that while the Yahoo! search engine will not favor these results in terms of search engine ranking, other sources may not be visited as often if they don't cough up the "deep" crawling fee - shades of GoTo.com, progenitor of Overture, Yahoo!'s new contextual ad division. It's easy to imagine how this could degrade into a war for paid search engine ranking that many thought was put to the side with contextual ads. But in more likelihood the real form taking shape is the outlines of what a premium content aggregator may look like in the future, collecting listing and click-through fees from publishers as a part of a greater content technology service rather than directly databasing content, as mentioned in this week's
news analysis. It's far from clear that Yahoo!'s strategy will help them in the short run with Google, which already provides a wide range of deep Web content without charging its suppliers, but in the long run it may be part of the framework that begins to make sense for giving high-value and premium sources the context that they need in the search services most used by consumers and professionals.