1) The notion that the systematic clipping of one or two sentences from every news story you can find on the Internet for the purpose of selling ads is "fair use" is totally absurd.
2) The "robots.txt" argument misses the point because the publishers never gave permission to use their content, so the presence of an "opt-out" mechanism is irrelevant. The mechanism must be "opt-in", i.e., Google must ask to use their content.
3)If we believe that intellectual property has become an obsolete concept altogether, we should be prepared to accept all of the consequences of this, e.g., a site might pop up using a new TLD that accepts a search query from a client, passes it on to Google using a spoofed IPv6 address, and passes the result back to the client. We also shouldn't care about plagiarism, trademark infringement, etc., because the Internet has clearly made these concepts obsolete and laws related to them unenforceable.
1) The debate is centered on Google News. As far as I know, News is one of the few services where Google haven't, don't and won't show ads. "Systematic clipping of one or two sentences [...] for the purpose of providing an ad-free overview and linking to the respective stories" sounds like "fair use" to me.
2) They are publishing their material on the web, for the world to see. If you'd want an opt-in for every crawler out there, no search engine would ever be able to take off and index a significant portion of the web.
By the way, an opting out from all (obedient) crawlers is as simple as "Disallow: /"
3) Passing a query on to google through a spoofed IPv6 address doesn't make any sense. Apart from that, there's several services offering just that functionality. Try DuckDuckGo, for example. (I'm not quite sure which search engine they use nowadays, but afaik, the project started out using Google.)
>The debate is centered on Google News. As far as I know, News is one of the few services where Google haven't, don't and won't show ads. "Systematic clipping of one or two sentences [...] for the purpose of providing an ad-free overview and linking to the respective stories" sounds like "fair use" to me.
Then Google should have no issue with not linking to publishers who don't want to be linked to. As an aside, I disagree that it's fair use, because it's done systematically as part of a commercial enterprise.
>They are publishing their material on the web, for the world to see. If you'd want an opt-in for every crawler out there, no search engine would ever be able to take off and index a significant portion of the web. By the way, an opting out from all (obedient) crawlers is as simple as "Disallow: /"
If you believe that Google is infringing the publishers' copyrights, then it is not the responsibility of the publishers to opt out. See my response to your first point.
There's a difference between linking to something and using an except from a published work without permission. (Yes, it would be fair use in isolation, but not as part of an automated, systematic process that's part of a commercial enterprise.)
> 2) The "robots.txt" argument misses the point because the publishers never gave permission to use their content, so the presence of an "opt-out" mechanism is irrelevant. The mechanism must be "opt-in", i.e., Google must ask to use their content.
The crux of the matter is that you're missing the newspapers motive. They don't want Google to ask their permission, they want google to pay them, which is a huge difference. They DON'T want google to ask for their permission and then delist them from Google News if they had to pay for it.
I am on Google's side on this matter, this sounds like extortion. Google should be free to delist them if they want money from their link, in a capitalist society you should be free to chose who you deal with in your business. I live in France, our government wants to tax google to help the newspapers and they didn't take google's answer (that they'd delist the newspapers) well, because they don't want google to respect their IP and ask for permission, they want Google to pay up no matter what.
>The crux of the matter is that you're missing the newspapers motive. They don't want Google to ask their permission, they want google to pay them, which is a huge difference. They DON'T want google to ask for their permission and then delist them from Google News if they had to pay for it.
I haven't missed their motive. Of course they don't want to be delisted from Google News. In a sense, they're like patent trolls, but with more of a moral high ground. Unlike patent trolls, they have produced something and sent it out into the world. It is not their job to preempt what they rightfully view as copyright infringement, even if that infringement might benefit them.
>Google should be free to delist them if they want money from their link
1) Normal search at google takes one sentence, title of the webpage and links them together. To support that service, they have ads. If news take two sentences, how is that totally absurd compared to one, and why is search totally acceptable but news search not?
2) as other has said, public sites are indexed in an opt-out system. Opt-in would not work, and would destroy any form of search system, indexing, yellow pages, and so on.
3) This is not an intellectual property issue. Its a term of use issue if one want to be included in google search. One can not say yes to be included in search, but not if there are ads around. Either you want google to give you service, or you don't. I wish I could get a gemail free from spying, a google search thats anonymous, and free telephone calls without any recording/advertizements, but no. I can't get that from google, because their services to have a price tag.
> the publishers never gave permission to use their content
Yes they did, yes they did. The publisher explicitly granted access by putting up the news content in an public subdirectory of a public webserver. News items don't just magically `pop up' in the public subdirectory of a webserver. Usually at least two employees (writer, reviewer) have to log into, and explicitly perform several steps in an CMS to publish a news item.
The decision of publisher is encoded as HTTP server configuration (including CMS, if any). The server obviously provides access only to some resources, as the operator wishes -- the ones meant for public consumption, while other resources are password-protected, only accessible from certain network addresses, or not accessible over HTTP at all.
Just as much as an image resource can be, and often is, protected from deep-linking based on HTTP `Referer' header, a news item can be protected from being indexed by Google News bot and displayed as a snipped in Google News based on `User-Agent' header.
In more details:
A typical User Agent that HTTP GETs some resources from the server describes itself with a `User-Agent' header. Google is honest here, and openly indicates its user agent as a spider bot, with explicit purpose of indexing content for web search.
Even better, Google uses a /separate/ spider bot for Google News service, and it is entirely reasonable to serve the general websearch bot with content -- so the website itself remains indexed -- and deny the news bot access, to prevent Google News from displaying news snippets.
Upon HTTP request, the server reads through request, including resource URL, headers (including User-Agent and Cookies, if any), compares that info to configuration. If, and only if, it matches configuration for unrestricted access, the resource is served. Otherwise, access denied condition is indicated and optionally authentication requested.
Let's assume for a moment the law goes through as requested by the press. The most probable implementation would be, Google includes a particular header in HTTP requests -- indicating itself as a `news-bot-that-has-subscription', and the server would, in turn, compare the header with configuration and either reply with news contents, or perhaps indicate access denied, should subscription expire, or for some other reason. The mechanics is pretty much same as it would be today to restrict Google News bot from gathering, and Google News from displaying (some or all) news snippets.
In short, the publishers have already configured some resources as protected from /some/ User Agents (based on request URL, headers, cookies, network address or anything else), and it is entirely their own choice they serve the news to both Google bots (websearch and news).
That's a strawman; the bots never indicated intention to read. They indicated (via `User-Agent' header) intention to index content and got HTTP 200 OK response to that.
Nothing in the HTTP protocol specs (RFC 2616) even suggests it is only meant for direct human consumption. To the contrary, it states,
user agent
The client which initiates a request. These are often browsers,
editors, spiders (web-traversing robots), or other end user tools.
...and the Google bots clearly identify themselves as indexing bots.
Please don't be obtuse. The whole matter is not about Google doing something strange or different than any other search engine. It's just about collective bargaining of one business group against another business. Some german press companies stood up as a (self-appointed and thus somewhat suspect) representation of whole german press industry and are trying to get a bigger slice of the cake.
If I don't lock my house, that doesn't give you the right to break in. You're absolutely right that publishers could make more use of the tools at their disposal to protect their content in whatever way they wish, but they have rights regardless of whether they do.
This analogy only muddles the picture. We are talking about private property, but not a home, more like a shop or pub.
Its only raison d'etre is to serve certain guests. We are talking about a commercial server that is managed by a professional admin, and is explicitly open to only some of the public, and it already screens and filters guests, based on various criteria. For example, the server may disallow hotlinking of images, serve some content only upon receipt of a HTTP cookie confirming payment and serve different content for mobile devices, some movies blocked in certain countries, based on IP geolocation. The decision is based on HTTP request data and will of the owner, encoded as server's configuration. Ask yourself, how much different is Google News bot?
How about this analogy instead: ``if I hire a professional bouncer for my high-street shop, and the bouncer stops some people from entering and lets others in, that doesn't mean those let in can indeed walk in''?
Which would be crazy, and press would have a field day with such a shop.
Some access control is really easy to implement, other types are not, or they're not cost-effective. It's perfectly reasonable for a content provider to combine technological measures (like e.g. IP geolocation) with legal ones (terms of use, copyright).
That said, you're absolutely right: how hard can it be to block the Google News bot for a news organization? Not very. My point is merely that you can't draw any legal or moral conclusions from a 200 OK response, that's absurd.
To riff on your example, if my high-street shop has a sign that says "only people over 18" but doesn't have a bouncer, does that mean people under 18 should feel free to enter because nobody's physically stopping them?
"That's a strawman; the bots never indicated intention to read. They indicated (via `User-Agent' header) intention to index content and got HTTP 200 OK response to that."
The publishers aren't objecting to being indexed. They are objecting because snippets of their articles are being copied, placed on Google News, and used to sell ads without their permission, and they're not being compensated.
1) The notion that the systematic clipping of one or two sentences from every news story you can find on the Internet for the purpose of selling ads is "fair use" is totally absurd.
2) The "robots.txt" argument misses the point because the publishers never gave permission to use their content, so the presence of an "opt-out" mechanism is irrelevant. The mechanism must be "opt-in", i.e., Google must ask to use their content.
3)If we believe that intellectual property has become an obsolete concept altogether, we should be prepared to accept all of the consequences of this, e.g., a site might pop up using a new TLD that accepts a search query from a client, passes it on to Google using a spoofed IPv6 address, and passes the result back to the client. We also shouldn't care about plagiarism, trademark infringement, etc., because the Internet has clearly made these concepts obsolete and laws related to them unenforceable.