That's a strawman; the bots never indicated intention to read. They indicated (via `User-Agent' header) intention to index content and got HTTP 200 OK response to that.
Nothing in the HTTP protocol specs (RFC 2616) even suggests it is only meant for direct human consumption. To the contrary, it states,
user agent
The client which initiates a request. These are often browsers,
editors, spiders (web-traversing robots), or other end user tools.
...and the Google bots clearly identify themselves as indexing bots.
Please don't be obtuse. The whole matter is not about Google doing something strange or different than any other search engine. It's just about collective bargaining of one business group against another business. Some german press companies stood up as a (self-appointed and thus somewhat suspect) representation of whole german press industry and are trying to get a bigger slice of the cake.
If I don't lock my house, that doesn't give you the right to break in. You're absolutely right that publishers could make more use of the tools at their disposal to protect their content in whatever way they wish, but they have rights regardless of whether they do.
This analogy only muddles the picture. We are talking about private property, but not a home, more like a shop or pub.
Its only raison d'etre is to serve certain guests. We are talking about a commercial server that is managed by a professional admin, and is explicitly open to only some of the public, and it already screens and filters guests, based on various criteria. For example, the server may disallow hotlinking of images, serve some content only upon receipt of a HTTP cookie confirming payment and serve different content for mobile devices, some movies blocked in certain countries, based on IP geolocation. The decision is based on HTTP request data and will of the owner, encoded as server's configuration. Ask yourself, how much different is Google News bot?
How about this analogy instead: ``if I hire a professional bouncer for my high-street shop, and the bouncer stops some people from entering and lets others in, that doesn't mean those let in can indeed walk in''?
Which would be crazy, and press would have a field day with such a shop.
Some access control is really easy to implement, other types are not, or they're not cost-effective. It's perfectly reasonable for a content provider to combine technological measures (like e.g. IP geolocation) with legal ones (terms of use, copyright).
That said, you're absolutely right: how hard can it be to block the Google News bot for a news organization? Not very. My point is merely that you can't draw any legal or moral conclusions from a 200 OK response, that's absurd.
To riff on your example, if my high-street shop has a sign that says "only people over 18" but doesn't have a bouncer, does that mean people under 18 should feel free to enter because nobody's physically stopping them?
"That's a strawman; the bots never indicated intention to read. They indicated (via `User-Agent' header) intention to index content and got HTTP 200 OK response to that."
The publishers aren't objecting to being indexed. They are objecting because snippets of their articles are being copied, placed on Google News, and used to sell ads without their permission, and they're not being compensated.
Nothing in the HTTP protocol specs (RFC 2616) even suggests it is only meant for direct human consumption. To the contrary, it states,
...and the Google bots clearly identify themselves as indexing bots.Please don't be obtuse. The whole matter is not about Google doing something strange or different than any other search engine. It's just about collective bargaining of one business group against another business. Some german press companies stood up as a (self-appointed and thus somewhat suspect) representation of whole german press industry and are trying to get a bigger slice of the cake.