150K hits in 30 minutes is 83 hits/second. That's a lot to ask for from a shared-hosting account.
When the traffic starts to impair neighboring sites, something has to be done. Just about any ISP will do the same thing: block the site with the surge, that could possibly make other arrangements, rather than inconvenience other customers whose traffic is as expected/usual.
The detail missing so far is why Pair noticed today, if it was the same level of traffic as before, or a slow build. Was a new threshold crossed? (Did someone's HN-focused tool go haywire?)
The Pair message suggests end-of-day logs will be the way to tell for sure.
So let's see... you're hosting a website that has gotten large; that is, it's grown to the point that it will need higher-cost services in order to meet demand. You have a chance to add a valuable customer to your client base. How best to handle this?
Yeah, Pair screwed themselves pretty badly by pulling this trick. They would have been far better off doing something like:
Dear [account_contact_user]
Your website traffic has risen beyond the maximum threshold of [threshold_amt] for the [name_of_level] level of service.
Since we appreciate your business of the last [length_of_service], we have given you a 24 hour courtesy upgrade to our next level of service -- [name_of_next_level]. If, by [end_time] you decide to keep this level of service you must contact our sales center to arrange payment. Otherwise we will have to start throttling traffic to your server so that it remains below the threshold of [threshold_amt] and does not impact our other customers.
If you have any questions about this courtesy upgrade, or wish to keep this new level of service, please contact [account_manager] at [account_manager_details].
Thank you for using Pair Networks for your hosting needs.
Of course, the flip side is that leaving it running adopts an attitude of "screw all our other customers, they can eat crappy service while we kiss up to the popular guys who are chewing up everybody else's server resources". Which isn't what I'd look for in a hosting provider...
False dichotomy. The correct way to handle this would have been to temporarily move the shared server to hardware where it won't impact other customers and notify YC that they need to move their server to a bigger server or the site will have to be shut down. Presumably with an ultimatum of a week or whatever.
I was being tongue-in-cheek, but there is a real point here.
As you say I doubt today was unusual for HN, it's the way Pair went from zero to shut-down.
One of the things I like about the way that Joyent operate as a cloud host is that they allow you burst on shared boxes because of those times you need it. At the same time they'll let you know you need to think about buying more resources without just slamming on the brakes.
Pair should do a much better job of noticing a soft limit earlier, so if a heavy traffic day had hit HN PG et al should have been already aware they were overusing their shared hosting and were planning a route out.
Exactly - they missed an obvious opportunity to upsell. If Trevor had gotten a mail stating that he needed to upgrade he would most probably just have done so and been happy. It's not like he can't afford it :-)
That technology is not out there... A static site might be fun at that level and not causing problems, where as a WordPress site with super cache is fine, but then a Drupal site can only handle 1/4th of that.
It currently is not really possible to track that per user in a shared environment and see who is using what resources from an individual perspective of MySQL / Apache / CPU / Mem.
Absolutely wrong. That technology exists and is being used millions of times a day at sites all over the world. DreamHost, for example, employs just such a technology for their shared hosting: http://wiki.dreamhost.com/index.php/CPU_minutes
The "ask" is not that a shared hosting account support 83 hits/second. It's that the hosting company demonstrate the bare minimum of respect and professionalism in contacting the hostee ASAP when their site is being shut down due to excess traffic. This isn't bitbucket or geocities, dollars are changing hands. If you don't like running a business that makes money then by all means, treat your paying customers like they are a burden to you, they will get the hint and go elsewhere.
I doubt it is a HN-Focused tool, as this affected the static content from www.ycombinator.com, and images and CSS are not the focus of bots usually.
Assuming it was not a DoS attack, a smart host should have noticed the traffic load increasing over time, and offered to upgrade to a less-loaded server and recommend a dedicated server.
By disabling the site, they have lost a customer, and lost on a up-sell to a dedicated server.
You're assuming the load today wasn't extraordinary. That's not yet clear. PG doesn't know of any surge, probably based on figures he has access to on other hosts and analytics tools. But what if HN/YC static resources at Pair were newly being deep-linked, by a much more popular site? Only seeing the end-of-day logs from Pair, compared to previous days, can definitively answer this question.
I'm sure that's what the general News.YC/YC stats show. But as the Pair.com tech mentioned, today's logs only become available at midnight ET. If there was a surge this morning -- to the static resources only, as if by offsite deep-linking -- Pair's logs are the only place it would be evident.
Todays's logs are similar to other days. I guess the traffic is high for a shared account, but I've never noticed it being slow so I didn't think about it.
I would have been happy to respond to an email saying I should upgrade & pay more, but even after a few emails with tech support that option didn't come up.
The only limit they advertise for that class of account is data transfer, and because it's mostly just serving uparrow.gif -type files we're well below that limit.
When the traffic starts to impair neighboring sites, something has to be done. Just about any ISP will do the same thing: block the site with the surge, that could possibly make other arrangements, rather than inconvenience other customers whose traffic is as expected/usual.
The detail missing so far is why Pair noticed today, if it was the same level of traffic as before, or a slow build. Was a new threshold crossed? (Did someone's HN-focused tool go haywire?)
The Pair message suggests end-of-day logs will be the way to tell for sure.