Uh, has anyone on this thread heard of HIPAA? I'm pretty sure having a summer intern get full access to actual patient data shouldn't be possible under a properly implemented set of HIPAA processes, and the same goes for the accidental UPDATE.

The story reminds me of the day I was introduced to "BEGIN TRANS", "COMMIT" and "ROLLBACK" when someone upgraded the Sybase console and helpfully changed the default setting so we didn't need those pesky semi-colons to finish a query any more. The result was:

  DELETE * FROM TABLE x
  131054 rows deleted
  WHERE a = "foo"
  >> Malformed query <<

Phone starts to ring a few seconds later as all the users saw their morning's work disappear.

This stuff is way too easy for us noobs. Thank goodness that with modern technology we've found ways to make sure it doesn't happen any more... :-)

Honestly, as long as he's had HIPAA training, there's no difference between an intern making the changes and a 10-year employee doing so.

We had a great experience at PIE, highly recommended.

This is fair comment, especially given that they actually emphasized this multi-AZ redundancy.

The genesis of the article was the press implying that to use the cloud your only choice was to trust AWS provided 100% up-time, and this is a position we disagree with.

Our point is it's unfair to blame Amazon 100% when there were viable and affordable ways round this outage. Some of our customers were in US East and had off-site backups, scripted server deployments and were able to get back online pretty quickly.

How were they able to get back online? Deploying to another AWS site or to another cloud provider? Presumably they didn't own enough servers themselves?

At what point did they make the call that the outage was too serious and they would lose all data since the last backup and start migrating? Had they pre-planned for it, or was it ad-hoc? Will they stay where they are now and use Amazon US East as their failover or migrate back in due course? Or rearchitect to handle this in future?

(thanks, re: name).

Is it just luck that the problem which happened was one they prepared for instead of one they didn't?

There's an upvote button for that.

Upvotes are literally a +/-1 of agreement. It's a bit of a tough crowd (probably related to the subject matter), but either way, comments are meant to add something to the discussion.

+1 Sometimes it's not cost effective to do all the layers. One of our clients is going to lose $2500 to $3000 today, which is less than they would have spent on avoiding the issue in the first place.

Yes, absolutely. This is where the product started, tracking EC2 billing. You have to pull down the raw usage data too as the daily Amazon bill updates can be a bit out of sync.