Saturday, July 26, 2008

Just Saying

Amazon has released information about the cause of the recent S3 outage and what they are doing to ensure that their "performance is statistically indistinguishable from perfect":
Here are the actions that we're taking: (a) we've deployed several changes to Amazon S3 that significantly reduce the amount of time required to completely restore system-wide state and restart customer request processing; (b) we've deployed a change to how Amazon S3 gossips about failed servers that reduces the amount of gossip and helps prevent the behavior we experienced on Sunday; (c) we've added additional monitoring and alarming of gossip rates and failures; and, (d) we're adding checksums to proactively detect corruption of system state messages so we can log any such messages and then reject them.
Except for (d), these actions don't really address the cause, but only mitigate the effects.