AWS’s S3 outage yesterday was so bad Amazon couldn’t get into its own dashboard to warn the world

Tuesday’s Amazon Web Services mega-outage knocked offline not only websites big and small, by yanking away their backend storage, but also knackered apps and Internet of Things gadgets relying on the technology.

In fact, the five-hour breakdown was so bad, Amazon couldn’t even update its own AWS status dashboard: its red warning icons were stranded, hosted on the broken-down side of the cloud.

Essentially, S3 buckets in the US-East-1 region in northern Virginia, US, became inaccessible at about 0945 PST (1745 UTC). Software, from web apps to smartphone applications, relying on this cloud-based storage quickly broke, taking out a sizable chunk of the internet as we know it.

AWS has many regions, and US-East-1 is just one of them. Developers are supposed to spread their applications over different data centers so when one region goes TITSUP, it doesn’t take your whole platform down. For various reasons – from the fact that programmers find distributed computing hard to the costs involved – this redundancy isn’t always coded in. And so here we are.

The carnage on Tuesday midday, Pacific Time, as shown on the AWS dashboard customers see after logging in
(click to enlarge)

This is by no means an exhaustive list of things that fell over or were wobbly today, due to the S3 downtime, but here’s a start: Docker’s Registry Hub, Trello, Travis CI, GitHub and GitLab, Quora, Medium, Signal, Slack, Imgur, Twitch.tv, Razer, heaps of publications that stored images and other media in S3, Adobe’s cloud, Zendesk, Heroku, Coursera, Bitbucket, Autodesk’s cloud, Twilio, Mailchimp, Citrix, Expedia, Flipboard, and Yahoo! Mail (which you probably shouldn’t be using anyway). Readers also reported that Zoom.us and some Salesforce.com services were having problems, as were Xero, SiriusXM, and Strava.