Ask HN: Is AWS S3 down?

[coltonv]

Our personal webapp is having S3 problems, and I can't seem to upload files for slack. Is anyone else having these issues?

Generally getting 503 slowdown errors and more recently 500 internal server errors.

[sigil]

And 10+ minutes after this thread was started, the AWS status page finally acknowledges there's an issue. HN really is a best-in-class monitoring service. https://status.aws.amazon.com/

[ajmarsh]

11:58 AM PDT We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.

12:21 PM PDT We can confirm that some customers are receiving throttling errors accessing S3. We are currently investigating the root cause.

[paulddraper]

For us, lots of errors, then retries, then rate limits because of the retries.

---

On our account dashboard

> 11:58 AM PDT We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.

On AWS's status page https://status.aws.amazon.com/

> No recent events.

Damn thing never works.

[cperciva]

Issues between 18:39 and 18:58 UTC. First InternalErrors, then SlowDowns -- I'm guessing due to nodes being taken out of service when they failed.

Given how quickly this was fixed, I suspect they were deploying new code and the rollout was automatically aborted when the elevated error rates were detected.

[ljosa]

Still seeing SlowDowns as of 19:08Z.

[kainosnoema]

Also getting 503 Slow Down and 500 internal server errors from S3 in us-east-1

[xfax]

Yup, it looks that way. Getting this error when using the CLI:

An error occurred (SlowDown) when calling the ListObjects operation (reached max retries: 4): Please reduce your request rate.

[leevlad]

Yup, getting elevated error rates uploading to S3 - writes failing as well as closing the uploads. Also seeing 503 slow down as reported by others.

[mmontagna9]

Yes we are getting slowdowns and internal errors as well. Also noticed one of our EC2 instances shutdown due to failed health checks.

[ihaveajob]

Question to folks out here: How do you deal with this? Ostrich strategy, or do you actively manage via multi-site redundancy?

[actuator]

We have support for other S3 regions in which we duplicate the data but the switch is manual not automatic.

[jwg2s]

Replicate to additional regions, point other services to these replicated buckets until issue is resolved.

[paulddraper]

Multi-region redundancy.

But with things in "read only" mode, to make the replicating simpler.

[actuator]

Encountering Slow Down errors as well

[actuator]

The errors seem to be reducing now.

[sigil]

Yup I'm seeing 503s in us-west-2 (EC2 instance talking to S3), but outside EC2 is okay.

[jpetrucc]

Getting 503's and slowdowns for everything on s3 and cloudfront randomly

[xfax]

Does anyone else have some of their buckets missing from the console list?

[bigjust]

S3 in both US-East-1 and US-East-2 are seeing these errors

[jwg2s]

Yep, we're seeing that but recovering now it seems.

[topstriker515]

having problems here as well. Slow down errors, 500 errors, and a random EC2 termination we suspect might've been related to EBS

[coreymayo]

I am getting a 503 Slow Down from S3 requests.

[coreyw]

We are noticing the slow down and the rate reduce issue only on GET requests. Our PUT requests seem to be going in just fine.

[greenhathacker]

We are seeing both "Slow Down" and "Please reduce your request rate."

[smmnyc]

same here

[vhost-]

Seeing issues with cloudformation as well.

[donPinOB]

getting this as well, 500 & 503