Ask HN: Cloudflare Workers are down?

[cristaloleg]

Got too much 500 on dozen of services now.

UPD: https://www.cloudflarestatus.com/incidents/l6x2h1zp69bc

[elithrar]

This should be resolved. We’re still investigating the underlying root cause, and intend to share a write-up once we have that in hand.

This is not the way we wanted anyone to start their week.

(I am the PM lead for Cloudflare Workers: Databases & Storage)

[elithrar]

Our public postmortem on the incident: https://blog.cloudflare.com/cloudflare-incident-on-october-3...

[nkcmr]

https://www.cloudflarestatus.com/incidents/l6x2h1zp69bc

[huerlisi1]

Just noted on HN and already incident upgrade. Much faster "response" than most other companies:-)

All the best to the people fixing!

[NicoJuicy]

Works for me: https://blog.sapico.me/

Seems to be 30 minutes. According to status page.

Fix is fast. Curious what it was.

[codegeek]

3:55 PM Eastern: Our entire website hosted on cloudflare pages is returning 500. I also cannot login to the dashboard either (it just spins)

EDIT 4:10 PM Eastern: Now I can login to the dashboard but "Workers and Pages" menu is returning errors and no access. Website still down :(

EDIT at 4:23 PM Eastern: RESOLVED. Website (cloudflare pages) is back up now for me.

Looks like they took about 25 mins to resolve.

[camjohnson26]

Our prod app and staging just completely died. Bad day for somebody at Cloudflare

[midasuni]

3.55pm eastern. My websites work

4.10pm eastern, still working

4.23 eastern. Yep you guessed it

Half an hour means they’ve lost their five nines for this year based on this outage alone.

[EthicalSimilar]

Us also, prod and staging are down and dashboard is resulting in API failure requests (500).

[madjam002]

And just 30 minutes ago we were about to flip the switch on a months long migration to Cloudflare Pages for our new website, I guess some things weren't meant to be :')

[codegeek]

Omg. What timing. I feel your pain. We recently migrated to Cloudflare Pages and I was happy at the speed and everything and now this :(. Never had a downtime when I self hosted on my DigitalOcean droplet. damn. Re-considering going back to old school nginx static site hosting.

[JohnMakin]

For any terraform users that may be using code like this:

data "cloudflare_ip_ranges" "cloudflare_ipv4_list" {}

This is coming back with an empty list on some fields and causing havoc in terraform.

[freedomben]

It is shocking to me how bad to non-existent error handling is in most terraform providers. It leads to some remarkably arcane and esoteric error messages

[JohnMakin]

In the time I made this post and now it's come back. Really wish that would've returned an error and not an empty list, that almost caused a disaster in my automation.

[TacticalCoder]

Anyone remember big iron and servers with uptimes of 5 or 7 nines?

I mean: it used to be a thing. Now we have the cloud.

[fizx]

7 9's is 3 seconds of downtime per year. That was never a thing.

[vicnov]

Auth0 seems to be down as well

[thom]

Yeah, can confirm this (for those looking at their status pages which claim otherwise).

[j-rom]

Complete Pages outage for me. I have several sites hosted on Cloudflare Pages and I can't access any of them, they're all returning 500's.

[gsanderson]

Yep, same for me :(

[campbellman]

Fun day to release a blog post[0] about cloudflare page functions, on a site hosted on cloudflare pages.

[0] https://interbolt.org/blog/split-it-and-forget-it

[tootie]

Apparently Auth0 as well. Possibly related.

[juancampa]

Most likely related, I see a `cf-ray` header in the 500 response.

[c22]

It's probably bad that I noticed this just due to a large percentage of my regular online-habits suddenly breaking. I liked the old internet where websites just broke one at a time.

[NicoJuicy]

That was before ddos became common and cheap to execute.

[blintz]

This is preventing new logins to ChatGPT.

[toomuchtodo]

Error 1101 Worker threw exception. Interestingly fronting their auth0 tenant with CF.

[xrd]

I can't login to my domain dashboard either. Maybe that is a downstream effect of workers being offline?

[dogweather]

Yes — Workaround is to disable your workers. That got my site back up and running.

EDIT 15:07 MDT: People are reporting that Workers are back up. Mine isn't in my site's critical path. So I'm going to leave the Worker disabled (un-routed) until tonight.

[codegeek]

Are you saying disable workers and then your cloudflare Pages will be back up ?

[gsanderson]

My sites have started coming back up now. Their site has also just started working again (previously got a 500): https://pages.cloudflare.com/

[pcblues]

I won't be the first or last to say these three things:

The internet was meant to stop reliance on single sources (in case of nuclear war)

The size of a house of cards increases the number of failure points

Marketers lie

[jve]

> The internet was meant to stop reliance on single sources

You have all the technical means. Your home server possibly won't be reachable, yes.

The global connectivity as-is is really, really, really fault tolerant.

[CommonGuy]

Cloudflare Pages aren't working on a few of my sites too

[mparnisari]

It is funny that just a few days ago the company that laughed at Okta for a breach and whose core competency is availability are now experiencing an outage.

[ystad]

You should probably indicate what you meant by laughed at okta. Do you have a link??

[gkfasdfasdf]

Ongoing DDoS attacks are targeting sites that raise funds for Gaza relief efforts: https://twitter.com/arblauvelt/status/1719027920054702363

I wonder if there's a connection.

[elithrar]

Not related.

(I am the PM lead for Workers databases & storage)