Ask HN: Is Google Compute down?

[hellcow]

I'm not able to ssh into any of my boxes or access any of the sites, yet my status monitor isn't showing downtime. Spotify is also down for me, which is another GCP customer.

I'm in Los Angeles but the servers are hosted on us-central1

[sethvargo]

Hi all - Seth from Google here. Our team is aware and we are working on mitigation. In short, a third party telco provider is advertising on one of our IP blocks. Unfortunately that's all the information I can share at this time.

[sethvargo]

We've updated our status page with as much information as we can provide at this time: https://status.cloud.google.com/incident/cloud-networking/18...

Our teams are continuing to work with upstream and downstream service providers to remedy the issue.

[konschubert]

EDIT: This is a general statement, I am not complaining to google here.

This kind of thing should not be possible. Are there any protocol proposals or other kind of upgrades to the routing protocols that would prevent these kind of mistakes/attacks?

[amingilani]

Reminds me of the time Pakistan knocked YouTube offline by hijacking their IPs globally[0]

Edit: Didn't someone recently share a tool to monitor BGP hijack attempts?

[0]: https://www.cnet.com/news/how-pakistan-knocked-youtube-offli...

[garysahota93]

I love that Google monitors this site. I really appreciate you reaching out and letting us know the current status!

[fxdoublecute]

thanks for the update! FWIW we started noticing the connectivity problems around 2018-11-12 21:17 UTC

[regnerba]

Google IPs seem to be being routed to China for us.

We have servers in San Jose that cannot access Google services. Trace route shows everything going to China when leaving the San Jose data center. We can access the same services from Vancouver just fine.

[docker_up]

How many times does this have to happen before China's privileges to do things like this get revoked? At this point, it can't be just a mistake and must be some state-sponsored hacking. Seems like a great way to find out where a particular Spotify user's IP address is.

[gred]

Related? https://news.ycombinator.com/item?id=18385920

[lostmsu]

Seems like its time to start or accelerate a working group on secure BGP.

[uji]

Related to https://twitter.com/thousandeyes/status/1062102171506765825

[jamalex]

Despite the subdomain, the IP for ChinaTelecom-gw.transtelecom.net (217.150.59.249) seems to be based in Russia, as does the carrier: https://en.wikipedia.org/wiki/TransTelekom

[sterlind]

Seems likely to be TT's gateway to CT. New theory: TransTelecom brought up a new gateway to ChinaTelecom, which incorrectly gossiped all advertisements from ChinaTelecom. This caused a leak, since CT has bgp highjacking of Google IP ranges for the GFW within China, but ordinarily doesn't leak them outside the country. TransTelecom misconfigured the gateway to broadcast everything advertised by ChinaTelecom, bringing external traffic into the GFW.

[xolox]

Reading through the comments here I'm recognizing "China Telecom" from an article on a BGP hijack that was published about a week ago, I still had the article open in my browser:

https://arstechnica.com/information-technology/2018/11/stran...

In another comment in this thread I read:

> Seems like its time to start or accelerate a working group on secure BGP.

Indeed things can't go on like this for much longer...

[faissaloo]

I kept getting SSH bruteforce attempts from IPs on China Telecom a while back. Wonder what they're up to...

[davismwfl]

I am on the East Coast, in Florida and seeing the same thing with traffic heading to China, lots of "chinatelecom-gw.transtelecom.net" in traceroutes I have never seen prior.

[scrollbar]

Getting this as well in SF. transtelecom.net WHOIS says they're Moscow-based

[CydeWeys]

We urgently need a solution for routing traffic to IP addresses that is better than BGP.

[dasm]

Agreed. This appears to be a repeat of the attack covered here: https://news.ycombinator.com/item?id=18385920

I'm not familiar with BGP routing attacks; the article above seems to imply the attacker needs to compromise certs in order to glean useful data from the attack.

If that's accurate, is this Google-oriented traffic vulnerable to this type of attack?

[raesene9]

or ISPs could implement the proposed BGP security standards... that have been proposed for , well a long time.

[aviv]

Funny, a day after I posted this...

https://news.ycombinator.com/item?id=18429099

Is our first time actually rolling over the entire stack to AWS - and it worked!

GCP outage currently is massive, can't even use other regions.

Edit: This also affected AWS Oregon region earlier. I do not know how yet, but they too were unreachable briefly. Seems to be okay now.

[infogulch]

So... what's the current state of a secure BGP? I feel like this in the top 3 security threats to the whole of the internet.

[TodayIsTheDay]

Does anybody else have chinatelecom-gw.transtelecom.net [217.150.59.249] in the traceroute for www.google.com

[kacy]

Also showing up on a traceroute to spotify.com for me.

[leesalminen]

What is today the day for?

[dickfickling]

yeah, GCP is having a serious outage. Our site is down, so's Pivotal Tracker

Edit: We're also in Los Angeles, connecting to us-central1. Seems to be a pattern?

[hellcow]

Bugsnag's app.bugsnag.com is down as well.

[syogi]

I'm in Los Angeles and I can access my GCP Console but I can't access Google services like google.com or Maps or Gmail.

EDIT: Some services are intermittently responsive. I had ~5 minutes of no access to anything. Some are slowly coming back.

[fxdoublecute]

we manage services deployed in every GCE region, and our monitoring in London is reporting every GCE region having intermittent connectivity. no problems with our services in the other major clouds (we use basically all of them)

[vamos_davai]

I have trouble accessing YouTube. I live in Sherman Oaks (a town of Los Angeles).

[gsibble]

Can confirm many IPs are being sent to China.

[crunchlibrarian]

This is nuts.

[cobookman]

There's a current BGP prefix hijacking issue currently being mitigated.

[bifrost]

I hate to break it to everyone, but the technology to filter this sorta thing has existed for a very long time, but people often don't use it. Most of the time this sort of thing is accidental (IE: operator error)so a lot of operators kinda ignore it. Check out "IRR Power Tools" if you're interested.

[jamalex]

Same thing here in San Diego. Traceroute to spotify.com going through LA, San Jose, NY, London, Amsterdam, Frankfurk, "mskn17ra-lo1.transtelecom.net", then ChinaTelecom-gw.transtelecom.net.

[RayHawk]

I'm getting the same thing. Servers are in us-east1 and tracert is ending at chinatelecom-gw.transtelecom.net [217.150.59.249]

[xstephen95x]

seems like https://arstechnica.com/information-technology/2018/11/stran...