Ask HN: Two people laid off due to known Google/Elastic billing bug. Now what?

[william-at-rain]

Facts:

* GCP + Elastic Cloud have an acknowledged (in writing) bug that is charging us 10x the proper amount. It started early on New Years Day and affects clients using Elastic Cloud purchased through Google Cloud's Marketplace.

* We are a bootstrapped startup with ~$1M ARR (launched product in May 2020). Still growing and just recently breakeven.

* Google has tied up 100% of our cash with their auto-bill. Credit cards are full ($32k in charges in a week).

* I laid off two people because I couldn't pay them. Everyone else is due a paycheck at the end of the month.

* Google is still charging us 10x normal rates every day. Elastic is "working to fix the issue."

* We can't get off Elastic Cloud because we can't increase quotas for machines until... we pay our GCP bill.

I always thought the stories on here about Google were 1 in a million... but here we are.

"Support" on both sides is a stone wall. What should I do to save this company and all our jobs?

[b112]

Old crusty greybeard here.

If you're being charged 10x @32k, that's 3k/week 'normal' charges. This makes me think that your infrastructure could be hosted for about $3k/year, redundant, with bare metal and minimal downtime.

And that's with renting bare metal. Not even the cheaper option of owning your own.

There are lots of paths here, but I doubt you want to leave GCP, even though you're now seeing the perils of:

1) Zero real support from places like AWS/GCP/etc

2) Zero real control, or ability to easily migrate, once you tie-in

That said, as you believe you can do nothing to move/mitigate, I believe this is a logical appeal to GCP to 'fix things', by shaming them with this horrid bug, via this post.

And they should be shamed! GCP, as AWS, is charging an insane premium for such services.

However, there isn't much for us to do, accept agree that the entire scenario is absurd. And my post serves that, as I hate to see so many young companies have all their runway eaten up by absurd, 100x to 1000x costs in the cloud, even outside of these bugs.

And bare metal is how you control your own destiny. It's also how you remain un-vendor locked into cloud 'extras'. You can also run your own 'cloud' on bare metal too.

NOTE: my email is in my profile. Suggest (if you want to remain unannounced) you create some sort of throw-away for yours, if you want to enable out-of-band communication on topics like this.

[william-at-rain]

Thanks.

> If you're being charged 10x @32k, that's 3k/week 'normal' charges

Close - about $2,500 a week.

I spent the morning adding SSDs to a bare metal HP DL360p with 32 cores and 192GB RAM on it.

Elastic fixed their bug this morning - so the bleed has stopped (confirmed on GCP dashboard).

We still have $32k tied up, so I bought the hardware out of pocket w/ personal savings. We started ingest to the VMs (new single-server, multi-node ES deployment) as well.

> You can also run your own 'cloud' on bare metal too.

I'm thinking about buying three more servers and throwing them in our rack. Everything will be slower, but we won't have the same version of existential crisis in our future. That adds so much OTHER risk (and expense) as well... Multicloud is looking appealing here (via, ironically, K8s all the things).

[warkolm]

[I work at Elastic]

can you shoot me an email at markw at elastic dot co and I will see what I can assist with

[william-at-rain]

That's... not true.

ES is critical infra for us, and I'm not really capable of managing the devops to make a switch happen. Otherwise I wouldn't be paying an override to ES for what amounts to a Kubernetes spec and some glue.

I've been engaged w/ Google and Elastic for a week today. Not sure where you get your certainty.