[william-at-rain]

Thanks.

> If you're being charged 10x @32k, that's 3k/week 'normal' charges

Close - about $2,500 a week.

I spent the morning adding SSDs to a bare metal HP DL360p with 32 cores and 192GB RAM on it.

Elastic fixed their bug this morning - so the bleed has stopped (confirmed on GCP dashboard).

We still have $32k tied up, so I bought the hardware out of pocket w/ personal savings. We started ingest to the VMs (new single-server, multi-node ES deployment) as well.

> You can also run your own 'cloud' on bare metal too.

I'm thinking about buying three more servers and throwing them in our rack. Everything will be slower, but we won't have the same version of existential crisis in our future. That adds so much OTHER risk (and expense) as well... Multicloud is looking appealing here (via, ironically, K8s all the things).

[b112]

Not clear on why everything will be slower. In my experience, when you run your own hardware, you have guaranteed responsiveness to processing power, RAM speed, I/O.

AWS rates, as an example, are so insanely high, that you can literally buy a server with 100x the processing power for the same price point as an AWS instance. That is, hardware purchase amortized over 2 year or so.

In terms of hosting with bare metal, you can easily procure bare metal, non-managed, quite cheaply on a monthly basis. Here you're often paying 1.5x or 2x the price of raw hardware, yet you have someone managing all of the infrastructure for you.

(Think of this as leasing costs, without having the hardware left over afterwards)

The key is to have redundancy, just as you would in the cloud. However, the more you scale, the less that redundancy costs, as you do not require 2x the hardware, merely spare hardware to swap in during a failure event.

EG, 2 DB servers means one must be redundant, 10 servers therefore means 10% of your capacity is for redundancy, not load.

And you can easily spread hardware out over multiple datacenters.

Worst case, you can always target said hardware for development, saving on costs for testing/devs, if you ensure that deployment is not specific to one cloud provider. You can easily roll out, for example, KVM and spawn whatever containers you wish inside each VM.

Although, I'll get all greybeard on you here, and suggest the merits of containerization don't overcome the reduced reliability due to additional complexity, the security issues due to unvetted upstream images, and the list goes on.

But that's a whole other conversation, and you likely have few free CPU cycles to think on such things right now.

EDIT:

I'm not sure what industry you're in, but most companies I deal with tend to have very consistent load. That is, day in and day out, load is relatively consistent, with little variance between December 3rd and May 5th.

Or, at least, peak load doesn't suddenly go off the charts.

If you're not consumer facing, but instead corporate facing, you often have a much better eye into load.

For example, you know more load is coming due to monthly subscription numbers.

So two things:

- as you literally pay 1/100th the cost for bare metal, scaling is typically not even a factor here. You save absurd amounts of cash, because you can already scale to 100x the load for the same price.

100x the load is an insane amount of scale.

You're almost certainly going to run into non-hardware issues at that scale. DB provisioning/size. Scale and scope of data storage. Code issues. Internal networking issues.

- in the old days, devs would spend more time optimizing code, for example, I've literally reduced DB load by astonishing numbers, not even joking here, 1000x load reduction due to query rewrites, 100k times faster due to code rewrites

By the way, I realise I have no optics here into your setup. Just consider this COVID lockdown-free, bored out of my mind advice. :P