[toenail]

Dunno, I've had three node clusters running very stable for years. Which issues did you have that require a full team?

[PedroBatista]

Even most toy databases "built in a weekend" can be very stable for years if:

- No edge-case is thrown at them

- No part of the system is stressed ( software modules, OS,firmware, hardware )

- No plug is pulled

Crank the requests to 11 or import a billion rows of data with another billion relations and watch what happens. The main problem isn't the system refusing to serve a request or throwing "No soup for you!" errors, it's data corruption and/or wrong responses.

[toenail]

I'm talking about production loads, but thanks.

[unethical_ban]

To be fair, I think it is chronically underprovisioned clusters that get overwhelmed by log forwarding. I wasn't on the team that managed the ELK stack a decade ago, but I remember our SOC having two people whose full time job was curating the infrastructure to keep it afloat.

Now I work for a company whose log storage product has ES inside, and it seems to shit the bed more often than it should - again, could be bugs, could be running "clusters" of 1 or 2 instead of 3.

[xeraa]

There are no 2-node clusters (it needs a quorum). If your setup has 2-node clusters, someone is doing this horribly wrong.

[toenail]

I'm not even sure "get overwhelmed" is a problem, unless you need real time analytics. But yeah, sounds like a resources issue.