[AJSDfljff]

Unfortunate the interesting part is missing.

Its not hard at all to scale to PB. Junk your data based on time, scale horizontally. When you can scale horizontally it doesn't matter how much it is.

Elastic is not something i would use for scaling horizontally basic logs, i would use it for live data which i need live with little latency or if i do constantly a lot of log analysis live again.

Did Binance really needed elastic or did they just start pushing everything into elastic without every looking left and right?

Did they do any log processing and cleanup before?

[fulmicoton]

This is their application logs. They need to search into it in a comfortable manner. They went for a search engine with Elasticsearch at first, and Quickwit after that because even after restriction the search on a tag and a time window "grepping" was not a viable option.

[jcgrillo]

This position has always confused me. IME logs search tools (ELK and their SaaS ilk) are always far too restrictive and uncomfortable compared to Hadoop/Spark. I'd much rather have unfettered access to the data and have to wait a couple seconds for my query to return than be pigeonholed into some horrible DSL built around an indexing scheme. I couldn't care less about my logs queries returning in sub-second time, it's just not a requirement. The fact that people index logs is baffling.

[AJSDfljff]

Would be curious what they are searching exactly.

At this size and cost, aligning what you log should save a lot of money.