top | item 43837818

(no title)

wwarner | 10 months ago

personally i think streaming is quite a bit simpler. but as you you point out, no one cares!

discuss

order

carefulfungi|10 months ago

Batch processing is just stream processing with a really big window ;-). More seriously, I find streaming windows are often the disconnect. Surprisingly often, users don't want windowed results. They want aggregation, filtering, uniqueness, ordering, and reporting over some batch. Or, they want to flexibly specify their window / partitioning / grouping for each reporting query. Modern OLAP systems are plenty fast enough to do that on the fly for most use cases - so even older streaming patterns like stream processing for real time stats in parallel with batch to an OLAP system aren't worth the complexity. Just query the DB and cache...