[carlio]

I work as a contractor so I move between places. I have found a few companies trying to introduce kafka, and every time it has been a solution in search of a problem.

I don't doubt that it has a good use case but I have so far only encountered the zealots who crowbar it into any situation and that has left a residual bad taste in my mouth. So I fall into the "hate it" side.

[snapetom]

> and every time it has been a solution in search of a problem.

More refined to this, in my experience at the last two jobs, the queue problem is there, but the Kafka solution is based solely on "enterpriseyness" of Kafka, not any practical reason. RabbitMQ is highly performant, SQS is really easy. Both are great queues. Kafka is muuch more, yet, Kafka is chosen because "it's enterprise."

[EdwardDiego]

Kafka isn't even a queue. I've done consulting on Kafka, and several times my recommendation is "You don't want or need Kafka".

A classic sign of "you wanted an MQ" is when a consumer writes a message to a topic to let the producer know it read the message the producer wrote...

[relay23]

Maybe "it's enterprise" means that's what the enterprise standardized on. There are a couple of practical reasons that come to mind on why that's the case - a) it's more resilient and durable than messaging platforms, and b) it is a platform of dumb pipes, so to make it a central data bus managed by platform teams means that they don't have to get into the detail of which queues perform what functions, have what characteristics, etc. Rather the client teams in the various business units can take care of all of their "smarts" the way they want. It also handles log/telemetry ingestion, data platform integration, and interservice comms use cases which is pretty multi-functional. That's the primary reason why Kafka has become such a pervasive and common platform, it's not because it's trendy, in fact most operations teams would rather not even have to operate the kafka platform.

[AtlasBarfed]

RabbitMQ is "highly performant" is a handwave. The words tell me nothing, just like any other tech/software that is described as "powerful".

In my last two major gigs, RabbitMQ was already being run in a clustered config, and it was not going well. Both places were in the process of doing arch changes to do a change to Kafka.

It seems like something that works great in a big scaled node and you can go to big nodes these days, but I don't think it is ready for cloud/distributed durability.

I'm not aware of Jepsen testing of RabbitMQ in distributed mode for example, and I wouldn't consider any distributed/clustered product that hasn't let Jepsen embarass it yet.

Cassandra and Kafka are frequent examples of YAGNI overengineering (although the fault tolerance can be nice without scale), the reality is that pumping up single-node solutions for too long is a big trap. Projects that start to stress single-nodes (I'm thinking like a 4xlarge anything on aws) should probably get to thinking about the need for jumping to dynamodb/cassandra/bigtable/kafka/etc.

RabbitMQ --> Kafka is a pretty easy lift if your messaging has good abstractions.

relational DB --> Cassandra is a lot bigger headache because of the lack of joins.

[cbsmith]

I always find the "it's enterprise" statement so humourous, given how much time I've had to invest in convincing enterprises that Kafka wasn't some weird fly-by-night technology that couldn't provide for the demanding enterprise.

[deepsquirrelnet]

It’s amazing how far that one word can go.

In a former life, I even heard this one:

- We use CentOS.

Why?

- RedHat is enterprise.

(But you’re not even paying for enterprise support)

[kfk]

The biggest issue for me is people using kafka for mqtt, mqtt is a pub/sub broker already. The other issue is thinking of Kafka as some kind of “innovative” data ingestion tool, so now instead of 50 extract jobs per day, you got to reconcile millions of events in realtime. I think message brokers make sense, but they are message brokers, nothing else, no?

[EdwardDiego]

It's really good when you're producing a lot of data fast that you don't want to lose, and you want multiple consumers to read.

It's a complex tool that solves a complicated problem. But if you don't actually have that problem, then that's a whole lot of complexity for no gain.

[tofuahdude]

Hypothetical use case I've been thinking about:

Say I want to log all http requests to the server (I know I said a keyword of log) and then process those logs into aggregates, stick them in a time series.

Would it be insane to "log" everything into kafka? Or what would be the more "correct" tool for that job?

[nithril]

Same finding.

That's why that sentence in the article "but almost every technology company uses it." should be rephrased to "but almost every technology company do not need it"