[olauzon]

I've been using Rust to process fairly large amounts of streaming financial data. I am using Protocol Buffers (and evaluating Cap'n Proto) for network serialization and Kafka for buffering/queuing, and overall the library support is quite good.

Previously I was using JVM languages for this purpose, but grew weary of the resource footprint, and especially the unpredictable GC pauses. I am aware of the Azul JVM which removes GC pauses and of various Java techniques to avoid GC altogether, but switching to Rust provided a GC-less model from the ground-up, a powerful type system, and familiar functional programming facilities at no cost.

[kibwen]

What Rust libraries are you using for protobufs?

[olauzon]

I have been using https://crates.io/crates/protobuf. It generates pure Rust protobuf implementation files.

[bjz_]

How easy is it to get Kafka and Rust working together?

[olauzon]

Fairly straightforward overall using https://crates.io/crates/kafka. The client doesn't automatically handle Kafka node failures, so that's the responsibility of your application code.

I haven't had the chance to try https://github.com/fede1024/rust-rdkafka yet, but it looks promising and partially wraps the C/C++ library https://github.com/edenhill/librdkafka.