Show HN: Tansu – Kafka Compatible Broker Transforms Proto/Avro/JSON into Parquet

[shortishly]

What is it?

- Tansu is an Kafka compatible broker with S3 or PostgreSQL storage - Broker schema validation of Proto/Avro/JSON backed topics - Proto/Avro/JSON backed topics are also written as Parquet record batches for easy ingestion by downstream tooling

Step by step tutorial: https://blog.tansu.io/articles/parquet

Tansu is written in async Rust with the Tokio runtime. We build a single static linked binary that contains the broker, proxy and admin/utility commands in ~60MB. Also available as a from scratch docker container.

The Kafka protocol is implemented with serde using proc macros to transform the JSON definitions into Rust data types (https://blog.tansu.io/articles/serde-kafka-protocol).

Proto/Avro/JSON is converted into Apache Arrow data types, and then into Apache Parquet with a sprinkle of Apache Datafusion.

We also have an example of it running on fly.io with tigris (https://blog.tansu.io/articles/fly-with-tigris) that scales brokers back to 0 between requests.

Looking forward to any feedback and hope that Tansu can improve your workflow!