[htn]

We're using Kafka as a log delivery platform and are quite happy with it. Kafka by nature is highly available and can be scaled quite trivially with the log load by adding new cluster nodes.

We've decided to use journald for storing all of our application data. We pump the entries from journald to Kafka by using a tool that we open sourced: https://github.com/aiven/journalpump.

From Kafka, we export the logs to Elasticsearch for viewing and analysis. Some specific logs are also stored in S3 for long term storage for e.g. audit purposes.