[darkbatman]

how are you ingesting from kinesis to clickhouse. are you using some custome sink connector or processes on ec2 or lambda?

[ashug]

We actually use Kafka rather than Kinesis, although they're very similar. For writing to ClickHouse from Kafka, we use the ClickHouse Kafka sink connector: https://github.com/ClickHouse/clickhouse-kafka-connect.

[darkbatman]

we are actually trying something similar but possible kinesis + clickhouse or kafka + clickhouse. Currently kinesis seems easier to deal with but not a good intergration or sink connector available to process records at scale for kinesis to put into clickhouse. Were you ever felt into similar problems where you had to process records at huge scale to be able to insert into clickhouse without much delay.

One more thing is kinesis can have duplicates while kafka is exactly once delivery.