The same way as Kafka. Jitsu nodes (=collection agents) writes to write ahead log, and then either sends data to destination right away, or sends data in batches.
Hi! My name is Sergey, I’m a Jitsu product engineer. I’ll gladly answer your question! AsyncLogger works asynchronously by design. There is a go channel which writes JSON’s to the log file.
Answering your question: the service doesn’t wait until data is written to the log prior to returning success to the client. WAL log is designed for keeping events JSON’s between Jitsu instance restarts to prevent data loss. When you deploy your Jitsu application, it will handle service restart signals (e.g. sigterm) and closes database connections as well as other resources. All incoming events are stored in WAL log in this time. So, after the Jitsu starts, all events from WAL log will be passed to the main events JSON pipeline and stored to the destinations.
tomnipotent|4 years ago
https://github.com/jitsucom/jitsu/blob/0aaa74b59eb9d8c885c80...
I see that it instantiates an "AsyncLogger" - does the service wait until data is written to the log prior to returning success to the client?
Is the WAL the same source used to feed both database storage destinations and other SaaS destinations?
xtreding|4 years ago