[lyziinc]

Hi I’m one of the logflare devs and I work on observability at Supabase.

Great question. To directly address some of the tools you mentioned:

- Logstash is the transport and transformation portion (along with Filbert) in the elastic stack, and it performs the same functions as vector. It is out of scope for Logflare, which focuses on acting as a centralised server to point all your logging pipelines to.

- Kibana is the visualisation layer of the elastic stack. For Supabase, this functionality is taken over by the Supabase Studio, and the reporting capabilities will eventually converge to compete to match APM services like sentry etc.

- Splunk’s core is not open source, and is very much geared to large contract enterprise customers. Their main product is also much more geared towards visualisation as opposed to bare log analysis.

When it comes to a logging server/service, you’d consider the following factors:

- Cost. Logging is quite expensive, and the way that Logflare leverages BQ (and in the future, other OLAP engines) cuts down storage costs greatly

- Reliability. The last thing that you would want is for your application to take high load and go down, but you’re unable to debug it because the high traffic led to high log load and subsequently took down your o11y server. Logflare is built on the BEAM and can handle high loads without breaking a sweat. We’ve handled over 10x average load for ingestion spikes and Logflare just chugs along.

- Querying capabilities. Storing logs isn’t enough, you need to effectively debug and aggregate your logs for insights. This incurs both querying costs and additional complexity in the sense that your storage mechanism must be able to handle such complex queries without breaking the bank. Logflare performs optimisations for these, performing table partitioning and caching to make sure costs are kept low. This allows Supabase to expose all logging data to users and perform joins and filters within the Logs Explorer to their hearts’ content.

[nextaccountic]

> - Reliability. The last thing that you would want is for your application to take high load and go down, but you’re unable to debug it because the high traffic led to high log load and subsequently took down your o11y server. Logflare is built on the BEAM and can handle high loads without breaking a sweat. We’ve handled over 10x average load for ingestion spikes and Logflare just chugs along.

Can logflare scale out into multiple containers / vms / machines? Is Supabase currently deploying something like autoscaling with kubernetes or something?

[lyziinc]

For the Logflare infra, we manage a cluster of around 6 VMs. Not a very complex setup, and no need for k8s as we have a monolith architecture. We also don't scale horizontally as much as the cross-node chatter increases with each additional node.

The supabase infra is much more complex, however.

[swyx]

those last 4 bullets were what i needed. thank you!