So, what is the closest thing in the open source world to what the author describes? (Setting aside the question of is it right for you, which, of course, depends.)
Any OLAP database that accepts unstructured data can be used in this manner.
The ELK stack is a popular choice, albeit with a focus on search rather than OLAP.
If SaaS is an option, a simple staring point in AWS might be Data Firehose into S3 with Athena. Snowflake can load and query the data too. All of these tools have multiple frontend options with a proportional relationship between cost and user-friendliness.
I honestly just do this in PostgreSQL until my project outgrows it. Create a table with a JSONB column and as few indexes as possible to improve write throughput. Cover a timestamp column with a BRIN index to filter by date range.
Where I work we’ve set up OpenTelemetry SDK in the applications to expose traces, logs and metrics.
Grafana agent as OTEL collector on the application hosts, Grafana Tempo as backend for traces, Loki for logs and Prometheus for Metrics.
The cool thing about Tempo it generates metrics for ingested spans and their labels (spanmetrics) so this allows us to explore “unknown unknowns” as the author calls it in a very cost efficient way.
mdavidn|2 years ago
The ELK stack is a popular choice, albeit with a focus on search rather than OLAP.
If SaaS is an option, a simple staring point in AWS might be Data Firehose into S3 with Athena. Snowflake can load and query the data too. All of these tools have multiple frontend options with a proportional relationship between cost and user-friendliness.
I honestly just do this in PostgreSQL until my project outgrows it. Create a table with a JSONB column and as few indexes as possible to improve write throughput. Cover a timestamp column with a BRIN index to filter by date range.
johnthescott|2 years ago
rtuin|2 years ago
Grafana agent as OTEL collector on the application hosts, Grafana Tempo as backend for traces, Loki for logs and Prometheus for Metrics.
The cool thing about Tempo it generates metrics for ingested spans and their labels (spanmetrics) so this allows us to explore “unknown unknowns” as the author calls it in a very cost efficient way.
viraptor|2 years ago
MuffinFlavored|2 years ago