[annanay]

The semi structured nature of logs works to the advantage of Tempo, because as developers we have the flexibility to log _anything_, high cardinality values like cust-id, request latency, gobble-de-gook .. the equivalent of span tags. Instead of indexing these as tags, we get advance search features through a powerful query language landing in Loki (LogQLv2).

[cbsmith]

But the data starts out structured. It becomes semi-structured when you log it.

I'm telling you, from first hand experience, this does not end well.

There's no reason that your tracing system should not be indexing your tags in an engine that provides advanced search features through a powerful query language.

[pushrax]

I agree, if anything the eventual goal should be to invert it. In applications I work on right now, trace tags contain the richest and best-described request metadata. Tags are indexed differently depending on their cardinality, and there is no cardinality limit.

Tempo's implementation seems pragmatic as a short to medium term solution though. Log engines still have a lot more investment and maturity than trace engines. In my work, even though the trace tags contain the best data quality, the tracing system is currently worse at answering a good deal of my questions. It's simply that Splunk has many tools that work well, and the tracing system is behind.