[haiku2077]

Something I've done in the past is send some logs to BigQuery for cheap mass storage and others to Grafana for fast querying and use in live dashboards. Basically a filter rule in our logging agent to send different events to different destinations. I think with some more hacking I could get both datasources into the same Grafana frontend...

[delusional]

The problem you'll run into, if you haven't already, is that you're now just programming against the log stream. The log stream has become part of the operational interface of the application. Changes to logs, and the setup around them, must now be managed with the same care as all the code in production.

The log filter is all of a sudden part of the application, but managed somewhere else. Everybody is now scared of touching the log lines because who knows what filters have been configured. You suddenly have to debug your log setup, and who logs the decisions about the logs that were filtered?

We already have a place to put that logic. It's the application.