top | item 42685609

(no title)

PaulWaldman | 1 year ago

> InfluxDB 3.0 open source will be called InfluxDB 3 Core, a recent-data engine persisting Parquet files and enabling queries against the last 72 hours of data. Development of Core will carry on under the permissive MIT or Apache 2 license

> InfluxDB 3 Core gives developers a new tool for time series data management—a high-performance recent-data engine optimized for querying the last 72 hours of data. This focused approach enables Core to deliver exceptional performance for real-time monitoring, data collection, and streaming analytics use cases. By optimizing specifically for this pattern, we’ve achieved query response times under 10ms for last-value queries and under 50ms for hour-long ranges.

The limitation of 72 hours of data is a bit disappointing. There are use cases that don't require 50ms query response times, but instead need longer on-prem storge. InfluxDB 1.X and 2.X supported these well. Would it be possible to reduce query performance and extend data retention?

Is there any guidance on a planned EOL for InfluxDB 1.X and 2.X once Core reaches GR?

discuss

order

pauldix|1 year ago

The data is persisted as Parquet files on object storage (or locally attached disk) and is queryable from any tool that can read Parquet. It isn't evicted by the DB, the 72 hour limit is just what is visible by the running database process.

It's a constraint that we could relax over time, but for now we wanted to limit the scope so we can focus on the recent data. We're also considering a free tier of Enterprise for at-home use cases (i.e. non-commercial hobbyist).

As for EOL on previous versions, we don't have anything planned at the moment. We're partnered with AWS on their hosted versions of InfluxDB 2.x OSS so I expect that to continue for quite some time.

AsianOtter|1 year ago

Adding Merge Tree compaction, a la ClickHouse, should solve this, right?