For my use case I wanted to put stuff together without writing too much novel code; zindex uses a SQLite file to store indices and checkpoints. I'm hoping to add json support to zq/zindex, which means a general text search isn't that useful.
Additionally the original log files I run against in my use case are generated upstream of me and are many many multi-gigabyte gzipped log files. I didn't want to add too much extra storage for my logs: the "key" I index on is a tiny part of my log, so the storage space for the index is considerably less than the original text (even accounting for the gzip checkpoints).
mattgodbolt|10 years ago
For my use case I wanted to put stuff together without writing too much novel code; zindex uses a SQLite file to store indices and checkpoints. I'm hoping to add json support to zq/zindex, which means a general text search isn't that useful.
Additionally the original log files I run against in my use case are generated upstream of me and are many many multi-gigabyte gzipped log files. I didn't want to add too much extra storage for my logs: the "key" I index on is a tiny part of my log, so the storage space for the index is considerably less than the original text (even accounting for the gzip checkpoints).