Quickwit 0.8: Indexing and Search at Petabyte Scale

[dracyr]

Never had the chance to use Quickwit at a $DAYJOB (yet?), but I really appreciate the fact that it scales down quite well too. Currently running it on my homelab, after a number of small annoyances using Loki in a single-node cluster, and it's been working very well with very reasonable resource usage.

I also decide to use Tantivy (the rust library powering/written by Quickwit) for my own bookmarking search tool by embedding it in Elixir, and the API and docs have been quite pleasant to work with. Hats of to the team, looking forward to what's coming next!

[tecleandor]

Ah Loki, I wanted to try it at my homelab bit it wasn't as simple as it says. Now I wanted to try Zincsearch or Openobserve. Have you tried that?

[nextaccountic]

Tantivity is great!

Here is a postgres extension that uses it to provide full text search

https://blog.paradedb.com/pages/introducing_bm25

https://news.ycombinator.com/item?id=37557127

[francoismassot]

Some companies are using it with AWS Lambda to scale to 0.

[godber]

We did some experimentation with quickwit about a year ago, writing about 1m docs/second of data into it for several months. It worked well and was pretty straight forward to learn and operate. If we didn’t also manage our own S3/Ceph it might be a big win, once feature complete. It’s definitely worth a look.

[lrx]

I think you can use quickwit with a self-hosted S3-compatible object store.

[halvorbo]

Amazing to see how far Tantiviy has come. Remember using and making some smaller contributions to this 3 years ago - slop to phrase queries for example. Curious how the design has changed to enable large scale production usage.

[francoismassot]

Thanks! Quickwit is the distributed engine built on top of tantivy, we basically separated compute and storage for search, I wrote this blog post to introduce the architecture: https://quickwit.io/blog/quickwit-101

PS: it’s tantivy!!!

[fulmicoton]

Very valuable contribution!

[up2isomorphism]

13.4GB/s with 200x6 vcpus, gives 11MB/s per core, it is good but hard to say impressive.

[francoismassot]

Building the inverted index is quite CPU-intensive, and we are also merging index files called "splits".

[fulmicoton]

What is your frame of reference?

[arisudesu]

musl support would be highly appreciated.

[fulmicoton]

We used to have one. Maybe we can revive it. What is your use case?