Show HN: Hacker Search – A semantic search engine for Hacker News

[jnnnthnn]

Hi HN!

I'm Jonathan and I built Hacker Search (https://hackersearch.net), a semantic search engine for Hacker News. Type a keyword or a description of what you're interested in, and you'll get top links from HN surfaced to you along with brief summaries.

Unlike HN's otherwise very valuable search feature, Hacker Search doesn't require you to get your keywords exactly right. That's achieved by leveraging OpenAI's latest embedding models alongside more traditional indexes extracted from the scraped and cleaned up contents of the links.

I think there are many more interesting things one could build atop the HN dataset in the age of LLMs (e.g. more explicitly searching for technical opinions, recommending stories to you based on your interests, and making the core search feature more useful). If any of those sound interesting to you, head over to https://hackersearch.net/signup to get notified when I launch them!

Note: at least one person has built something similar before (https://news.ycombinator.com/item?id=36391655). Funnily enough, I only found out about this through my own implementation, and I based on my testing, I think Hacker Search generally performs better when doing keyword/sentence searches (vs. whole document similarity lookup), thanks to the way the data is indexed.

[v1sea]

Can you go into more detail on how it works or provide some references for articles/papers on implementing a system like this? Is it just RAG?

Testing it out, I'd say the results for "graph visualization" are focused if a bit incomplete. So to me it has high precision, but lower recall.

I don't see this searching comments. That could be a nice extension. Thanks for sharing.

[jnnnthnn]

It is mostly RAG, although I suppose that doesn't say much about the system: one thing I've found is that the way you clean and process the data substantially changes the quality of the results. I'll write a little blog post sharing some of the learnings!

If you feel up for it, you should share your email in the righthand "Unhappy with your results?" widget. My plan is to manually look into the disappointing searches and follow-up with better results for folks, in addition to fixing whatever can be fixed.

Agreed re: searching comments (which it indeed currently doesn't do).

[isoprophlex]

There's at least three of us ;) I built https://searchhacker.news

Loving your LLM generated summaries! Very nice user experience to see at a glance what a hit is about. Also your back button actually works, haha.

Well done!

[jnnnthnn]

Thank you, and likewise!

[codethief]

Nice work!

Unfortunately, though, it didn't find what I was looking for in the following real-word test case: The other day I tried to remember the name of an SaaS to pin/cache/ back up my apt/apk/pip dependencies, which I think I had read about either here[0] or here[1]. After quite a bit of time and some elaborate Google-fu, I did end up finding those HN threads again. However, they did not show up on hackersearch.net for me, neither when entering the service's name nor when I searched for "deterministic Docker builds" or "cache apt apk pip dependencies".

[0]: https://news.ycombinator.com/item?id=39684416

[1]: https://news.ycombinator.com/item?id=39723888

[jnnnthnn]

Thanks for the feedback! I made a cutoff at stories with <50 points when ingesting content, and it looks like that cut off what you were looking for. Definitely not great.

I'm planning to fix that in short order, feel free to sign up at https://hackersearch.net/signup if you care to receive an update when that goes live!

[awendland]

Following @isoprohplex, I'll be the fourth comment to say I also built a variant of this: https://hnss.alexwendland.com/

I built mine on top of an RSS feed I generate from Hacker News which filters out any posts linking to the top 1 million domains [1] and creates a readable version of the content. I use it to surface articles on smaller blogs/personal websites—it's become my main content source. It's generated via Github Actions every 4 hours and stored in a detached branch on Github (~2 GB of data from the past 4 years). Here's an example for posts with >= 10 upvotes [2].

It only took several hours to build the semantic search on top. And that included time for me to try out and learn several different vector DBs, embedding models, data pipelines, and UI frameworks! The current state of AI tooling is wonderfully simple.

In the end I landed on (selected in haste optimizing for developer ergonomics, so only a partial endorsement):

  - BAAI/bge-small-en as an embedding model
  - Python with
    - HuggingFaceBgeEmbeddings from langchain_community for creating embeddings
    - SentenceSplitter from llama_index for chunking documents
    - ChromaDB as a vector DB + chroma-ops to prune the DB
    - sqlite3 for metadata
    - FastAPI, Pydantic, Jinja2, Tailwind for API and server-rendered webpages
  - jsdom and mozilla-readability for article extraction

I generated the index locally on my M2 Mac which ripped through the ~70k articles in ~12 hours to generate all the embeddings.

I run the search site with Podman on a VM from Hetzner—along with other projects—for ~$8 / month. All requests are handled on CPU w/o calls to external AI providers. Query times are <200 ms, which includes embedding generation → vector DB lookup → metadata retrieval → page rendering. The server source code is here [3].

Nice work @jnnnthnn! What you built is fast, the rankings were solid, and the summaries are convenient.

[1] https://majestic.com/reports/majestic-million

[2] https://github.com/awendland/hacker-news-small-sites/blob/ge...

[3] https://github.com/awendland/hacker-news-small-sites-website...

[jasonjmcghee]

I would strongly consider incorporating a hybrid search strategy (keyword or bm25 etc) as currently a number of searches I tried resulted in rather surprising results

[jnnnthnn]

Fun! Thanks for sharing! It's super fast and the bias toward smaller websites really does surface interesting things. Very reminiscent of the Kagi small web site, also!

[ofermend]

Super cool.

Clearly many of us see the need here. I have also been working on a similar demo: https://search-hackernews.vercel.app/ 1. Stack: Vectara for RAG, Vercel for hosting 2. Results show the main story and top 3-4 comments from the story 3. Focused mostly on the search aspect - so if you click it redirects you to the HN page itself. No summaries although it'd be easy to add.

Would love to get some feedback and any suggestions for improvement. I'm still working on this as a side project.

Example query to try: "What did Nvidia announce in GTC 2024?" (regular HN search returns empty)

[rdli]

This is cool! I've been trying out bits & pieces of the RAG ecosystem, too, exploring this space.

Here's a question for this crowd: Do we see domain/personalized RAG as the future of search? In other words, instead of Google, you go to your own personal LLM, which has indexed all of the content you care about (whether it's everything from HN, or an extra informative blog post, or ...)? I personally think this would be great. I would still use Google for general-purpose search, but a lot of my search needs are trying to remember that really interesting article someone posted to HN a year ago that is germane to what I'm doing now.

[jnnnthnn]

I definitely think there are opportunities to provide more useful & personalized search than what Google offers for at least some queries.

Quality aside, I think the primary challenge is in figuring out the right UX for delivering that at scale. One of the really great advantages of Google is that it is right there in your URL bar, and that for many of the searches you might do, it works just fine. Figuring out when it doesn't and how to provide better result then seems like a big unsolved UX component of figuring out personalized search.

[tinyhouse]

Nice work! I'm sure you know that you can also search Google with site:https://news.ycombinator.com. If I were you I would probably think of a niche where one can get better results by searching HN and other relevant data sources. Another suggestion is not about the search so much but about the UI. One of the worse things about sites like HN is that it's really hard to follow a long thread. If you can fix that by doing some data transformation and build a nice UI for search results, it'd be pretty useful. Good luck!

[jnnnthnn]

Thanks for the feedback!

One big distinction with the "site:https://news.ycombinator.com" hack is that the search on Hacker Search directly runs against the underlying link's contents, rather than whatever happens to be on HN. We also more directly leverage HN's curation by factoring in scores.

Appreciate your suggestions; will look into building those!

[manca]

I love projects like this. It shows the true potential of what LLMs and RAG can unlock. Imagine applying the same method on the actual content within the threads and extract the sentiment, as well as summarize the key points of a particular thread -- the options are limitless.

My only piece of advice, though: try to do the reranking using some other rerankers instead of an LLM -- you'll save both on the latency AND the cost.

Other than that, good job.

[jnnnthnn]

Thanks! I tried a few other approaches and found the LLM results were overall better (latency and cost aside). Maybe that should be an option made available to users though...

[hubraumhugo]

Great to see cool stuff being built on top of HN data! Many of us rely on this platform as one of the primary sources of information.

> recommending stories to you based on your interests

I built this as a service that monitors and classifies HN stories based on your interests (solved my FOMO): https://www.kadoa.com/hacksnack

[jnnnthnn]

Nice!

[BohuTANG]

Nice work. It's recommended to start with a keyword search (including titles, comments, and the keyword itself) before using an LLM-based search. For example, if I search for 'databend' on https://hackersearch.net/search?q=databend&period=last-3-yea..., it defaults to an LLM-based search. However, I prefer using a simple keyword search instead.

[jnnnthnn]

There's a place for both, yes! I think Hacker Search should offer each (https://news.ycombinator.com/item?id=40241332) and ideally intelligently figuring out what to do based on your query.

[pjot]

I’ve done something very similar! But with duckDB as a vector store/engine.

https://github.com/patricktrainer/duckdb-embedding-search

[jasonjmcghee]

Have you tried the new first class vss support from DuckDB?

[jnnnthnn]

Sweet! I'll try running it tonight.

[curious_cat_163]

So, I played with it some more. I think that this is a good starting point. You can tune various parameters for what you have indexed and it will get better. I am sure it will evolve in interesting directions from here.

> e.g. more explicitly searching for technical opinions...

Yes, please! I would love to be able to search for strongly held opinions by folks who _know_ what they are talking about.

> recommending stories to you based on your interests...

I am curious how, in principle, you would you do that? Where do you think the signal that indicates my "interest" lies?

[jnnnthnn]

Thank you!

To learn your interests we'd at a minimum need to know what HN stories you tend to click or comment on, e.g. by a different reader view or using a browser extension. Presumably your comments and submissions could provide useful signal as well :)

[fuzzfactor]

This is good to have!

Apparently the old Algolia search has not been accessible around the world for a few months at least.

[simonw]

It looks like you generated an LLM summary of every page you indexed. What model did you use for that, and how much did it cost?

[jnnnthnn]

Hi Simon! Big fan of your blog.

I actually generate two summaries: one is part of the ingestion pipeline and used for indexing and embedding, and another is generated on-the-fly based on user queries (the goal there is to "reconcile" the user query with each individual item being suggested).

I use GPT-3.5 Turbo, which works well enough for that purpose. Cost of generating the original summaries from raw page contents came down to about $0.01 per item. That could add up quickly, but I was lucky enough to have some OpenAI credits laying around so I didn't have to think much about this or explore alternative options.

GPT-4 would produce nicer summaries for the user-facing portion, but the latency and costs are too high for production. With GPT-3.5 however those are super cheap since they require very few tokens (they operate off of the original summaries mentioned above).

Worth noting that I've processed stories by score descending, and didn't process anything under 50 points which substantially reduced the number of tokens to process.

[avereveard]

doesn't seem like it's semantic https://hackersearch.net/search?q=solutions+for+postgres+clu... sure result are about postgres but none was relevant to clustering solutions

[jnnnthnn]

I suspect this is probably because of a bit of a bias in the indexed dataset: at present, the indexed stories tend to bias toward high-scores ones, and at a glance I don't see that many stories about Postgres clustering in that distribution.

[levkk]

Pretty cool. A little slow for a search engine, have you tried in-database embeddings for semantic search like PostgresML?

[jnnnthnn]

Thanks for trying it out!

Agreed it could be faster for uncached queries. The embeddings retrieval itself is actually pretty fast (uses pgvector). However, I found that having a LLM rerank results + generate summaries related to the search query made results more useful, which is what accounts for much of the latency.

Maybe I should make that a user-customizable setting!

[HanClinto]

Nice work! Love seeing RAG work get developed!

What about using the embeddings for nearest-neighbor search for similar articles? I.E., for any given article, can you use the embeddings of an article to run a search, rather than encoding my query? That would let me find similar / related articles much more easily.

[jnnnthnn]

Thanks!

Yup, totally feasible. I might add that!

[xwowsersx]

Thank you so much for this; it's very nicely done. I'm finding exactly what I'm looking for on topics I'm eager to explore further through conversation. Bookmarked. Please let us know if and how we can help.

[jnnnthnn]

Thank you!

[Fudgel]

Pretty neat. I think I found a small UI glitch: If I search and then click the browser back button when I'm on the results page, the search button on the home page shows the spinning icon.

[jnnnthnn]

Snap! I can't repro this. Which browser are you using?

[Scene_Cast2]

What's your stack like?

[jnnnthnn]

- TypeScript

- Next.js

- OpenAI's embeddings and GPT endpoints

- Postgres with pgvector (on neon.tech)

- Tailwind

- tRPC

- Vercel for web hosting

- Google Cloud products for data pipelines (GCS, Cloud Tasks)