Launch HN: Bloop (YC S21) – Code Search with GPT-4

[louiskw]

Hey HN, we’re the cofounders of bloop (https://bloop.ai/), a code search engine which combines semantic search with GPT-4 to answer questions. We let you search your private codebases either the traditional way (regex or literal) or via semantic search, ask questions in natural language thanks to GPT-4, and jump between refs/defs with precise code navigation. Here’s a quick demo: https://www.loom.com/share/8e9d59b88dd2409482ec02cdda5b9185

Traditional code search tools match the terms in your query against the codebase, but often you don’t know the right terms to start with, e.g. ‘Which library do we use for model inference?’ (These types of questions are particularly common when you’re learning a new codebase.) bloop uses a combination of neural semantic code search (comparing the meaning - encoded in vector representations - of queries and code snippets) and chained LLM calls to retrieve and reason about abstract queries.

Ideally, a LLM could answer questions about your code directly, but there is significant overhead (and expense) in fine-tuning the largest LLMs on private data. And although they’re increasing, prompt sizes are still a long way off being able to fit a whole organisation’s codebase.

We get around these limitations with a two-step process. First, we use GPT-4 to generate a keyword query which is passed to a semantic search engine. This embeds the query and compares it to chunks of code in vector space (we use Qdrant as our vector DB). We’ve found that using a semantic search engine for retrieval improves recall, allowing the LLM to retrieve code that doesn’t have any textual overlap with the query but is still relevant. Second, the retrieved code snippets are ranked and inserted into a final LLM prompt. We pass this to GPT-4 and its phenomenal understanding of code does the rest.

Let’s work through an example. You start off by asking ‘Where is the query parsing logic?’ and then want to find out ‘Which library does it use?’. We use GPT-4 to generate the standalone keyword query: ‘query parser library’, which we then pass to a semantic search engine that returns a snippet demonstrating the parser in action: ‘let pair = PestParser::parse(Rule::query, query);’. We insert this snippet into a prompt to GPT-4, which is able to work out that pest is the library doing the legwork here, generating the answer ‘The query parser uses the pest library’.

You can also filter your search by repo or language - What’s the readiness delay repo:myApp lang:yaml. GPT-4 will generate an answer constrained to the respective repo and language.

We also know that LLMs are not always (at least not yet) the best tool for the job. Sometimes you know exactly what you’re looking for. For this, we’ve built a fast, trigram index based regex search engine based on Tantivy. Because of this, bloop is fast at traditional search too. For code navigation, we’ve built a precise go-to-ref/def engine based on scope resolution that uses Tree-sitter.

bloop is fully open-source. Semantic search, LLM prompts, regex search and code navigation are all contained in one repo: https://github.com/bloopAI/bloop.

Our software is standalone and doesn’t run in your IDE. We were originally IDE-based but moved away from this due to constraints on how we could display code to the user.

bloop runs as a free desktop app on Mac, Windows and Linux: https://github.com/bloopAI/bloop/releases. On desktop, your code is indexed with a MiniLM embedding model and stored locally, meaning at index time your codebase stays private. Indexing is fast, except on the very largest repos (GPU indexing coming soon). ‘Private’ here means that no code is shared with us or OpenAI at index time, and when a search is made only relevant code snippets are shared to generate the response. (This is more or less the same data usage as Copilot).

We also have a paid cloud offering for teams ($12 per user per month). Members of the same organisation can search a shared index hosted by us.

We’d love to hear your thoughts about the product and where you think we should take it next, and your thoughts on code search in general. We look forward to your comments!

[tkiolp4]

I don’t understand these new saas that rely on GPT. It surely is my lack of understanding, but since the interface to GPT is mainly text: why any company would pay for N different GPT saas (e.g., one for searching for code, another for searching for documentation on confluence, another for allowing your business analysts with their queries, etc.)?

Wouldn’t it make more sense for any company to have only ONE interface to use GPT that is wired with some (or all) parts of the company digital assets? Text is already flexible enough to allow such a universal interface.

[tedsanders]

To me, your question seems to boil down to the economics of vertical integration.

Two thoughts:

First, GPT is all marginal cost, no fixed cost. So the marginal economics of 1 super app vs 10 specialized mini apps is roughly the same.

Second, imagine the same argument applied to a utility like electricity. "Why would any company pay for lightbulbs from one company, HVAC from a second company, and appliances from a third company?"

Early on, electricity was complicated and you might actually purchase all of these from the Edison Illuminating Company. But there are returns to specialization, and today the winning formula is different companies that specialize in each. The company best at manufacturing lightbulbs is not necessarily the company that is best at manufacturing washing machines.

Similarly, you could ask: "Why would any company pay for a laptop from one company, an operating system from a second company, and office apps from a third company?"

In some cases it may make sense to vertically integrate (e.g., Apple is happy to sell you a combined laptop + OS + Pages/Numbers), but in many cases the specialized players still do fine (e.g., you might buy a Dell laptop, a Microsoft OS, and Notion/Sheets).

So I think it's very much an open question as to whether the winning approach will be a Swiss army knife (single product) or a toolkit (multiple products).

[hn_throwaway_99]

The point is that the interface to Bloop (or whatever 3rd party SaaS product) is just plain text. In programming parlance I would call this a facade [1] on top of GPT-4. The benefit is that you, the end user, can just continue typing things like "Where do we create our database pool?" in a search box, but under the covers Bloop can use GPT-4, but then eventually upgrade to GPT-5, GPT-6, or some other much better currently-super secret competitor version.

1. https://en.wikipedia.org/wiki/Facade_pattern

[s1k3s]

Investors want AI. Do you see how crappy their landing pages look? This one is pretty decent compared to what we've seen showcased here in the past months. It's as if someone at YC told them to ship it asap, with no consideration for anything other than "slap AI on it".

[101008]

I would like also to understand the amount of startups funded in 2021 (or around) that relies on GPT-{n} now. They pivoted or found out that they solution was much easier with GPT-{n}?

Also, why as a customer would care if you use GPT or something else? It is because it is a buzzword?

[jstummbillig]

> why any company would pay for N different GPT saas

Simple: You do it because you pay for what SaaS does, not what GPT does. A user is not interested in postgres or nginx, even though they are the tools used to build the tool they care about and pay for. If what the new tool uniquely does is not adding enough value for enough users, it's going to fail.

[jacobr1]

Some of these app are too niche, but maybe not all of them. There have been many attempts at a "unified enterprise search app" for example, but they all have generally failed. Maybe customer indexing + GPT + form fact (slackbot, gui, etc ..) is enough for a company. Vanilla ChatGPT probably isn't going do that.

Or consider codex low-code type use cases. Yes you could generate generic code, but integrated into some kind of platform that knows what APIs your company uses (internal cataglog) and some kind of IAM/Auth platform, PAAS to host the results integrated into a single tool might make sense, especially if the prompts are already injecting the boilerplate about how to interact with the ecosystem.

Or consider where just adding some kind of generative feature is just a _feature_ of broader product.

[karmasimida]

You are not wrong on this one.

The moat they are going to build is going to incredibly ... low.

The main intelligence is going to be provided by GPT models.

[boplicity]

There's a lot of work you could theoretically put into automating many of the queries you would want to do with a GPT chat interface.

For example, I'm no a medical diet and need a meal plan. This diet has quite a few restrictions.

I could set up automated queries that produce a weekly meal plan, and grocery list, and also checks all of the ingredients against the allowable / not allowable food lists. This could include multiple queries, one to get an initial response, another to validate it, etc. Everything could be organized nicely, so I wouldn't have to input any text to get what I need -- I'd just click buttons. There's lots of ways this could be further customized to be more useful, such as saving recipes, generating new recipes based on old favorites, generating recipes based on foods already at home, etc. (Heck, you could wire it up to a camera in the fridge.)

Anyways -- text input to get text output is the most basic form of this technology. There's a lot you can do with such text to make it easier to interact with for specific use cases.

[louiskw]

Most of bloop's interface is search results, which are driven by the LLM and have highlighted line ranges and go-to-ref/def. Down the line, many other LLM powered tasks will need specialised interfaces too, like showing a diff or a git history tree.

This is quite different from a GPT tool for most other jobs, and I think having granular control of the interface layer certainly helps us ship a better product.

That's not to say everything needs to be an app. If the output is just conversational text, it can and probably will be some kind of 'Alexa skill' like plugin.

[teaearlgraycold]

I think you could say similar things about cloud computing, but we pay for different apps that are all ultimately startups re-selling AWS resources.

[coffeebeqn]

If you can give it proprietary training that takes nontrivial work then I don’t see why it couldn’t be worthwhile. Collecting, curating, cleaning up, manipulating data to make the best model of X would easily be worthwhile to make and to then purchase for hundreds of companies.

If it’s just appending a prompt to chatgpt then it’s certainly useless

[menzoic]

It seems you're doing the opposite of "missing the forest for the trees". You see the forest (the app) but you're not seeing the individual trees.

The answer to why companies don't have one interface with GPT is that they would still need to invest time and money into building the application that uses GPT. Checkout the bloop source code to get an idea of what's needed https://github.com/bloopAI/bloop there are also many complex things to deal with. You can't just shove an entire codebase into GPT-4, it has limited ability to track context. The codebase needs to be indexed and stored in a way that sematic search can be done on it very quickly. Every app has their own unique challenges towards getting the data into GPT and making it fast.

TL;DR companies are buying apps that use GPT-4 API, not just access to GPT-4. They'd rather buy the apps instead of building and maintaining them.

[li4ick]

Ok, I'll be the bad guy and say that the linked demo on loom is just not impressive at all. It's just 10x slower than what a simple regex query would've answered. Again, I'll be the bad guy in this comment section, but ML-based search methods for code are just not that useful. They do sound nice for non-coders, that's about it. I also feel like I'm qualified to say this because I have first-hand experience building code search tooling at scale.

[boplicity]

> nice for non-coders, that's about it.

I'm not a coder but I code; not sure if that makes sense. I sort of know a bit of regex, but find it utterly painful and not worth the time.

I'm an amateur, effectively, with little time. I like AI tools for coding, because I can input a request, and get sample code. I know enough to be able to read most of the code that I get back, but not enough to be able to easily write such code on my own.

These types of tools, in my opinion, have the potential to make people like me, and even people who are much less knowledgable than me, productive programmers. This could be transformative.

Sure, we won't be as good or productive as real programmers, but that's beside the point.

In other words, what you're referring to with "that's about it" could still transform quite a few lives.

[intalentive]

Where to go next? A Toolformer that calls an AST parser. In VS Code I am running Copilot and IntelliSense side by side. Two competing tab completions -- ridiculous. They should be integrated. I want a Copilot that listens to the interpreter / compiler, so that it only makes correct suggestions.

[cjonas]

I can't stand how co-pilot hijacks the standard code completion, often overriding the typescript completion I actually want or generating something distracting.

There should be a way to only run completions when prompted.

Please upvote this issue if you run into the same problems: https://github.com/community/community/discussions/9817

[nico]

> I want a Copilot that listens to the interpreter / compiler, so that it only makes correct suggestions.

And that builds a test first. Then iterates on the new code against the test until it passes.

ChatTDD?

[renewiltord]

Very interesting. I want you to know that

> ‘Private’ here means that no code is shared with us or OpenAI at index time, and when a search is made only relevant code snippets are shared to generate the response. (This is more or less the same data usage as Copilot).

is the reason I went from "ha, cool tool" to "okay, let me go download it".

Quite surprised that you don't actually have a login wall to download. Missing an opportunity to track downloads, upsell to paid plan, etc. etc. imho.

[evrydayhustling]

I'm interested why this made a difference to you. If you are concerned about sharing code with Bloop/OpenAI, isn't sharing code during use as bad as sharing it at index time? The more you use it, the more of your code will be exposed...

[purplecats]

at the cost of fewer downloads

[dazbradbury]

"Precise code navigation in 10+ languages helps you move quickly through refs and defs"

I can't find any mention of which languages are supported - can anyone point me in the right direction?

[bebrws]

I tried to implement this using OpenAI's embeddings then using cosign similarity between produces vectors instead of the vector DB ( as shown in OpenAI's example code here ). I do the same thing where I take the highest ranking code snippets (vectors) and include them in a prompt to ChatGPT with an original prompt. My code is a mess since I just hacked it together after a long day of work but its a small file. Like 100 lines.

I used TreeSitter which I thought was pretty awesome though because it allows for parsing a TON of different languages. I had to parse the languages to create the different code snippet strings. I don't want to create a code snippet of half a function for example..

So TreeSitter parses the code into an AST and I send each different AST node to OpenAI to get the vector (I optimized this so multiple nodes of the same AST type are combined). Send the prompt to OpenAI to get a vector. Find the most similar code snippets to the prompt and include them at the top of a prompt to ChatGPT.

This is the same idea right? If anyones interested it can be found here: https://bbarrows.com/posts/using-embeddings-ada-and-chatgpt-...

https://github.com/bebrws/openai-search-codebase-and-chat-ab...

[ggordonhall]

We currently support Go, JS/TS, JSX/TSX, Python, Rust, C#, Java, C/C++

[dabei]

This is awesome and has huge potential to improve developer productivity! The only thing that it gives me pause is that it requires a very broad Github authorization. Do you really need to "be able to read and write all public and private repository data"?

[x-complexity]

> This is awesome and has huge potential to improve developer productivity! The only thing that it gives me pause is that it requires a very broad Github authorization. Do you really need to "be able to read and write all public and private repository data"?

Compared to a regular search engine, the permissions required are pretty much the same. Both this & regular search engines need to go through a repo's codebase to be even able to give results in the first place.

Privacy-wise, they could probably make it better by requiring each repo to be approved before they can be searched, but that would make for a more friction-laden developer UX. The broad permissions are likely just a consequence of not wanting to ask the user every time a new repo is to be searched through.

[louiskw]

For the desktop version we implemented GitHub OAuth using the device flow, so you can hold credentials on your local device. The tradeoff is there's no granular control of permissions, it has whatever access your account has.

On bloop cloud we use the GitHub App permission system which is more granular and only request read access.

[bjtitus]

This looks great! Is there documentation in the repository that goes into greater detail about the query process? It might be nice to add some diagrams or other explanations about the architecture there.

[knexer]

Query flow looks to be roughly along the lines of:

- llm rephrases the last conversation entry into a standalone, context-free search query

- rephrased query is embedded, top-k results retrieved from the vector db

- llm selects a top-1 winner from the top-k results

- llm answers the question given conversational context and the top-1 code search result

(from https://github.com/BloopAI/bloop/blob/8905a36388ce7b9dadaedf...)

[duped]

How do you manage "hallucinations?" One of the things I've noticed about using ChatGPT for programming tasks is that it will happily give you garbage.

[Alifatisk]

ChatGPT to my understanding is only meant for one purpose, to be extremely good at conversing. That’s it! It should be so good at it that it almost convinces you.

OpenAi’s codex and the newer models should be used for actual coding related prompts.

[elanzini]

Congrats on the launch!

If multiple pieces of code from different files are being referenced in the response, it would be nice to have clickable refs that take you to that piece of code in the repo.

[ggordonhall]

Yes that's a great point. At the moment we in the 'Results' section we display the code snippets returned by the semantic search engine. These don't always line up with the code that GPT is referring to because we include as much of the file that the snippet is from in the prompt.

We're working on a new interface to make this clearer, where we'll display the lines of code that GPT has referenced in its answer.

[btbuildem]

Tangential note to the person doing the video voiceover: pls process the audio a bit (high-pass filter at least?), or use higher a quality headset.

[louiskw]

Apologies, was very hastily recorded in between about a million other pre-launch tasks.

I thought my Sennheiser Momentum 4's would do a better job, but even they were no match for a glass call booth.

[avinassh]

what is a high-pass filter?

[swyx]

congrats on launching!

i honestly feel like a bad user for this but i have yet to adopt a semantic search engine for code for some reason, respite codeium and sourcegraph also offering more advanced code search thingies. any ideas on how to break force of habit?

[sqs]

The burden is on those products to compel you, not you to break a force of habit. :)

Code search historically has been adopted by <10% of devs, although it's usually binary within each company, with equilibriums at both ~1% adoption or >80%+ adoption. My model of LLMs applied to code search is that they make it so even a first-time user can use (easily, via the LLM) features that were previously only accessible to power users: regexps, precise code navigation, large-scale changes/refactors, diff search, other advanced search filters, and all kinds of other code metadata (ownership, dep graph, observability, deployment, runtime perf, etc.) that the code search engine knows. The code search engine itself is just used under the hood by the LLM to answer questions and write great code that uses your own codebase's conventions.

I bet you (and the ~90% of devs who don't use code search today) will be compelled when you see and use /that/.

[louiskw]

That's the million $ question. Autocomplete (Copilot) is install and forget. We don't have that luxury, search has to give you that 'aha' moment the first time you use it.

For one, having the engine respond in natural language makes a big difference. The last generation of semantic code search (we built one) used transformers to retrieve code, but as a user you'd still have to read through an entire code chunk to find the answer.

Also, LLMs (probably starting with GPT-4) can now reason properly. The capability to make a search, read the results and execute an entirely new search based on its reasoning skills, and do this iteratively, until it finds the answer, is a huge jump from just using semantic search on its own.

[amcaskill]

Very excited to give this a spin. Congratulations!

Any plans to enable the tool to edit code for me? For example if I ask for a suggestion on how to fix something, could I just “accept” the suggestion and have the changes made to the file?

[mehlmao]

Do Bloop searches also include the license of the code it returns? For example, if a search returns code copied from a GPL 3 project, will it let users know they need to comply with the GPL?

[samwillis]

Unless I'm mistaken Bloop is about searching your own codebase, not public code. It would never bring back code that isn't already in your codebase.

[avinassh]

Instead of asking for read permissions to repos, it is asking me to login to Github, via Bloop. Why such setup was done? Isn't just read access enough?

[Freddie111]

Congrats on the launch! How do you plan to solve distribution if you're not integrating with IDEs anymore?

[meken]

This is awesome. I wanted this yesterday

[ccQpein]

Very cool stuff, I was wondering if there is any tool like this days ago. Cannot wait to test it.

[sorokod]

Would be interesting to see precision and recall numbers - do you have any?

[andre-z]

Congrats on the launch! And thanks for using Qdrant ;)

[shashanoid]

Cool, kinda like buildt.ai I used before. That cannot inject code. Can you do that? You're in YC I'd expect your products to better than what's out there.