WingNews logo WingNews GitHub [2]
top | item 35279155

Launch HN: Helicone.ai (YC W23) – Open-source logging for OpenAI

166 points| justintorre75 | 3 years ago

Hi HN - Justin, Scott, and Barak here. We're excited to introduce Helicone (https://www.helicone.ai) an open-source logging solution for OpenAi applications. Helicone's one-line integration logs the prompts, completions, latencies, and costs of your OpenAI requests. It currently works with GPT, and can be integrated with one line of code. There’s a demo at https://www.helicone.ai/video.

Helicone's core technology is a proxy that routes all your OpenAI requests through our edge-deployed Cloudflare Workers. These workers are incredibly reliable and cause no discernible latency impact in production environments. As a proxy, we offer more than just observability: we provide caching and prompt formatting, and we'll soon add user rate limiting and model provider back off to make sure your app is still up when OpenAI is down.

Our web application then provides insights into key metrics, such as which users are disproportionately driving costs and what is the token usage broken down by prompts. You can filter this data based on custom logic and export it to other destinations.

Getting started with Helicone is quick and easy, regardless of the OpenAI SDK you use. Our proxy-based solution does not require a third party package—simply change your request's base URL from https://api.openai.com/v1 to https://oai.hconeai.com/v1. Helicone can be integrated with LangChain, LLama Index, and all other OpenAI native libraries. (https://docs.helicone.ai/quickstart/integrate-in-one-line-of...)

We have exciting new features coming up, one of which is an API to log user feedback. For instance, if you're developing a tool like GitHub Copilot, you can log when a user accepted or rejected a suggestion. Helicone will then aggregate your result quality into metrics and make finetuning suggestions for when you can save costs or improve performance.

Before launching Helicone, we developed several projects with GPT-3, including airapbattle.com, tabletalk.ai, and dreamsubmarine.com. For each project, we used a beta version of Helicone which gave us instant visibility into user engagement and result quality issues. As we talked to more builders and companies, we realized they were spending too much time building in-house solutions like this and that existing analytics products were not tailored to inference endpoints like GPT-3.

Helicone is developed under the Common Clause V1.0 w/ Apache 2.0 license so that you can use Helicone within your own infrastructure. If you do not want to self-host, we provide a hosted solution with 1k requests free per month to try our product. If you exceed that we offer a paid subscription as well, and you can view our pricing at https://www.helicone.ai/pricing.

We're thrilled to introduce Helicone to the HackerNews community and would love to hear your thoughts, ideas, and experiences related to LLM logging and analytics. We're eager to engage in meaningful discussions, so please don't hesitate to share your insights and feedback with us!

72 comments

order
[+] ianbicking|3 years ago|reply
I have _specifically_ thought of writing something just like this, so it's awesome to see it!

One thing I would really like to be storing with my requests is the template and parameters that created the concrete prompt. (This gets a little confusing with ChatGPT APIs, since the prompt is a sequence of messages.) Custom Properties allow a little metadata, but not a big blob like a template. I see there's a way to have Helicone do the template substitution, but I don't want that, I have very particular templating desires. But I _do_ want to be able to understand how the prompt was constructed. There's probably some danger on the client side that I would send data that did not inform the prompt construction, and balloon storage or cause privacy issues, so there is some danger to this feature.

Backoffs and other rate limiting sounds great. It would be great to put in a maximum cost for a user or app and then have the proxy block the user once that was reached as a kind of firewall of overuse.

Note your homepage doesn't have a <title> tag.

[+] justintorre75|3 years ago|reply
This is so great! Thanks for the feedback. This is something we have thought a lot about, and really want to make sure we are adding the "correct" features that don't disrupt the user experience.

I kicked off a thread here https://github.com/Helicone/helicone/discussions/164 and would love your input!

P.S. that for the <title> note

[+] ianbicking|3 years ago|reply
I'll note the jump in cost from Free to Starter will keep me from using this for my own projects, 1000 requests just isn't that much, and if I'm doing something more serious then 50,000 requests also doesn't seem like a lot.
[+] smithclay|3 years ago|reply
Congrats on the launch. Like a lot of the people right now, am doing some side project with OpenAI + LangChain and immediately got some value out of this, specifically:

* When your chains get long/complex enough in LangChain, it's really hard to understand from debug output what's final prompt that actually being sent, how much it costs, or catching runaway agents. This pretty much solves that for me.

As a "prompt developer", one thing that'd be incredibly useful is a way to see/export all of my prompts and responses over time to help me tune them (basically a "Prompt" button in the left nav).

Congrats on the launch. So nice to see a tool in the space that lets you get up and running in 4 minutes.

[+] NetOpWibby|3 years ago|reply
Y'all committed an env file https://github.com/Helicone/helicone/pull/136/files O___O
[+] justintorre75|3 years ago|reply
Yeah this was added by accident, luckily we have pretty good management of local development and production testing, so these keys were not production keys
[+] pjot|3 years ago|reply
lgtm.

A global .gitignore is one of the first things I install on a new machine - and then I never think of it again.

  $ git config --global core.excludesfile ~/.gitignore
[+] berkle4455|3 years ago|reply
I mean they launched a YC-funded company that will soon be replaced by a "Usage" dashboard and some admin settings over at OpenAI.
[+] juniordev7|3 years ago|reply
It's a ritual at my prev workplace. They didn't do anything even after i warned about it. Of course they got hacked..
[+] kanyethegreat|3 years ago|reply
Lol, has their SB creds. Someone could do a bunch of stuff with that
[+] StablePunFusion|3 years ago|reply
The entire company is based around the idea of providing metrics for one closed-source platform's API? Or is the "OpenAI application observability" just one part of what the company does? Otherwise it seems like taking the "all eggs in one basket" to the extreme.
[+] justintorre75|3 years ago|reply
OpenAI is just the start to capture most of the market and iterate on some ideas first :). We are adding multiple providers soon. ~Stay tuned~
[+] quadcore|3 years ago|reply
"all eggs in one basket" to the extreme.

You couldn't describe better what a startup must do.

[+] version_five|3 years ago|reply
To be fair, there are companies based on providing metrics for AWS, e.g. Vantage, so it's not surprising to see VCs making bets in this area
[+] gregschlom|3 years ago|reply
"Solve one pain point for OpenAI's users, get some nice traction, get acquired by OpenAI in a couple of months" is probably the plan here. Not a bad one to turn a quick profit.
[+] snacktaster|3 years ago|reply
It seems pretty brittle to me also. And their solution is to actually _proxy_ OAI's API wholesale which doesn't seem like a very good idea and might not even be permitted.
[+] throwaway280382|3 years ago|reply
I guess they are betting that openAI will be such a big platform that many companies will need this capability. Not a bad idea I guess
[+] VWWHFSfQ|3 years ago|reply
Does this run afoul of OpenAI's terms of of service in any way? Using a commercial proxy/broker like this to access their API services instead of using directly.
[+] 3np|3 years ago|reply
More direct concern: PII leaking from end-users will be a problem if the usage of this service isn't made clear and consented with from each individual. I really think the responsible thing by providers like OP is to remind their customers about this.

If you have end-users in for example EU then integrating a service like Helicone would usually require new informed consent to be legal.

[+] justintorre75|3 years ago|reply
We see a lot of applications on the market essentially "proxying" traffic to OpenAI, moving to a non-proxy implementation is not too hard if they have any issues with us doing this.
[+] olliepop|3 years ago|reply
The reality is that virtually all of the public AI tech available right now is from OpenAI. Over time as more models are commercialised and generally available, it's probable that Helicone will serve them too.

Congrats Justin and team! Excited for you.

[+] social_quotient|3 years ago|reply
Could be cool if you offer quota enforcement. So if something goes rogue on the application side, your layer could not only offer observability but a ceiling to protect against unexpected cost overruns.
[+] yawnxyz|3 years ago|reply
Wow so cool! Does this act kind of like a logger and would we be able to have access to the logs later on, or should we bring our own logger as well?

(Also just curious, are you guys just using D1 or KV under the hood?)

[+] otterley|3 years ago|reply
What is the market for this solution? Whose pain point(s) are you solving? And what stops OpenAI from "Sherlocking"* you, i.e., making whatever you're building a free included feature and extinguishing the market?

*Or, to use a more modern analogy, "Evernoting"

[+] justintorre75|3 years ago|reply
Great question! The short answer is nothing! We just have to out pace them and make sure we develop a better product experience.

Two key advantages that we have over OpenAI. 1. We are open source and are trying to build a community of developers that can out run any single LLM provider. 2. We are completely provider agnostic, which allows us to aggregate and share common features across tools. (One analogy we like to use, is we want to emulate how dbt is database agonistic, and Helicone will grow to be provider agnostic)

[+] samstave|3 years ago|reply
This is great, however, I am concerned that the various AI 'ecosystem' of all the bolt-on, add-on, plug-ins etc... will be like a billion services all looking for payment - and any complex startup that needs a bunch of these services to build their own service/product/platform, it will be like the current state of streaming services.

So, youll be trying to manage a ton of SLAs, contracts, payment requirements, limits on service access that may be out of your budget to pay for all the various services, API calls, etc.

This is going to be an interesting cluster....

So we need a company thats a single service to access all available AI connects and the multiple billing channels.

However, then you have that as a single POF

[+] dcreater|3 years ago|reply
So route traffic through you so that you can monetize the insights from the data??
[+] killthebuddha|3 years ago|reply
LLM infrastructure is the spot to be right now for startups.
[+] curo|3 years ago|reply
Happy Helicone customer here. It's a dead simple setup. It's great to have the extra charts and logging to debug issues and make sure all is running well.

Congrats to the team!

[+] justintorre75|3 years ago|reply
Thanks! Please reach out to the team on discord if you have any questions
[+] antonok|3 years ago|reply
congrats! Helicone provides one of the biggest missing pieces from the the AI tool dev experience today, thanks for building this and sharing it with the rest of us!
[+] Hansenq|3 years ago|reply
Congrats! We've been happy users of Helicone for the past few months--it literally helped us solve a bug with OpenAI's API where we didn't know why requests were failing and we failed to log some of their responses. Helicone helped us debug that it was a token limit issue really quickly, especially since the logging around hasn't been great.

Love how easy it was to integrate too--just one line to swap out the OpenAI API with theirs.

[+] justintorre75|3 years ago|reply
Thanks! Glad Helicone was able to help your workflow.
[+] CGamesPlay|3 years ago|reply
Perfect! I was just today thinking about how I need to build up my own data set and should be logging all of my transcripts. This is exactly what I wanted.

I want to gather up my chat transcripts, then identify poor experiences in the chat, and then use that to guide fine-tuning. I don't believe that OpenAI actually provides anything to enable this as part of their platform, right?

[+] nico|3 years ago|reply
Awesome.

Is there a consumer version of this?

Like an alternative ChatGPT client or chrome extension that will save my prompts/conversations, tell me which ones I liked more and let me search through them?

[+] haolez|3 years ago|reply
From a VC perspective, this sounds like easy money, since it'll probably be acquired by OpenAI or Microsoft if it succeeds just a little.