I find it surprising that many developers find using ai prompts easier than an actual programming language (not the first time I hear about it, but now it seems to be serious).
When I tried to “talk” to gpts, it was always hard to formulate my requests. Basically, I want to code. To tell something structured, rigorous, not vague and blah-blah-ey. To check and understand every step*. Idk why I feel like this, maybe for me the next ten years will be rougher than planned. Maybe because I’m not a manager type at all? Trust issues? I wonder if someone else feels the same.
Otoh I understand the appeal partially. It roots in the fact that our editors and “IDEs” suck at helping you to navigate the knowledge quickly. Some do it to a great extent, but it’s not universal. Maybe a new editor (plugin) will appear that will look at what you’re doing in code and show the contextual knowledge, relevant locations, article summaries, etc in another pane. Like a buddy who sits nearby with a beer, sees what you do and is seasoned enough to drop a comment or a story about what’s on your screen any time.
Programming in a natural language and trusting the results I just can’t stand. Hell, in practice we often can’t negotiate or align ourselves in basic things even.
This looks quite interesting. So I have question for these AI powered editors: what advantage would a dedicated editor like this have over just using an AI plugin for VsCode? How do you fundamentally build the editor differently if you are thinking about AI from the ground up?
Our editor isn't a regular coding editor. You don't actually write code with e2b. You write technical specs and then collaborate with an AI agent. Imagine it more like having a virtual developer at your disposal 24/7.
It's built on a completely new paradigms enabled by LLMs. This enables to unlock a lot of new use cases and features but at the same time there's a lot that needs to be built.
> How do you fundamentally build the editor differently if you are thinking about AI from the ground up?
Great question. I would love to hear the devs thoughts here. This is one of those questions where my intuition tells me there may be a really great "first principles" type of answer, but I don't know it myself.
If you could use it without submitting data to some ai company, or if it came with a non-disgusting terms of service, that would be a killer feature for me.
For example, the last ai company installer I just clicked "decline" to (a few minutes ago) says that you give it permission to download malware, including viruses and trojans, onto your computer and that you agree to pay the company if you infect other people and tarnish their reputation because of it. Literally. It was a very popular service too. I didn't even get to the IP section
edit: those terms aren't on their website, so I can't link to them. They are hidden in that tiny, impossible to read box during setup for the desktop installer
I built this https://github.com/campbel/aieditor to test the idea of programming directly with the AI in control. Long story short, VS Code plugin is better IMO.
In essence when working with code stops being the major thing you do (you abstract that away) and start managing the agents working on code and writing the spec, you need new tools and working environment to support that.
Hey everyone, I quite frankly really didn't expect our project getting on HN front page in 2 minutes after posting. I'm one of the creators of this project.
It's pretty nascent and a lot of work needs to be done so bear with us please. I'm traveling in 30 minutes but I'm happy to answer your questions.
A little about my co-founder and myself:
We've been building devtools for years now. We're really passionate about the field. The most recent thing we built is https://usedevbook.com. The goal was to build a simple framework/UI for companies to demo their APIs. Something like https://gradio.app but for API companies. It's been used for example by Prisma - we helped them build their playground with it - https://playground.prisma.io/
Our new project - e2b - is using some of the technology we built for Devbook in the past. Specifically the secure sandbox environments where the AI agents run are our custom Firecracker VMs that we run on Nomad.
If you want to follow the progress you can do follow the repo on GH or you can follow my co-founder, me, and the project on Twitter:
My biggest concern with tools like this is reproducibility and maintainability. How deterministically can we go from the 'source' (natural language prompts) to the 'target' (source code)? Assuming we can't reasonably rebuild from source alone, how can we maintain a link between the source and target so that refactoring can occur without breaking our public interface?
This is a valid concern and we are still experimenting with how to do this right.
A combination of preserving the reasoning history, having the generated code, and using tests to enforce the public interface (and fix it if anything breaks) looks promising.
I think the crucial part is indeed not being able to deterministically go from NL to code but to take an existing state of the codebase and spec and "continue the work".
Pretty simple it’s just like any abstraction. This AI will not work when nobody would deliver the answer beforehand. LLMs are given inputs of existing code. When you abstract that you better hope you have good code in it.
So my question would be, what is the use case?
I guess it’s more like planing software and not implementing it.
You can pretty well plan your software with ChatGPT. But it will just help you not really doing the job.
> mlejva: Our editor isn't a regular coding editor. You don't actually write code with e2b.
then what licensing problems arise from its use? In theory, if you only prompt the AI to write the software, is the software even your intellectual property?
It seems like this is a public domain software printing machine if you really aren’t meant to edit the output.
Good observation. I did notice that a lot of the types that jumped from framework to framework are now jumping onto ai. Please keep them there, at least that way js will stand a chance at becoming sane.
Is there examples / case studies of more complex apps being built by LLMs? I've seen some interesting examples but they were all small and simple examples. I'd love to see more case studies of how well these tools perform in more complex scenarios.
My gut feeling is we're still a few LLMs generations away from this being really usable but I'd love to hear how the authors are thinking about this.
Can you give an example of complex? I’ve used ChatGPT to help me build an app that authenticates a user using Oauth. That information creates a user in the backend (Rails). That user can then import issues tagged with specific information from a 3rd party task management tool (Linear). The title for these issues are then listed in the UI. From there, the user can create automatic release notes from those issues. They can provide a release version, description, tone, audience, etc.
All of that (issues list, version, tone, etc) is then formulated into a GPT prompt. The prompt is structured such that it returns written release notes. That note is then stored and the user can edit it using a rich text editor.
Once the first note is created the system can help the user write future notes by predicting release version, etc.
This isn’t that complex imo, but I’m curious to see if this is what people consider complex.
While you mention that you can bring your own model, prompt, etc, the current main use case seems to be integrating with OpenAI. How, if at all, do you plan to address the current shortcoming that the code generated by it often doesn't work at all without numerous revisions?
Good point and feedback, thank you. We'll update readme.
A lot of UX, UI, DX work related to LLMs is completely new. We ourselves have a lot of new realizations.
> While you mention that you can bring your own model, prompt, etc, the current main use case seems to be integrating with OpenAI
You're right. This is because we started with OpenAI and to be fair it's easiest to use from the DX point of view. We want to make it more open very soon. We probably need more user feedback to learn what would be the best way how this integration should look like.
> How, if at all, do you plan to address the current shortcoming that the code generated by it often doesn't work at all without numerous revisions?
The AI agents work in a self-repairing loop. For example they write code, get back type errors or a stack trace and then try to fix the code. This works pretty well and often a bigger problem is short context window.
We don't think this will replace developers, rather we need to figure out the right feedback loop between the developer and these AI agents. So we expect that developers most likely will edit some code.
Based on my experiences having spent a bunch of time experimenting with LLMs for writing code, I think the hardest part that I haven't yet seen a solution to is modifying existing code bases.
Sure, if you're doing greenfield, just ask it to write a new file. If you only have a simple script, you can ask it to rewrite the whole file. The tricky bit however, is figuring out how to edit relevant parts of the codebase, keeping in mind that the context window of LLMs is very limited.
You basically want it to navigate to the right part of the codebase, then scope it to just part of a file that is relevant, and then let it rewrite just that small bit. I'm not saying it's impossible (maybe with proper indexing with embeddings, and splitting files up by i.e. functions you can make it work), but I think it's very non-trivial.
Anyway, good luck! I hope you'll share your learnings once you figure this out. I think the idea of putting LLMs into Firecracker VMs to contain them is a very cool approach.
I think this will get easier as the supported context sizes get larger. Of course the tooling needs to take care of most of the drudgery, but I'm not sure there's any underlying limitation of an LLM that makes refactoring existing code any different from creating new code. It's just a matter of feeding it the layout of the repo and the code itself in a way that the LLM can retrieve and focus on the relevant files when you ask it for help with a task.
This is actually possible, I made a proof of concept agent that does exactly this. The agent has access to several commands that get executed on demand - ListFiles, ReadFile and WriteFile. You give it a prompt - for example "add a heading to the homepage and make it yellow" and it will use ListFiles to locate the correct file, then use ReadFile to read it and then finally use WriteFile to write the new contents of that file.
It was just a PoC so it wasn't bulletproof and I only ever made it work on a small Next.js codebase (with every file being intentionally small so it fits the context window) but it did work correctly.
All these tools starts with grand proclamation of “open” and then the first thing you notice is the field to add your OPENAI_KEY. My humble suggestion is that if you are building something truly open please use some other models like LLAMA or BERT as default example and keep options for adding other models as needed.
Hey, thank you for the feedback. I understand that having a requirement for OpenAI key isn’t good for folks. The project is 4 weeks old and OpenAI is what we started with as a LLM. We want to create interface for people to bring and connect their own models though. We will do a better work of explaining this in our readme
> The current idea is to offer the base cloud version for free while having some features for individuals behind a subscription. We'll share more on pricing for companies and enterprises in the future.
What happens if you use the README.md and associated documentation as a prompt to re-implement this whole thing?
Maybe to spite Mircosoft? I would love to completely remove Microsoft from my workflow. The way telemetry creeps into everything bugs the hell out of me. I also really don't like the security model of vscode where it's assumed that any code you look at comes from a trusted source, and might execute code on your local system. That's a ridiculous security assumption for a text editor, but not surprising considering that these people also made MS Office.
I don't think the integration with vscode would work well here. We're ditching a lot of existing workflows for a completely new interaction between the developer and the agent. It's not like copilot at all. E2b is more like having a virtual developer at your disposal 24/7. You write a technical spec and e2b codes builds the app for you
This is awesome, scary and very interesting. But, for me, it comes with a personal concern:
For some time I've been giving serious thought about an automated web service generator. Given a data model and information about the data (relationships, intents, groupings, etc.) output a fully deployable service. From unit tests through container definitions, and everything I can think of in-between (docs, OpenAPI spec, log forwarder, etc.)
So far, while my investment hasn't been very large, I have to ask myself: "Is it worth it?"
Watching this AI code generation stuff closely, I've been telling myself the story that the AI-generated code is not "provable". A deterministic system (like I've been imagining) would be "provable". Bugs or other unintended consequences would be directly traceable to the code generator itself. With AI code generation, there's no real way to know for sure (currently).
Some leading questions (for me) come down to:
1. Are the sources used by the AI's learning phase trustworthy? (e.g. When will models be sophisticated enough to be trained to avoid some potentially problematic solutions?)
2. How would an AI-generated solution be maintained over time? (e.g. When can AI prompt + context be saved and re-used later?)
3. How is my (potentially proprietary) solution protected? (e.g. When can my company host a viable trained model in a proprietary environment?)"
I want to say that my idea is worth it because the answers to these questions are (currently) not great (IMO) for the AI-generated world.
But, the world is not static. At some point, AI code generators will be 10x or 100x more powerful. I'm confident that, at some point, these code generators will easily surpass my 20+ years of experience. And, company-hosted, trained AI models will most likely happen. And context storage and re-use will (by demand) find a solution. And trust will eventually be accomplished by "proof is in the pudding" logic.
Basically, barring laws governing AI, my project doesn't stand a cold chance in hell. I knew this would happen at some point, but I was thinking more like a 5-10 year timeframe. Now, I realize, it could be 5-10 months.
Not OP but I've been playing with similar technology as a hobby.
>1. Are the sources used by the AI's learning phase trustworthy? (e.g. When will models be sophisticated enough to be trained to avoid some potentially problematic solutions?)
Probably not, but for most domains reviewing the code should be faster than writing it.
>2. How would an AI-generated solution be maintained over time?
I would imagine you don't save the original prompts. Rather, when you want to make changes you just give the AI the current project and a list of changes to make. Copilot can do this to some extent already. You'd have to do some creative prompting to get around context size limitations, maybe giving it a skeleton of the entire project and then giving actual code only on demand.
> When can my company host a viable trained model in a proprietary environment?
Hopefully soon. Finetuned LLaMA would not be far off GPT-3.5, but nowhere close to GPT-4. And even then there are licencing concerns.
> Given a data model and information about the data (relationships, intents, groupings, etc.) output a fully deployable service. From unit tests through container definitions, and everything I can think of in-between (docs, OpenAPI spec, log forwarder, etc.)
AWS roughly has one of these in Amplify. The data mapping parts are pretty great, though lots of the rest of it suck. The question I'd ask is if those parts suck by nature of the setup, or is it just amplify that has weird assumptions
IMO Emacs is a perfect candidate for this kind of thing, or maybe something akin to LSP so you can bring your own editor. New GPT extensions are coming out daily for Emacs, e.g. https://github.com/xenodium/chatgpt-shell
e2b isn't like copilot. You don't really write code in our "IDE". Currently, it works in a way that you write a technical spec and then collaborate with an AI agent that builds the software for you.
It's more like having a virtual developer available for you.
[+] [-] wruza|3 years ago|reply
When I tried to “talk” to gpts, it was always hard to formulate my requests. Basically, I want to code. To tell something structured, rigorous, not vague and blah-blah-ey. To check and understand every step*. Idk why I feel like this, maybe for me the next ten years will be rougher than planned. Maybe because I’m not a manager type at all? Trust issues? I wonder if someone else feels the same.
Otoh I understand the appeal partially. It roots in the fact that our editors and “IDEs” suck at helping you to navigate the knowledge quickly. Some do it to a great extent, but it’s not universal. Maybe a new editor (plugin) will appear that will look at what you’re doing in code and show the contextual knowledge, relevant locations, article summaries, etc in another pane. Like a buddy who sits nearby with a beer, sees what you do and is seasoned enough to drop a comment or a story about what’s on your screen any time.
Programming in a natural language and trusting the results I just can’t stand. Hell, in practice we often can’t negotiate or align ourselves in basic things even.
* and I’m not from a slow category of developers
[+] [-] celeritascelery|3 years ago|reply
[+] [-] mlejva|3 years ago|reply
Our editor isn't a regular coding editor. You don't actually write code with e2b. You write technical specs and then collaborate with an AI agent. Imagine it more like having a virtual developer at your disposal 24/7. It's built on a completely new paradigms enabled by LLMs. This enables to unlock a lot of new use cases and features but at the same time there's a lot that needs to be built.
[+] [-] rco8786|3 years ago|reply
Great question. I would love to hear the devs thoughts here. This is one of those questions where my intuition tells me there may be a really great "first principles" type of answer, but I don't know it myself.
[+] [-] prepend|3 years ago|reply
[+] [-] barbariangrunge|3 years ago|reply
For example, the last ai company installer I just clicked "decline" to (a few minutes ago) says that you give it permission to download malware, including viruses and trojans, onto your computer and that you agree to pay the company if you infect other people and tarnish their reputation because of it. Literally. It was a very popular service too. I didn't even get to the IP section
edit: those terms aren't on their website, so I can't link to them. They are hidden in that tiny, impossible to read box during setup for the desktop installer
[+] [-] campbel|3 years ago|reply
[+] [-] swyx|3 years ago|reply
per the end of the readme:
- one step to go from a spec to writing code, run code, debugging itself, install packages, and deploying the code
- creates "ephemeral UI" on demand
[+] [-] ValentaTomas|3 years ago|reply
[+] [-] mlejva|3 years ago|reply
It's pretty nascent and a lot of work needs to be done so bear with us please. I'm traveling in 30 minutes but I'm happy to answer your questions.
A little about my co-founder and myself: We've been building devtools for years now. We're really passionate about the field. The most recent thing we built is https://usedevbook.com. The goal was to build a simple framework/UI for companies to demo their APIs. Something like https://gradio.app but for API companies. It's been used for example by Prisma - we helped them build their playground with it - https://playground.prisma.io/
Our new project - e2b - is using some of the technology we built for Devbook in the past. Specifically the secure sandbox environments where the AI agents run are our custom Firecracker VMs that we run on Nomad.
If you want to follow the progress you can do follow the repo on GH or you can follow my co-founder, me, and the project on Twitter:
- https://twitter.com/e2b_dev
- https://twitter.com/t_valenta
- https://twitter.com/mlejva
And we have a community Discord server - https://discord.gg/U7KEcGErtQ
[+] [-] peteradio|3 years ago|reply
[+] [-] ivan888|3 years ago|reply
[+] [-] ValentaTomas|3 years ago|reply
I think the crucial part is indeed not being able to deterministically go from NL to code but to take an existing state of the codebase and spec and "continue the work".
[+] [-] larsonnn|3 years ago|reply
So my question would be, what is the use case?
I guess it’s more like planing software and not implementing it.
You can pretty well plan your software with ChatGPT. But it will just help you not really doing the job.
[+] [-] graypegg|3 years ago|reply
> mlejva: Our editor isn't a regular coding editor. You don't actually write code with e2b.
then what licensing problems arise from its use? In theory, if you only prompt the AI to write the software, is the software even your intellectual property?
It seems like this is a public domain software printing machine if you really aren’t meant to edit the output.
[+] [-] halfjoking|3 years ago|reply
I think everyone had this idea and is building something similar. I know I am.
[+] [-] gumballindie|3 years ago|reply
[+] [-] mlejva|3 years ago|reply
[+] [-] bgirard|3 years ago|reply
My gut feeling is we're still a few LLMs generations away from this being really usable but I'd love to hear how the authors are thinking about this.
[+] [-] ryanSrich|3 years ago|reply
All of that (issues list, version, tone, etc) is then formulated into a GPT prompt. The prompt is structured such that it returns written release notes. That note is then stored and the user can edit it using a rich text editor.
Once the first note is created the system can help the user write future notes by predicting release version, etc.
This isn’t that complex imo, but I’m curious to see if this is what people consider complex.
ChatGPT wrote 90% of the code for this.
[+] [-] Mali-|3 years ago|reply
[+] [-] Manjuuu|3 years ago|reply
[+] [-] jkettle|3 years ago|reply
[+] [-] Etheryte|3 years ago|reply
[+] [-] mlejva|3 years ago|reply
A lot of UX, UI, DX work related to LLMs is completely new. We ourselves have a lot of new realizations.
> While you mention that you can bring your own model, prompt, etc, the current main use case seems to be integrating with OpenAI
You're right. This is because we started with OpenAI and to be fair it's easiest to use from the DX point of view. We want to make it more open very soon. We probably need more user feedback to learn what would be the best way how this integration should look like.
> How, if at all, do you plan to address the current shortcoming that the code generated by it often doesn't work at all without numerous revisions?
The AI agents work in a self-repairing loop. For example they write code, get back type errors or a stack trace and then try to fix the code. This works pretty well and often a bigger problem is short context window.
We don't think this will replace developers, rather we need to figure out the right feedback loop between the developer and these AI agents. So we expect that developers most likely will edit some code.
[+] [-] anonzzzies|3 years ago|reply
It doesn’t so for many/most humans either. I hope they have a revisions prompting, but I did not try it yet.
I noticed adding in a feedback/review loop often fixes it, but the you still need someone saying ‘this is it’ as it doesn’t know the cut off point.
[+] [-] cube2222|3 years ago|reply
Sure, if you're doing greenfield, just ask it to write a new file. If you only have a simple script, you can ask it to rewrite the whole file. The tricky bit however, is figuring out how to edit relevant parts of the codebase, keeping in mind that the context window of LLMs is very limited.
You basically want it to navigate to the right part of the codebase, then scope it to just part of a file that is relevant, and then let it rewrite just that small bit. I'm not saying it's impossible (maybe with proper indexing with embeddings, and splitting files up by i.e. functions you can make it work), but I think it's very non-trivial.
Anyway, good luck! I hope you'll share your learnings once you figure this out. I think the idea of putting LLMs into Firecracker VMs to contain them is a very cool approach.
[+] [-] chatmasta|3 years ago|reply
[+] [-] M4v3R|3 years ago|reply
It was just a PoC so it wasn't bulletproof and I only ever made it work on a small Next.js codebase (with every file being intentionally small so it fits the context window) but it did work correctly.
[+] [-] la64710|3 years ago|reply
[+] [-] mlejva|3 years ago|reply
[+] [-] teddyh|3 years ago|reply
What happens if you use the README.md and associated documentation as a prompt to re-implement this whole thing?
[+] [-] exhaze|3 years ago|reply
Don’t most devs tend to be extremely sticky to their preferred dev env (last big migration was ST -> VS Code back in 2015-2017)
Read all the comments in here. Still not getting why this isn’t a VS Code plugin.
Distribution almost always beats product.
[+] [-] px43|3 years ago|reply
[+] [-] Thiebaut|3 years ago|reply
[+] [-] mlejva|3 years ago|reply
[+] [-] nico|3 years ago|reply
[+] [-] Jupe|3 years ago|reply
For some time I've been giving serious thought about an automated web service generator. Given a data model and information about the data (relationships, intents, groupings, etc.) output a fully deployable service. From unit tests through container definitions, and everything I can think of in-between (docs, OpenAPI spec, log forwarder, etc.)
So far, while my investment hasn't been very large, I have to ask myself: "Is it worth it?"
Watching this AI code generation stuff closely, I've been telling myself the story that the AI-generated code is not "provable". A deterministic system (like I've been imagining) would be "provable". Bugs or other unintended consequences would be directly traceable to the code generator itself. With AI code generation, there's no real way to know for sure (currently).
Some leading questions (for me) come down to:
1. Are the sources used by the AI's learning phase trustworthy? (e.g. When will models be sophisticated enough to be trained to avoid some potentially problematic solutions?)
2. How would an AI-generated solution be maintained over time? (e.g. When can AI prompt + context be saved and re-used later?)
3. How is my (potentially proprietary) solution protected? (e.g. When can my company host a viable trained model in a proprietary environment?)"
I want to say that my idea is worth it because the answers to these questions are (currently) not great (IMO) for the AI-generated world.
But, the world is not static. At some point, AI code generators will be 10x or 100x more powerful. I'm confident that, at some point, these code generators will easily surpass my 20+ years of experience. And, company-hosted, trained AI models will most likely happen. And context storage and re-use will (by demand) find a solution. And trust will eventually be accomplished by "proof is in the pudding" logic.
Basically, barring laws governing AI, my project doesn't stand a cold chance in hell. I knew this would happen at some point, but I was thinking more like a 5-10 year timeframe. Now, I realize, it could be 5-10 months.
[+] [-] sebzim4500|3 years ago|reply
>1. Are the sources used by the AI's learning phase trustworthy? (e.g. When will models be sophisticated enough to be trained to avoid some potentially problematic solutions?)
Probably not, but for most domains reviewing the code should be faster than writing it.
>2. How would an AI-generated solution be maintained over time?
I would imagine you don't save the original prompts. Rather, when you want to make changes you just give the AI the current project and a list of changes to make. Copilot can do this to some extent already. You'd have to do some creative prompting to get around context size limitations, maybe giving it a skeleton of the entire project and then giving actual code only on demand.
> When can my company host a viable trained model in a proprietary environment?
Hopefully soon. Finetuned LLaMA would not be far off GPT-3.5, but nowhere close to GPT-4. And even then there are licencing concerns.
[+] [-] 8note|3 years ago|reply
AWS roughly has one of these in Amplify. The data mapping parts are pretty great, though lots of the rest of it suck. The question I'd ask is if those parts suck by nature of the setup, or is it just amplify that has weird assumptions
[+] [-] mlejva|3 years ago|reply
[+] [-] sjm|3 years ago|reply
[+] [-] mfts0|3 years ago|reply
[+] [-] mlejva|3 years ago|reply
[+] [-] euroderf|3 years ago|reply
[+] [-] mlejva|3 years ago|reply
[+] [-] meatjuice|3 years ago|reply
[+] [-] mov_eax_ecx|3 years ago|reply
Copilot already exists and copilot X already packs the features this package promises AND much more, why use this application over Copilot?.
[+] [-] mlejva|3 years ago|reply
[+] [-] jumpCastle|3 years ago|reply