[robmck]

Thanks so much for the thoughts! Happy to respond.

-Schema: We agree. After seeing how many times it has asked to see the database schema when working on a task, that is definitely something we will be adding.

-Manual / RAG: We did some RAG magic in the first two iterations of our product and found it was really hard to get right when the task was not well defined and as the size of the codebase grew. We found that our users were watching it do the search for the files and basically saying, "Let's see if it finds the code file I already know we need." So, we decided to go manual so at least the user isn't unpleasantly surprised by the results while we determine the right path forward with RAG. Or maybe even combine both where the user gives an "entry point," and the AI figures out the rest of the files

-I think you're right to be skeptical. In our experience, the LLMs are really best at breaking down a task to be just one notch more specific. So, I think the output here wouldn't be the best for a more high-level task. Our plan to handle this is to allow plans to be solved recursively, where you continually create a tree of subtasks, and only when you get to a specific enough "leaf" do you actually go through with generating code.

-You're right; I think we used the term "technical design doc" more generally to steer people's thinking toward software design in English rather than code. I like that you mention diagrams, though, because we would love to incorporate diagrams into our documents as our tool and LLM's capabilities increase!

-This is a very interesting idea! We used a version of this (a select set of elements of the AST that we tried to rank in importance) in our initial RAG system to give the AI a "starting point" for the RAG code search. It worked pretty well. My concern would be that the quality of the search might then depend on the naming/structure of the code (I'm thinking some 1000-line mega functions or something)

Thanks so much for the thoughtful questions!

EDIT: I see Maks also responded and I think we both offer similar messaging with slightly different delivery :)

[StarlaAtNight]

In case it helps as a tool recommendation, I’ve had some success having ChatGPT writing & rendering MermaidJS (diagrams as code)

[robmck]

This looks amazing thanks for the recommendation! Much better than the LaTeX diagram thread I was starting to pull on.