[hellodanylo]

This should give a second life to Test-Driven Development.

One of the under-appreciated wisdoms of TDD is that there is a complexity asymmetry in many problems between finding a solution and (fully or partially) verifying it. Examples of asymmetric problems: inverting matrices, sorting an array, computing a function's gradient, compressing a byte stream, etc.

Human writes the easier part -- the test suite, the language model writes the harder part -- the solution. This can be a net gain in productivity.

[BaseballPhysics]

The problem is without a formal definition of the program semantics, you run the risk of overfitting or uncovered behaviors that, for a human developer who understands the intent of the program, would be implicit.

And given how hard formal verification is, I don't know that you'll ever get away with not having to manually check these programs, at which point I question just how much productivity you've gained.

It's kinda like self-driving cars: when they work, they work great. But when they fail, they fail in ways a human never would, and therefore a human struggles to anticipate or trust their behaviour.

That said, I'm waiting to see the rise of programming languages designed with LLMs in mind, where a human could use contract oriented programming or similar (think: Ada) combined with TDD methods to more formally specify the problem that an LLM is being asked to solve.

[hellodanylo]

> I question just how much productivity you've gained.

Me too. It's an empirical question to be answered by those who will dare to try.

> It's kinda like self-driving cars

Strong disagree. Yes, neural nets are blackboxes, but the generated code can be idiomatic, modular, easy to inspect with a debugger, etc.

> more formally specify the problem that an LLM is being asked to solve.

That would be a great direction to explore.

[fodkodrasz]

Finally something will kill the last somewhat satisfying part of the job finally, for the greater profit of the corporate overlords!

[ramesh31]

>Human writes the easier part -- the test suite, the language model writes the harder part -- the solution. This can be a net gain in productivity.

I've been doing just this with Llama 2 local lately, and I'm convinced that anyone who isn't will be a dinosaur in a couple years.

Fact is, these things are damn good (at JS at least) when dealing with pure unit-testable functions. To the point that I can accept practically any solution it comes up with will be better than mine. Everything is still verified of course, but the days of writing algorithms from scratch are basically over for me.

[hk__2]

How do you avoid overfitting and how do you test performance?

[doctoboggan]

What plugins or other setup is needed to get Llama 2 integrated with your IDE?

[awesomepeter]

Any tips on how to use it locally this way?

[hughesjj]

How readable is the code? Does it follow SOLID design and the likes?

[michaelfeathers]

I think people aren't getting yet is that this generalizable beyond software. We can do this with LLM and anything that can be designed.

https://michaelfeathers.silvrback.com/prompt-hoisting-for-gp...

[zitterbewegung]

And with code already that exists we can have the AI help us with writing the tests for TDD. Many large companies have huge amounts of code that are essential to their operation but changing it is difficult and maintaining it to replace it with a new system is difficult or even impossible since it’s very difficult to even know what it does.

[visarga]

Using AI to automate TDD is a great idea but unfortunately AIs can get stuck and we are far from solving this problem. Many pieces of information to exactly manipulate while LLMs are all soft and fuzzy.

[darepublic]

the LLM should write the test suite too, from the description of the app