You can't compare the accuracy of speech recognition to LLM task completion rates. A nearly-there yet incomplete solution to a Github issue is still valuable to an engineer who knows how to debug it.
Sure, and no doubt people paying for speech recognition 25 years ago were finding uses for it too. It depends on your use case.
A 13% success rate is both wildly impressive and also WAY below the level where I would personally find something like this useful. I can't even see reaching for a tool that I knew would fail 90% of the time, unless I was desperate and out of ideas.
I disagree. I think about this a bit as having a developer intern, on whom I can't rely to take much of a workload, and definitely nothing on the critical path, but I could say to them "Take a look at these particular well-defined tasks on the backlog and see which ones you could make some progress on" - I feel there's good value in that.
And the nice thing about an AI here is that I think it will actually find a different subset of these tasks to be easy than a human would.
> A nearly-there yet incomplete solution to a Github issue is still valuable to an engineer who knows how to debug it.
Not sure if I can agree. There would definitely be a value in looking at what libraries the solution uses, but otherwise it may be easier to write it oneself, especially when the mistakes are not humanlike.
HarHarVeryFunny|1 year ago
A 13% success rate is both wildly impressive and also WAY below the level where I would personally find something like this useful. I can't even see reaching for a tool that I knew would fail 90% of the time, unless I was desperate and out of ideas.
falcor84|1 year ago
And the nice thing about an AI here is that I think it will actually find a different subset of these tasks to be easy than a human would.
oytis|1 year ago
Not sure if I can agree. There would definitely be a value in looking at what libraries the solution uses, but otherwise it may be easier to write it oneself, especially when the mistakes are not humanlike.