What percent does an average junior engineer solve? If it is even close, these models can be run all day and night for cheaper than one yearly SWE salary.
The problem is that you still need a human in the loop to determine if you're in the 13% success bucket, or 87% failure bucket, and the time it takes to make that determination is still a significant fraction of just solving the problem.
So the actual value here is not "13% of all issues fixed for the cost of compute," but more like "a discount on human time for 13% of the issues". But you also have to factor in the time taken on the 87% of issues where leading you down a wrong path can be adding time versus human only. It's not clear to me how it all shakes out, and would require large-sample experiments with humans to determine. I would bet the final margins are small though.
You raise a good point - AI + human review might end up being more time than just a human doing everything. I can see a certain subset of issues could be simple enough to done by AI and a quick review - like changing a button color or fixing clearly defined bugs. Time will tell how much work gets shifted over to AI + human review, but I'm betting on most of it.
Juniors already have bad and sometimes even negative ROI but today's working junior is the trusted engineer of tomorrow and the senior of the day after that. The problems they work on impart the knowledge and instincts that advance them through towards mastery and real value.
Budget-myopic executives already tried transfering that work to cheaper labor markets, but it worked much less than they expected and most ended up with unmaintainable software and loss of any hope for an actual engineering advantage against competitors. There's nothing new here.
There will be organizations that find a good and smart use for fully automated code generation, just like there is for outsourcing/offshoring, but it's not a universal win to just go with what's "cheaper" and organizations that don't look at the big picture are (as usual) trading short-term accounting gains for long-term value erosion.
jonahx|1 year ago
So the actual value here is not "13% of all issues fixed for the cost of compute," but more like "a discount on human time for 13% of the issues". But you also have to factor in the time taken on the 87% of issues where leading you down a wrong path can be adding time versus human only. It's not clear to me how it all shakes out, and would require large-sample experiments with humans to determine. I would bet the final margins are small though.
pstorm|1 year ago
swatcoder|1 year ago
Budget-myopic executives already tried transfering that work to cheaper labor markets, but it worked much less than they expected and most ended up with unmaintainable software and loss of any hope for an actual engineering advantage against competitors. There's nothing new here.
There will be organizations that find a good and smart use for fully automated code generation, just like there is for outsourcing/offshoring, but it's not a universal win to just go with what's "cheaper" and organizations that don't look at the big picture are (as usual) trading short-term accounting gains for long-term value erosion.