[worble]

I'd be curious in how well it passes 100th Coin's NES accuracy tests https://github.com/100thCoin/AccuracyCoin

[utopiah]

Indeed, that's what I kind of hinted at in https://news.ycombinator.com/item?id=46442195 and coincidentally https://news.ycombinator.com/item?id=46437688 briefly after, namely that OK, one can "generate" a "solution", that's much easier than before... but until we can verify somehow that it actually does what it say it does (and we know of hallucinations and have no reason to believe this changed) then testing itself, especially of well know "problems" is more and more important.

That being said, it doesn't answer the "why" in the first place, an even more important question. At least though it does help somehow to compare with existing alternatives.

[garciasn]

Isn’t this how all software development works? Folks commit code, it’s tested, and reviewed, and then deployed.

Why would this be any different?

[roger_]

I’m sure you can point Claude at that page and have it make the necessary changes to pass.

[deadbabe]

Or it could loop infinitely, never quite being able to pass all the tests.