[jimhefferon]

At this moment, the error rate seems to be that of a beginning graduate student. Or at least, that's what Terry Tao thinks. That's pretty good.

[lupire]

That is not at all what Tao said.

https://mathstodon.xyz/@tao/113132502735585408

"Here the results were better than previous models, but still slightly disappointing: the new model could work its way to a correct (and well-written) solution if provided a lot of hints and prodding, but did not generate the key conceptual ideas on its own, and did make some non-trivial mistakes. The experience seemed roughly on par with trying to advise a mediocre, but not completely incompetent, (static simulation of a) graduate student. "

[buneskamin]

Yea it does seem pretty clear that it's mainly Terrance's contributions to the context window that is bringing the model to the right answer