[LordHumungous]

How do you pass the bar exam without the ability to reason?

[qbasic_forever]

You're trained on a massive dataset that includes tens of thousands of practice exams with nearly the exact same questions and answers, just slightly different words/template to adjust for your answer.

[1970-01-01]

The MBE exam is mostly multiple choice elimination, and these multiple choices are filled with legal jargon and case law. GPT-4 picks the best predicted answer and can mimic reasons for the choice. This ability to mimic reasoning is good enough to receive a passing score on the MBE and many other exams.

[Ekaros]

And there is often certain common indicators to what answers exclude. So just excluding enough of these is already quite efficient.

Also the questions are unlikely to be very creative. I think it could be possible to train someone with good enough memory just based on existing tests.

[code_duck]

I presume by using training data of similar questions like Mad Libs.

[fluoridation]

Exam are designed to filter out entities that are already assumed to be able to reason based on their knowledge of some specific domain. A hypothetical entity with no ability to reason but great ability to remember facts and able to pass an exam, is conceivable.

[ddtaylor]

By remembering a lot of case law.