[Bromeo]

Very interesting. The abstract claims that although GPT-4 was claimed to score in the 92nd percentile on the bar exam, when correcting for a bunch of things they find that these results are overinflated, and that it only scores in the 15th percentile specifically on essays when compared to only people that passed the bar.

That still does put it into bar-passing territory, though, since it still scores better than about one sixth of the people that passed the exam.

[falcor84]

If I understand currently, they measured it at the 69th percentile for the full test across all test takers, so definitely still impressive.