[BoiledCabbage]

That's what so surprising to me - they data clearly shows the experiment had terrible results. And the write up is nothing but the author stating: "glowing success!".

And they didn't even bother to test the most important thing. Were the LLM evaluations even accurate! Have graders manually evaluate them and see if the LLMs were even close or were wildly off.

This is clearly someone who had a conclusion to promote regardless of what the data was going to show.

[wanderingbit]

> And they didn't even bother to test the most important thing. Were the LLM evaluations even accurate!

This is not true; the professor and the TAs graded every student submission. See this paragraph from the article:

(Just in case you are wondering, I graded all exams myself and I asked the TA to also grade the exams; we mostly agreed with the LLM grades, and I aligned mostly with the softie Gemini. However, when examining the cases when my grades disagreed with the council, I found that the council was more consistent across students and I often thought that the council graded more strictly but more fairly.)

[leoc]

At the risk of perhaps stating the obvious, there appears to be a whiff of aggression from this article. The "fighting fire with fire" language, the "haha, we love old FakeFoster, going to have to see if we change that" response to complaints that the voice was intimidating ... if there wasn't a specific desire to punish the class for LLM use by subjecting them to a robotic NKVD interrogation then the authors should have been more careful to avoid leaving that impression.

[Hnrobert42]

You can try out the voice yourself. It's not that bad.

https://elevenlabs.io/app/talk-to?agent_id=agent_8101k9d1pq4...

[plagiarist]

The belligerence about changing the voice is so weird. And it does sort of set a tone straight off. "We got feedback that the voice was frightening and intimidating. We're keeping it tho."

[knallfrosch]

I found "well, the LLMs converge when given each other's scores, so they agree and are correct" to be quite the jump to a conclusion.

[bsenftner]

I've got a long standing disagreement with an AI CEO that believes LLM convergence indicates greater accuracy. How to explain basic cause and effect in these AI use cases is a real challenge. The essential basic understanding of what an LLM is is not there, and that lack of comprehension is a civilization wide issue.

[pooper]

accuracy versus precision is something we learn in high school chemistry.

https://i.imgur.com/EshEhls.png

When someone at that level pretends to not understand it, there is no way to mince words.

This is malice.

[bjt]

They did compare the automated grades to the author's own manual ones. It's in there if you read more closely.

[chairmansteve]

As far as I can tell, there is very little empirical evidence of efficacy for most modern educational "advances".

Having said that, LLMs can be good tutors if used correctly.

[skybrian]

I don't think they're terrible, but I'm grading on a curve because it's their first attempt and more of a trial run. It seems promising enough to fix the issues and try again.