[enjo]

> it's literally "let me paste the assignment into ChatGPT and see what it spits out, change a few words and submit that".

My wife is an accounting professor. For many years her battle was with students using Chegg and the like. They would submit roughly correct answers but because she would rotate the underlying numbers they would always be wrong in a provably cheating way. This made up 5-8% of her students.

Now she receives a parade of absolutely insane answers to questions from a much larger proportion of her students (she is working on some research around this but it's definitely more than 30%). When she asks students to recreate how they got to these pretty wild answers they never have any ability to articulate what happened. They are simply throwing her questions at LLMs and submitting the output. It's not great.

[Zanfa]

ChatGPT is laughably terrible at double entry accounting. A few weeks ago I was trying to use it to figure out a reasonable way to structure accounts for a project given the different business requirements I had. It kept disappearing money when giving examples. Pointing it out didn’t help either, it just apologized and went on to make the same mistake in a different way.

[andai]

Using a system based on randomness for a process that must occur deterministically is probably the wrong solution.

I'm running into similar issues trying to use LLMs for logic and reasoning.

They can do it (surprisingly well, once you disable the friendliness that prevents it), but you get a different random subset of correct answers every time.

I don't know if setting temperature to 0 would help. You'd get the same output every time, but it would be the same incomplete / wrong output.

Probably a better solution is a multi phase thing, where you generate a bunch of outputs and then collect and filter them.

[Suppafly]

>Pointing it out didn’t help either, it just apologized and went on to make the same mistake in a different way.

They really should modify it to take out that whole loop where it apologizes, claims to recognize its mistake, and then continues to make the mistake that it claimed to recognize.

[vintermann]

You'd think accounting students would catch on.

[davedx]

> me, just submitted my taxes for last year with a lot of help from ChatGPT: :eyes:

[samuel]

I guess this students don't pass, do they? I don't think that's a particularly hard concern. It will take a bit more, but will learn the lesson (or drop out).

I'm more worried about those who will learn to solve the problems with the help of an LLM, but can't do anything without one. Those will go under the radar, unnoticed, and the problem is, how bad is it, actually? I would say that a lot, but then I realize I'm pretty useless driver without a GPS (once I get out of my hometown). That's the hard question, IMO.

[Stubbs]

As someone already said, parents used to be concerned that kids wouldn't be able to solve maths problems without a calculator, and it's the same problem, but there's a difference between solving problems _with_ LLMs, and having LLMs solve it _for you_.

I don't see the former as that much of a problem.

[lr4444lr]

How many people are "good drivers" outside their home town? I am not that old, but old enough to remember all adults taking wrong turns trying to find new destinations for the first time.

[shinycode]

For your GPS at worst you follow directions road sign by road sign. For a job without the core knowledge what’s the goal of hiring one person vs an unqualified one doing just prompts or worse, hiring no one and let agents do the prompting ?

[xhkkffbf]

All tech becomes a crutch. People can't wash their clothes without a machine. People can't cook without a microwave. Tech is both a gift and a curse.

[9rx]

Back in my day they worried about kids not being able to solve problems without a calculator, because you won't always have a calculator in your pocket.

...But then.

[DSingularity]

This is now reality -- fighting to change the students is a losing battle. Besides in terms of normalizing grade distributions this is not that complicated to solve.

Target the cheaters with pop quizzes. Prof can randomly choose 3 questions from assignments. If students cant get enough marks on 2/3 of them they are dealt a huge penalty. Students that actually work through the problems will have no problems with scoring enough marks on 2/3 of the questions. Students that lean irresponsibly on LLMs will lose their marks.

[cellularmitosis]

Why not just grade solely based on live performance? (quizzes and tests)

Homework would still be assigned as a learning tool, but has no impact on your grade.

[rrr_oh_man]

Maybe we'll revert to Soviet bilet-style oral exams...

[el_benhameen]

I wonder to what extent this is students who would have stuck it out now taking the easy way and to what extent it’s students who would have just failed now trying to stick it out.

[anon35]

This is an extremely important question, and you’ve phrased it nicely.

We’re either handicapping our brightest, or boosting our dumbest. One part is concerning, the other encouraging .

[woodrowbarlow]

my partner teaches high school math and regularly gets answers with calculus symbols (none of the students have taken any calculus). these students aren't putting a single iota of thought into the answers they're getting back from these tools.

[pc86]

To me this is the bigger problem. Using LLMs is going to happen and there's nothing anyone can do to stop it. So it's important to make people understand how to use them, and to find ways to test that students still understand the underlying concepts.

I'm in a 100%-online grad school but they proctor major exams through local testing centers, and every class is at least 50% based on one or more major exams. It's a good way to let people use LLMs, because they're available, and trying to stop it is a fool's errand, while requiring people to understand the underlying concepts in order to pass.

[iNic]

The solution is making all homework optional and having an old-school end of semester exam.

[Ekaros]

You can always give extra points for homework which then compensate from lacking in tests. If you get perfect points in test, well maximum grade. If less than perfect, you can up grade with those extra points. Fair for everyone.

[Suppafly]

>The solution is making all homework optional and having an old-school end of semester exam.

Not really. While doing something to ensure that students are actually learning is important, plenty of the smartest people still don't always test well. End of semester exams also tend to not be the best way to tell if people are learning along the way and then fall off part way through for whatever reason.