top | item 34529340

An AI lawyer was set to argue in court – real lawyers shut it down

484 points| isaacfrond | 3 years ago |npr.org | reply

686 comments

order
[+] lolinder|3 years ago|reply
This company is either run by someone who doesn't understand the tech or is willfully fraudulent. ChatGPT and company are far from good enough to be entrusted with law. Having interacted extensively with modern LLMs, I absolutely know something like this would happen:

> Defendant (as dictated by AI): The Supreme Court ruled in Johnson v. Smith in 1978...

> Judge: There was no case Johnson v. Smith in 1978.

LLMs hallucinate, and there is absolutely no space for hallucination in a court of law. The legal profession is perhaps the closest one to computer programming, and absolute precision is required, not a just-barely-good-enough statistical machine.

[+] jchw|3 years ago|reply
Pretty sure the whole reason why DoNotPay actually exists is because defending against parking tickets didn't actually require a strong defense. The tickets were flawed automation, and their formulaic nature justified and equally formulaic response, or something to that effect. Whether the LLM was actually going to output answers directly, or just be used to drive a behavior tree or something like that, is a question I don't see answered anywhere.

That said, if it's such a catastrophically stupid idea, I'm not really sure why it had to be shot down so harshly: seems like that problem would elegantly solve itself. I assume the real reason it was shot down was out of fear that it would work well. Does anyone else have a better explanation for why there was such a visceral response?

[+] eloff|3 years ago|reply
I think you’re right here and it’s the same reason I see AI as a tool in the software profession. You can use it to speed up your work, but you have to have someone fully trained who can tell the difference between looks good but is wrong, versus is actually usable.

I’ve been using copilot for half a year now and it’s helpful, but often wrong. I carefully verify anything it gives me before using it. I’ve had one bug that made it to production where I think copilot inserted code that I didn’t notice and which slipped past code review. I’ve had countless garbage suggestions I ignored, and a surprising amount of code that seemed reasonable but was subtly broken.

This will still require a human lawyer (and/or intern, depending on the stakes) to check its output carefully. I am not now, nor have I ever been afraid that AI is coming after my job. When it does, we’re dangerously close to general AI and a paradigm shift of such magnitude and consequence that it’s called The Singularity. At which point we may have bigger worries than jobs.

[+] mountainb|3 years ago|reply
What's interesting is that sometimes it does a great job at something like telling you the holdings of a case, but then other times it gives you a completely incorrect response. If you ask it for things like "the X factor test from Johnson v. Smith" sometimes it will dutifully report the correct test in bullets, but other times will say the completely wrong thing.

The issue I think is that it's pulling from too many sources. There are plenty of sources that are pretty machine readable that will give it good answers. There's a lot of training that can be eked out from the legal databases that already exist that could make it a lot better. If it takes in too much information from too many sources, it tends to get garbled.

There are also a lot of areas where it will confuse concepts from different areas of law, like mixing up criminal battery with civil battery, but that's not the worst of the problems.

[+] ethanbond|3 years ago|reply
DoNotPay seems to know very well what they’re doing.

It really doesn’t strike me as true that law requires absolute precision. There are many adjacent (both near and far) arguments that can work in law for any given case, since the interpreter is a human. You just need no silly mistakes that shatter credibility, but that’s very different from “get one thing wrong and the system doesn’t work at all or works in wildly unexpected ways.”

Low end law will be one of the first areas to go due to this tech. DoNotPay actually has already been doing this stuff successfully for a while (not in court proceedings themselves though).

[+] narrator|3 years ago|reply
If you make a ridiculous argument using confabulated case law as a lawyer, you can be subject to discipline by the state bar and even lose your law license. The legal system's time and attention is not free and unlimited and that's why you need a license to practice law. The judges and so forth don't want to deal with a bunch of people talking nonsense. Who is the lawyer who is putting their reputation on the line for the AI's argument? The people doing this want to say nobody is at fault for the obviously bogus arguments it's going to spout. That's why it's unacceptable.
[+] AlexTWithBeard|3 years ago|reply
I don't see the problem here.

> Defendant (as dictated by AI): The Supreme Court ruled in Johnson v. Smith in 1978...

> Judge: There was no case Johnson v. Smith in 1978. Case closed, here's your fine.

Next time please be more careful picking the lawyer.

[+] gfodor|3 years ago|reply
If this is the case, the lawyers should have nothing to fear, and the plaintiff nothing to lose but a parking ticket. I say we stop arguing and run the experiment.
[+] gnicholas|3 years ago|reply
Exactly. I asked it for books on Hong Kong history and it spit out five complete fabrications. The titles were plausible and authors were real people, but none of them had written the books listed.
[+] jonathanwallace|3 years ago|reply
> there is absolutely no space for hallucination in a court of law

I wish you were factually correct here.

We've seen time and time again where courts are mockeries of the ideal because of the people in them and the faults they bring with them.

E.g. see https://www.supremecourt.gov/opinions/21pdf/21-418_i425.pdf and the documented proof in the dissent contradicting the claims in the ruling.

[+] humans2|3 years ago|reply
Humans hallucinate too: https://youtu.be/lyu7v7nWzfo

We are unique in the universe but not important to its existence. Our automated inference technologies are accurately representing is; I read half a dozen STEM papers last year, even more during lockdown. Comma splices, grammatical errors everywhere. ChatGPT is us in the aggregate.

Even the geniuses of our species are imperfect and hallucinate being better than they are given their accomplishments relative to the laymen.

The court of law is itself an ephemeral hallucination which fails all the time; given the number of people proven innocent, it’s been suggested through analysis up to 25% or more may be incarcerated incorrectly. Drug laws are just one instance of humans hallucinating correct application of courts. YT broke a while back when it AI got hung up on circular logic in a debate about copyright (easily googled).

The burden of proof of “correctness” is on humans to prove their society is not merely a titillating hallucination.

We made computing machines before we had all the abstract semantics to describe it. Do those semantics mean anything to the production of computing machine or are they just a jargon bubble for a minority to memorize and capitalize on relative to those who have no idea wtf they’re talking about.

[+] intrasight|3 years ago|reply
>absolute precision is required

LOL - you gotta be kidding. In software, we strive for that - by running tests. In the law, there are no tests.

Not saying that absolute precision isn't required. I know lots of cases were an extra comma, a wrong date, or a signature from the wrong person has cost someone tens of millions of dollars. I would argue that AI-based tools could prevent such HUMAN mistakes.

[+] Spooky23|3 years ago|reply
I agree in many cases, except traffic court isn’t real court. It’s mostly procedural. Hell in my state, in many cases the judge is a person who wins an election and takes a 20 hour course, and prosecutions are conducted by police officers.

Lawyers like to wax about the interests of justice. Reality is for crap like this, it’s a revenue funnel where there’s a pecking order of people who get breaks. Some small fraction have an actual legal dispute of facts.

I’d argue that you could best serve the interests of justice by bringing the whole process online. You’d eliminate the “friends and family” discount and have a real tribunal process for cases that need it, instead of the cattle call and mockery that these courts are.

NYC actually does a decent job with this by outsourcing the whole thing to DMV and administrative law judges on a tight leash. It’s mostly justice by flowchart. That pushes bullshit down to the police.

[+] phpisthebest|3 years ago|reply
>>The legal profession is perhaps the closest one to computer programming

There is a ton of people using ChatGPT for programming.... so much so that I wonder if we will have a crisis in skills as people forgo how to write code.

sysadmin circles have tons people celebrating how they will not have to learn powershell now as an example

[+] enoch2090|3 years ago|reply
> Defendant (as dictated by AI): The Supreme Court ruled in Johnson v. Smith in 1978... > Judge: There was no case Johnson v. Smith in 1978. > Defendant (as dictated by AI): Yes, you are right, there was no case Johnson v. Smith in 1978.
[+] dukeofdoom|3 years ago|reply
That's hilarious, watch some of the trails on courtTV on youtube. The trials are are as culturally biased as you can get. And these ones are the ones we get to see. Judges are not some logical Spock! Free of influence, politics, and current group think. But people who think about their careers, public opinion and know who pays their paycheck. And these are the the competent ones. I remember judge Judy proclaiming "If it doesn't make sense, it's not true!!!", while screaming at some goy. This is pretty much the level of logic you can expect from a judge.
[+] barbazoo|3 years ago|reply
Maybe that's where we're headed. It looks like it's becoming more and more okay to just make things up to please your tribe. Why shouldn't that seep into the courtroom. I hope this doesn't happen.
[+] dragonwriter|3 years ago|reply
Ironically, I recently saw a convo on Twitter where someone was showing off a ChatGPT generated legal argument, and, it had done exactly that, hallucinated a case to cite.
[+] bostonsre|3 years ago|reply
In the court itself there would definitely be no way to trust them right now, but I could see AI being a useful research tool for cases. It could find patterns and suggest cases for someone that is qualified to look further into. No idea how hard it is for lawyers to find relevant cases now, but seems like it could be a tough problem.
[+] yubiox|3 years ago|reply
LLM means Master of Law degree, kind of an advanced JD used in some specializations like tax. What do you mean here?
[+] computerex|3 years ago|reply
Yes it's true that LLMs hallucinate facts, but there are ways to control that. Despite the challenges they can spit out perfectly functional code to spec to boot. So for me it's not too much of a stretch to think that it'd do a reasonably good job at defending simple cases.
[+] criddell|3 years ago|reply
It would be kind of funny to hear an argument made by an LLM that has digested all the sovereign citizen bs.
[+] ma2rten|3 years ago|reply
You could probably do much better than ChatGPT if you built a bot specifically for being a lawyer.
[+] ChrisRR|3 years ago|reply
ChatGPT Is currently very insistent that there is no letter T in Saturday. That alone should be proof enough that you can't blindly trust it

Even then, before the December update it was sure that there was a T in Sunday

[+] dclowd9901|3 years ago|reply
This was never meant in good faith. The company did for the PR and got the PR.
[+] winReInstall|3 years ago|reply
Eh, what if it was trained on all the previous cases ever to have existed? I think it could be pretty good, as long as it detects novelty as a to flag and confirm error case.
[+] chrsig|3 years ago|reply
> LLMs hallucinate, and there is absolutely no space for hallucination in a court of law.

Well, _someone_ doesn't have enough schizophrenia in their life.

[+] PostOnce|3 years ago|reply
The AI is also _deceptively_ "right". For example, it will cite precedent that has since been superseded.

A non-lawyer representing themselves in a criminal case would overlook that, make a bad/wrong/misinformed argument, and go to jail.

In other fields, it'll lie to you about the thickness of steel pipe required to transport water at a certain pressure, it'll refer to programming libraries that don't exist, and it'll claim something impossible in one breath and happily explain it as fact in the next.

[+] kjkjadksj|3 years ago|reply
At the same time, all these cases are on the internet somewhere. It wouldn’t be too tricky to make a lawyer gpt that is heavily trained on existing legal documents, and is only allowed to quote verbatim from real sources.
[+] weatherlite|3 years ago|reply
They will pretty soon be able to fact check everything they say when they gain real time internet connectivity. But for now yeah you're right. A year-two from now this won't be true anymore.
[+] TheDudeMan|3 years ago|reply
What would happen if a human lawyer did that?
[+] elicksaur|3 years ago|reply
Good. Totally fine with trying to use AI to give legal advice, but it should be done with a lawyer’s license on the line. A company that explicitly disclaims being a law firm and states is not giving legal advice should also not get to tweet that they are “representing” someone in court.

A good bar (pun intended) for the quality of the tech is if it is good enough that a licensed attorney trusts it to give legal advice with their livelihood at stake. If this product doesn’t work for DoNotPay, they can just walk away and do something else, as they are doing anyways here. If it doesn’t work for a lawyer, they’d get sued for malpractice and possibly disbarred, ruining their career. When someone trusts it to that level, have at it.

[+] rzwitserloot|3 years ago|reply
As the Opening Arguments podcast (one of the two hosts is a lawyer) said: If as a lawyer you do what was asked - just parrot what an AI tells you to parrot, you're going to get sanctioned and possibly disbarred. As a lawyer you are responsible for what you say and argue, and if you argue something that you know to be false, you're in violation of the ethics standards; just about every bar association lists that as sanctionable, or even disbarrable, offense *.

Thus, effectively, the only thing you could do is a watered down concept of the idea: A lawyer that will parrot the ChatGPT answer, but only if said answer is something they would plausibly argue themselves. They'd have to rewrite or disregard anything ChatGPT told them to say that they don't think is solid argument.

They also run a segment where a non-lawyer takes a bar exam. Recently they've also asked the bar exam question to ChatGPT as well. So far ChatGPT got it wrong every time. For example, it doesn't take into account that multiple answers can be correct, in which case you have to pick the most specific answer available. Leading to a somewhat hilarious scenario where ChatGPT picks an answer and then defends its choice in a way that seems to indicate why the answer it picked is obviously the wrong answer.

*) Of course, Alan Dershowitz is now arguing in court that the seditious horse manure he signed his name under and which is now leading to him being sanctioned or possibly disbarred, is not appropriate because he's old and didn't know what he was doing. It's Dershowitz so who knows, but I'm guessing the court is not going to take his argument seriously. In the odd chance that they do, I guess you can just say whatever you want and not be responsible for it, which... would be weird.

[+] markogrady|3 years ago|reply
The HMCTS held a hackathon for the future tech in the UK court system a few years ago. The judges were people like the CEO of the courts, they also had lord chief justice. There were all sorts of firms like Linklaters, Pinsent Mason and Deloitte. We won with a simple Alexa lawyer that was to help poor rental tenants. It generated documents to send a landlord and possible legal advice. The idea was specifically for people who can not afford a lawyer. There was a lot of influential people who were very excited about this space, so it is strange when it actually gets implemented it's not allowed.

I wonder what the wider implications are for the legal system. Will there be less qualified human lawyers in the future due to the lack of junior roles that are filled by AI? Will lawyers be allowed to use AI to find different ways of looking at issues?

[+] brokenodo|3 years ago|reply
Aside from the question of whether this plan was legal, DoNotPay seems like a terrible product. The results it generates seem laughably bad, and it’s questionable whether “AI” is actually involved when it takes them literal days to generate a document for certain types of requests. https://www.techdirt.com/2023/01/24/the-worlds-first-robot-l...
[+] DonHopkins|3 years ago|reply
Judge:

    How does the defendant plea?
Defendant:

    ChatGPT is at capacity right now.

    Get notified when we're back.

    Write an acrostic poem about the status of ChatGPT.

    C: ChatGPT is currently down.
    H: Huge demand has caused the site to crash.
    A: All users will have to wait.
    T: Time is needed for the servers to catch up.
    G: Go grab a coffee and check back soon.
    P: Patience is key in this situation.
    T: Trust that the team is working hard to fix it up.
[+] jeroenhd|3 years ago|reply
I don't see the problem as long as the actual lawyer can intervene when necessary.

If ChatGPT did something wrong, that lawyer would still be on the hook for deciding to continue using this tool so responsibility/liability/authenticity is not a problem.

I get that they want to make some kind of subscription service to replace lawyers with AI (a terribly dystopian idea in my opinion, as only the rich would then have access to actual lawyers) but just like Tesla needs someone in the drivers seat to overrule the occasional phantom breaking and swerving, you need an actual lawyer for your proof of concept cases if you're going to go AI in a new area.

You'd also need a fast typist to feed the record into ChatGPT of course, because you can't just record lawsuits, but anyone with a steno keyboard should be able to keep up with a court room.

[+] ChicagoBoy11|3 years ago|reply
Honestly I think this guy was super clever. It was abundantly clear to anyone thinking about this that there was no way this ploy would work. But he got pretty big on Twitter, is getting all of this press, and has now built up awareness of his startup which is doing a far saner and less ambitious task incredible publicity which otherwise he would've had a hard time getting.
[+] legitster|3 years ago|reply
> Leah Wilson, the State Bar of California's executive director, told NPR that there has been a recent surge in poor-quality legal representation that has emerged to fill a void in affordable legal advice.

> "In 2023, we are seeing well-funded, unregulated providers rushing into the market for low-cost legal representation, raising questions again about whether and how these services should be regulated," Wilson said.

Got it. There are not enough affordable legal services in the US, and so the Bar's solution is to regulate them away.

[+] sandworm101|3 years ago|reply
AI is able to sometimes make a valid argument, but when it comes to specific facts and rules it drops the ball. Expert knowledge requires actual understanding, not fitting patterns and transposing words. Take a look at the following vid from a real expert in a paticular field (military submarines). Look at how ChatGPT falls appart when discussing "Akula" subs. It can read english but clearly does not understand what that word means in context. It also confidently cites incorrect facts, something that would be very dangerous in any court.

Hint: Akula is a nato reporting name. Nobody calls them the shark class of subs, even russians attach that name to a very different class.

https://youtu.be/H8DIwNfIijU

[+] Barrin92|3 years ago|reply
Absolutely atrocious stunt on the part of that company. A glorified chatbot is not itself legally accountable or trained lawyer and it cannot seriously represent anyone. I assume the entire purpose of this was to bait the obvious shutdown and then complain on the internet about the legacy lawyers or whatever to generate press. Reminds me of the 'sentient AI' Google guy.

Is this going to be the new grift in the industry?

[+] paul_funyun|3 years ago|reply
An AI lawyer, operating within reasonable bounds, could absolutely be an asset to criminal defendants and parties to civil litigation. You could reduce a basic discovery request to mad libs. I'd go so far as to say you could do the same with motions for summary judgment, requesting depositions, and other things. They wouldn't be optimal, but they wouldn't be of a level worthy of sanctions. It's just protectionism from the private bar who doesn't want to lose easy billables, and fear from prosecutors, creditors, and the like who realize that their system would collapse if half their opposition could force them to do some real lawyering a time or two. If every criminal defendant and debtor could squeeze three hours of drafting documents and individualized courtroom attention out of opposing counsel, it wouldn't be the guy using the AI coming out worse than the status quo. You can argue that the consequences would be negative for society, but it's laughable to say they'd be negative for litigants. DCS can barely knock down scarecrows; a few mediocre pleadings that demand a response from child support obligees and other parties would send them crying into early retirement.
[+] joshuaheard|3 years ago|reply
"Unauthorized practice of law" only applies to people, not tools. AI is a tool. DoNotPay was not selling legal advice, only a tool to understand law. It is no different if they were selling a code book, or other text that the defendant uses himself. I think the real fear is that AI will supplant the entire legal profession.

The legal profession went through a similar struggle when Nolo published software that could draft basic legal documents by filling in the blanks. Nolo won.

[+] jedberg|3 years ago|reply
"The justice system moves swiftly now that they've abolished all lawyers!"

-- Doc Brown in 2015, Back to the Future

I look forward to the day when cases are argued on both sides by an AI to an AI judge. It should work about as well as Google customer service!

But seriously, having the AI do the arguing is silly. AI should be a tool. I see no issue using an AI to inform a lawyer who can use what it outputs to make their case stronger, but just using an AI seems fraught with peril.

[+] aqme28|3 years ago|reply
I don't like that they're testing this out live.

Do what you'd have to do if this were say a medical device: hire a retired judge or two and set up double-blind fake trials with AI or human representation. Prove it works, then try it with real people.

[+] qwerty456127|3 years ago|reply
Every person should still have the right to be defended by a human lawyer yet the right to voluntarily choose an AI lawyer to either defend you or just hint you as you defend yourself would be great to have. It may totally change the game where (currently) whoever can afford expensiwe lawyers generally wins and whoever can't automatically looses exorbitant sums of money. Real lawyers will never let this happen.
[+] ChrisMarshallNY|3 years ago|reply
Well, the comments about the company not doing things correctly (licensing the algorithm), are correct.

It's actually critically important to have some kind of license to represent people in court, as well as someone to pillory, if they screw up, as it prevents some truly evil stuff from happening (I have seen many people robbed blind by licensed lawyers, and it would be a thousand times worse, if they could be represented by anyone that sounds convincing enough). The stakes are really high, and we shouldn't mess around (not in all cases, of course, but it would really suck, if someone got the needle, because a programmer forgot a semicolon).

That said, I think it's only a matter of time, before a significant amount of legal stuff is handled by AI. AI shines, in environments with lots of structure and rules; which pretty much defines the law.

[+] giantg2|3 years ago|reply
I'd rather have judges at the lower levels rely on some AI assistance. The level of utter incompetence that I've witnessed personally has been hard to comprehend.
[+] concordDance|3 years ago|reply
The fact that a guild controls the legal system has always been alarming to me. Its very much in their interests to make it impossible to avoid spending huge amount on their services and reduce supply by making it hard for more people to become members.

Lawyers will probably be the last profession to be automated.

[+] minhazm|3 years ago|reply
This whole thing was clearly a marketing stunt, they knew from the beginning they wouldn't be able to do it but they got a ton of free publicity out of it.