Springer Nature book on machine learning is full of made-up citations

[misja111]

"He (the author) did not answer our questions asking if he used an LLM to generate text for the book. However, he told us, “reliably determining whether content (or an issue) is AI generated remains a challenge, as even human-written text can appear ‘AI-like.’ This challenge is only expected to grow, as LLMs … continue to advance in fluency and sophistication.”

Lol, that answer sounds suspiciously much like LLM generated as well ..

[DebtDeflation]

It's true that "AI detection algorithms" are not particularly reliable.

It's also true that if you have fake CITATIONS in your works that such algorithms aren't necessary to know the work is trash - either it was written by AI or you knowingly faked your research and it doesn't really matter which.

[haffi112]

You would think that Springer did the due diligence here, but what is the value of a brand such as Springer if they let these AI slops through their cracks?

This is an opportunity for brands to sell verifiability, i.e., that the content they are selling has been properly vetted, which was obviously not the case here.

[WillAdams]

Back when I was doing academic publishing I'd use a regex to find all the hyperlinks, then a script (written by a co-worker, thanks again Dan!) to determine if they were working or no.

A similar approach should work w/ a DOI.

[cess11]

Why would one think that? All of the big journal publishers have had paper millers and fraudsters and endless amounts of "tortured phrases" under their names for a long, long time.

[ludicrousdispla]

>> LLM-generated citations might look legitimate, but the content of the citations might be fabricated.

Friendy reminder that the entire output from an LLM is fabricated.

[bryanrasmussen]

probably a better word here would be fabulated.

on edit: that is to say the content of the citations might be fabulated, while the rest is merely fabricated.

[xg15]

Technically yes, but not all of it has lost grounding with reality?

[amelius]

Fabricate is a word with ambiguous meaning. It can mean both "make up", but also simply "produce".

[Isamu]

One of the potential uses of AI that I have most wanted is automated citation lookup and validation.

First check if the citation references a real thing. Then actually read and summarize the referenced text and give a confidence level that it says what was claimed.

But no, we have AI that are compounding the problem. That says something about unaligned incentives.

[pyrale]

> One of the potential uses of AI that I have most wanted is automated citation lookup and validation.

Also one of the things AI is likely the least suited for.

best I could imagine an AI can do is offer sources for you to check for a given citation.

[dandanua]

We are approaching publishers' heaven, where AI reviewers review AI written books and articles (with AI editors fixing their style), allowing publishers to keep collecting billions from essentially mandatory subscriptions from institutions.

[flohofwoe]

It's fine, because human readers will also be replaced with AI which produce a quick summary ;)

[veltas]

Unfortunately not surprising, the quality of a lot of textbooks has been bad for a long time. Students aren't discerning and lecturers often don't try the book out themselves.

[gammalost]

I agree. I feel that Springer is not doing enough to uphold their reputation. One example of this being a book on RL that I found[1]. It is clear that no one seriously reviewed the content of this book. They are, despite its clear flaws charging 50+ euro.

https://link.springer.com/book/10.1007/978-3-031-37345-9

[antegamisou]

Understandably I'm becoming a bit dogmatic but I'll say it again, AIMA/PRML/ESL are still the best reference textbooks for foundational AI/ML and will be for a long time.

AIMA is Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig

PRML is Pattern Recognition and Machine Learning by Christopher Bishop.

ESL is Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani and Jerome Friedman.

[techas]

I saw this recently on some congress abstracts. I think it is just AI generated content. References look real and don’t exist.

[PicassoCTs]

To imagine this driving a singularity, meanwhile its putting the final nail in science, together with paper-spam and research-reward decline. They are going to hang us tech-priests from the lamp-posts when the consequences of this bullshit artistry hit home.

[Vinayak_A_B]

If I make a citation verifier, will conference/journal guys pay for it? First verify if the citation is legit, like the paper actually exists, after that another LLM that reads the paper cited and gives a rating out of 10, whether it fits the context or not. [ONLY FOR LIT SURVEY]

[SiempreViernes]

No, they aren't paying the reviewers in the first place.

[PeterStuer]

Given that the existence of a reference is fairly trivial to check, I'd wager the authors would not care enough to pay for this. As for 'fit', this is very much in the eye of the beholder and a paper can be cited for the most trivial part. Overcitation is usually not seen as a problem. Omitting citations the reviewer considers 'essential', often from their own lab or circles, is seen as non-negotiability.

So the better 'idea' would be to produce a CYA citation assistant that for a given paper adds all the remotely plausible references for all the known potential reviewers of a journal or conference. I honestly think this is not a hard problem, but doubt even that can be commercialized beyond Google Ads monetization.

[codewench]

So given that the output of an LLM is unreliable at best, your plan is to verify that a LLM didn't bullshit you by asking another LLM?

That sounds... counterproductive

[thoroughburro]

You’re offering to doublecheck measurements made with a bad ruler by using that same ruler.

[b00ty4breakfast]

This seems like the very thing that AI advocates would want to avoid. It certainly doesn't fill me, as an outsider to the whole thing, with much confidence for the future of AI-generated content but maybe I'm not the target sucker....err, I mean target demographic

[alok-g]

A next development would be people developing and using citation checkers. That would fix just that problem. The deeper underlying quality problem with statements in the text often remaining unverified/incorrect would remain unfixed.

If the authors are to manually and genuinely put some citations, chances would be higher that they are familiar with the cited work and the statement for which the work is cited is actually corroborated by the citation.

[zero_k]

Springer? You mean the publisher we are currently fighting so they won't mess up our peer-reviewed research paper that we wrote and paid for the privilege for them to mess up (ehm, sorry "publish")? Colour me surprised.

[amelius]

So was the entire text machine-generated?

Or did they take a human-written text and asked a machine to generate references/citations for it?

[ktallett]

Many write and then find citations to fit what they said rather than write based on what citable sources suggest

[passwordoops]

Why would anyone write a book then ask for citations?

[MengerSponge]

My "Plagiarism Machine #1 Fan" shirt has people asking a lot of questions already answered by my shirt.

[PeterStuer]

Would it be possible to 'squat' the non existent references and turbo boost oneself into 'most cited author' territory? :)

[rbanffy]

Use AI to do it, then write a paper about what you did.

[MarlonPro]

Bad news for old-school people who still love books as a learning resource.

[dwayne_dibley]

'Based on a tip from a reader, we checked 18 of the 46 citations in the book.' Why not just check them all?

[maweki]

They didn't just click a link. They contacted the supposed authors for comment. That would be a reason for not checking all of them.