[SeanLuke]

I developed and maintain a large and very widely used open source agent-based modeling toolkit. It's designed to be very highly efficient: that's its calling card. But it's old: I released its first version around 2003 and have been updating it ever since.

Recently I was made aware by colleagues of a publication by authors of a new agent-based modeling toolkit in a different, hipper programming language. They compared their system to others, including mine, and made kind of a big checklist of who's better in what, and no surprise, theirs came out on top. But digging deeper, it quickly became clear that they didn't understand how to run my software correctly; and in many other places they bent over backwards to cherry-pick, and made a lot of bold and completely wrong claims. Correcting the record would place their software far below mine.

Mind you, I'm VERY happy to see newer toolkits which are better than mine -- I wrote this thing over 20 years ago after all, and have since moved on. But several colleagues demanded I do so. After a lot of back-and-forth however, it became clear that the journal's editor was too embarrassed and didn't want to require a retraction or revision. And the authors kept coming up with excuses for their errors. So the journal quietly dropped the complaint.

I'm afraid that this is very common.

[mnw21cam]

A while back I wrote a piece of (academic) software. A couple of years ago I was asked to review a paper prior to publication, and it was about a piece of software that did the same-ish thing as mine, where they had benchmarked against a set of older software, including mine, and of course they found that theirs was the best. However, their testing methodology was fundamentally flawed, not least because there is no "true" answer that the software's output can be compared to. So they had used a different process to produce a "truth", then trained their software (machine learning, of course) to produce results that match this (very flawed) "truth", and then of course their software was the best because it was the one that produced results closest to the "truth", whereas the other software might have been closer to the actual truth.

I recommended that the journal not publish the paper, and gave them a good list of improvements to give to the authors that should be made before re-submitting. The journal agreed with me, and rejected the paper.

A couple of months later, I saw it had been published unchanged in a different journal. It wasn't even a lower-quality journal, if I recall the impact factor was actually higher than the original one.

I despair of the scientific process.

[timr]

If it makes you feel any better, the problem you’re describing is as old as peer review. The authors of a paper only have to get accepted once, and they have a lot more incentive to do so than you do to reject their work as an editor or reviewer.

This is one of the reasons you should never accept a single publication at face value. But this isn’t a bug — it’s part of the algorithm. It’s just that most muggles don’t know how science actually works. Once you read enough papers in an area, you have a good sense of what’s in the norm of the distribution of knowledge, and if some flashy new result comes over the transom, you might be curious, but you’re not going to accept it without a lot more evidence.

This situation is different, because it’s a case where an extremely popular bit of accepted wisdom is both wrong, and the system itself appears to be unwilling to acknowledge the error.

[BLKNSLVR]

It seems that the failure of the scientific process is 'profit'.

Schools should be using these kinds of examples in order to teach critical thinking. Unfortunately the other side of the lesson is how easy it is to push an agenda when you've got a little bit of private backing.

[a123b456c]

Many people do not know that Impact Factor is gameable. Unethical publications have gamed it. Therefore a higher IF may or may not indicate higher prominence. Use Scimago journal rankings for non-gameable scores.

[bargle0]

If you’re the same Sean Luke I’m thinking of:

I was an undergraduate at the University of Maryland when you were a graduate student there in the mid nineties. A lot of what you had to say shaped the way I think about computer science. Thank you.

[domoregood]

Comments like this are the best part HN.

[sizzle]

Imagine if you did a bootcamp instead

[oawiejrlij]

When I was a grad student I contacted a journal to tell them my PI had falsified their data. The journal never responded. I also contacted my university's legal department. They invited me in for an hour, said they would talk to me again soon, and never spoke to me or responded to my calls again after that. This was in a Top-10-in-the-USA CS program. I have close to zero trust in academia. This is why we have a "reproducibility crisis".

[neilv]

PSA for any grad student in this situation: get a lawyer, ASAP, to protect your own career.

Universities care about money and reputation. Individuals at universities care about their careers.

With exceptions of some saintly individual faculty members, a university is like a big for-profit corporation, only with less accountability.

Faculty bring in money, are strongly linked to reputation (scandal news articles may even say the university name in headlines rather than the person's name), and faculty are hard to get rid of.

Students are completely disposable, there will always be undamaged replacements standing by, and turnover means that soon hardly anyone at the university will even have heard of the student or internal scandal.

Unless you're really lucky, the university's position will be to suppress the messenger.

But if you go in with a lawyer, the lawyer may help your whistleblowing to be taken more seriously, and may also help you negotiate a deal to save your career. (For example of help, you need the university's/department's help in switching advisors gracefully, with funding, even as the uni/dept is trying to minimize the number of people who know about the scandal.)

[bflesch]

Name and shame these frauds. Let me guess, was it Stanford?

[sizzle]

Name and shame?

[consp]

This reminds me of my former college who asked me to check some code from a study, which I did not know it was published, and told him I hope he did not write it since it likely produced the wrong results. They claimed some process was too complicated to do because it was post O(2^n) in complexity, decided to do some major simplification of the problem, and took that as the truth in their answer. End result was the original algorithm was just quadratic, not worse, given the data set was easily doable in minutes at best (and not days as claimed) and the end result did not support their conclusions one tiny bit.

Our conclusion was to never trust psychology majors with computer code. And like with any other expertise field they should have shown their idea and/or code to some CS majors at the very least before publishing.

[trogdor]

> it became clear that the journal's editor was too embarrassed

How sad. Admitting and correcting a mistake may feel difficult, but it makes you credible.

As a reader, I would have much greater trust in a journal that solicited criticism and readily published corrections and retractions when warranted.

[steveklabnik]

Unfortunately, academia is subject to the same sorts of social things that anything else is. I regularly see people still bring up a hoax article sent to a journal in 1996 as a reason to dismiss the entire field that one journal publishes in.

Personally, I would agree with you. That's how these things are supposed to work. In practice, people are still people.

[ameligrana]

I take the occasion to say that I helped making/rewriting a comparison between various agent-based modelling software at https://github.com/JuliaDynamics/ABMFrameworksComparison, not sure if this correctly represents all of them fairly enough, but if anyone wants to chime in to improve the code of any of the frameworks involved, I would be really happy to accept any improvement

[ameligrana]

SeanLuke, I tried to fix an issue about Mason I opened when I was looking into this a while back two years ago and tried to notify people about that (https://github.com/JuliaDynamics/ABMFrameworksComparison/iss...) with https://github.com/JuliaDynamics/ABMFrameworksComparison/pul..., hopefully the methodology is correct, I know very little about Java...In general, I don't think there is any very good comparison on performance in this field unfortunately at the moment, though if someone is interested in trying to make a correct one, I will be happy to contribute

[achillean]

I had a similar experience where a competitor released an academic paper rife with mistakes and misunderstandings of how my software worked. Instead of reaching out and trying to understand how their system was different than mine they used their incorrect data to draw their conclusions. I became rather disillusioned with academic papers as a result of how they were able to get away with publishing verifiably wrong data.

[pseudohadamard]

I reviewed for Management Science years ago, once. Once. They had a ridiculously baroque review process with multiple layers of reviewing and looping within them where a paper gets re-reviewed over and over. I couldn't see any indication that it improved the quality over the standard three-people-review-then vote process. The papers I was given were pure numerology, long equations involving a dozen or more terms multiplied out where changing any one of them would throw the results in a completely different direction. And the weightings in some of the equations seemed pretty arbitrary, "we'll put a 0.4 in here because it makes the result look about right". It really didn't inspire confidence in the quality of the stuff they were publishing.

Now I'm not saying that everything in M-S is junk, but the small subset I was exposed to was.

[orochimaaru]

I think the publish or perish academic culture makes it extremely susceptible to glossing over things like this - especially for statistical analysis. Sharing data, algorithms, code and methods for scientific publications will help. For papers above a certain citation count, which makes them seem "significant", I'm hoping google scholar can provide an annotation of whether the paper is reproducible and to what degree. While it won't avoid situations like what the author is talking about, it may force journal editors to take rebuttals and revisions more seriously.

From the perspective of the academic community, there will be lower incentive to publish incorrect results if data and code is shared.

[ecshafer]

I am going to assume you are the author of MASON and this was agents.jl?

They make a lot of claims on how much faster they are than MASON, Netlogo, and Mesa. But in practice I am not finding that to be the case. Also they arent counting the Julia compilation step which takes an absurdly long time, and by the time that gets done similar simulations are already done, then they start the clock on their own benchmark.

Agents.jl and Mesa have the selling point of having better languages / libraries for numerical computation. But thats really a subset of msor ABM I think.

[scott_waddell]

Academic comparisons rarely capture the messy reality of production use. The "hipper language" advantage tends to fade when you're debugging at 2am with actual users.

[cannonpalms]

Is this the kind of thing that retractions are typically issued for, or would it simply be your responsibility to submit a new paper correcting the record? I don't know how these things work. Thanks.

[contrarian1234]

maybe naiive but isnt this what "comments" in journals are for?

theyre usually published with a response by the authors