[rontoes]

This is an incredibly poor take. It's well known that academic software doesn't follow great software engineering practices. The model is an incredibly complex piece of software that attempts to model stochastic behaviour. That it isn't fully deterministic (with a fixed random seed) doesn't make thee research invalid and shouldn't discredit research that builds on the model.

>On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector.

This is another level of crazy.

[vikinghckr]

>This is an incredibly poor take.

No it isn't.

> It's well known that academic software doesn't follow great software engineering practices.

That doesn't make it acceptable. Also, it's also "well known" that vast majority of academic work has zero tangible effect on society. This isn't one of those works. It's possibly the most important piece of academic work that has happened in recent memory. So the bar for this is MUCH higher than typical academic research work.

>. That it isn't fully deterministic (with a fixed random seed) doesn't make thee research invalid and shouldn't discredit research that builds on the model.

It does make it invalid, when the difference between runs is as big as 80,000 estimated deaths which can lead to dramatically different government policies.

> This is another level of crazy.

No it's not. Academia is way behind the industry when it comes to modeling the economy and the real world.

[spamizbad]

> No it's not. Academia is way behind the industry when it comes to modeling the economy and the real world.

The insurance industry is expected to ask the government for bailouts because none of their models can account for the fallout from this, just like AIG did during the '08 crisis.

[Nimitz14]

> It does make it invalid, when the difference between runs is as big as 80,000 estimated deaths which can lead to dramatically different government policies.

Wrong. Nobody decides policy based on whether there will be 320k or 400k deaths. What matters is the order of magnitude.

[smsm42]

There's a big difference between randomized model and code giving random results because you have race conditions in the code. If you do an experiment, you can expect some randomness because no experiment gives same value ever. But behind it is the common reality that you can measure. Now, if during an experiment somebody would come and kick the equipment, spit on it, spill water on it, subject it to random electric shocks, shoot at it, etc. and then when results come out all over the place say "well, it's supposed to be random!", completely ignoring what was done to it - you'd say that person is insane, it's supposed to be random but not like that.

If you code has race conditions that make your simulation roll out and produce different results, then you aren't measuring the result of the simulation, you are measuring race conditions.

Of course, you could be lucky and race condition could be harmless or contribute almost nothing to the result. But this is unusual, needs proof and indications are it's not the case. Responding to that "well, it's stochastic so what do you want!" is not the way it works.

Poor code quality is a sad reality in many scientific projects, yes. But it's not just a code that is unreadable. It's the code where race conditions knowingly mess with the model results! You can't just dismiss that.

[below43]

Suggesting all academic epidemiology be defunded really highlights the true intentions and bias of the writer.

It might be better, perhaps, for an academic to perform a code review instead.

[lbeltrame]

Regardless of the bias of the piece, AFAICT the GH issues linked to weren't made by the author (to be honest, my own bias suggested, before reading the text, that the author was the only one that pointed out the mistakes) but by others trying to use the model.

This makes the criticism (although partisan) at least worthy of attention.

[ilyaeck]

Big decisions impacting many livelihoods were made based on this model's projections. Its scope is way beyond academic.

[hartator]

> It's well known that academic software doesn't follow great software engineering practices.

Well, know that literally trillions of dollars have been lost, it's not a surprise that the computer model at the origin of the panic is being scrutinized.

[s9w]

Why do you think that's a crazy conclusion? That code is worthless - such quality of work would not be accepted in the private sector. And it's doubtful it can be of any value at all.

So when the work they do is this bad, I think it's reasonable to question whether they should be continued to be paid for no value to the public.

[lbeltrame]

> such quality of work would not be accepted in the private sector.

To be fair, if such code would be used for a paper and the flaws came out, I think it would warrant a retraction.

I think I saw not too long ago a story right here where a software error caused completely wrong results.