Statisticians use a technique that leverages randomness to deal with the unknown

[Jun8]

Not one mention of the EM algorithm, which is, as far as I can understand, is being described here (https://en.m.wikipedia.org/wiki/Expectation%E2%80%93maximiza...). It has so many applications, among which is estimating number of clusters for a Gaussian mixture model.

An ELI5 intro: https://abidlabs.github.io/EM-Algorithm/

[Sniffnoy]

It does not appear to be what's being described here? Could you perhaps expand on the equivalence between the two if it is?

[CrazyStat]

EM can be used to impute data, but that would be single imputation. Multiple imputation as described here would not use EM since the goal is to get samples from a distribution of possible values for the missing data.

[miki123211]

> It has so many applications, among which is estimating number of clusters for a Gaussian mixture model

Any sources for that? As far as I remember, EM is used to calculate actual cluster parameters (means, covariances etc), but I'm not aware of any usage to estimate what number of clusters works best.

Source: I've implemented EM for GMMs for a college assignment once, but I'm a bit hazy on the details.

[clircle]

Does any living statistician come close to the level of Donald Rubin in terms of research impact? Missing data analysis, causal inference, EM algorithm, any probably more. He just walks around creating new subfields.

[selimthegrim]

Efron?

[j7ake]

Mike Jordan, Tibshirani, Emmanuel Candes

[selectionbias]

Also approximate Bayesian computation, principal stratification, and the Bayesian Bootstrap.

[aquafox]

Andrew Gelman?

[xiaodai]

I don’t know. I find quanta articles very high noise. It’s always hyping something

[jll29]

I don't find the language of the article full of "hype"; they describe the history of different forms of imputation from single to multiple to ML-based.

The table is particularly useful as it describes what the article is all about in a way that can stick to students' minds. I'm very grateful for QuantaMagazine for its popular science reporting.

[vouaobrasil]

I agree with that. I skip the Quanta magazine articles, mainly because the titles seem to be a little to hyped for my taste and don't represent the content as well as they should.

[MiddleMan5]

Curious, what sites would you recommend?

[TaurenHunter]

Donald Rubin is kind of a modern day Leibniz...

Rubin Causal Model

Propensity Score Matching

Contributions to

Bayesian Inference

Missing data mechanisms

Survey sampling

Causal inference in observations

Multiple comparisons and hypothesis testing

[light_hue_1]

I wish they actually engaged with this issue instead of writing a fluff piece. There are plenty of problems with multiple imputation.

Not the least of which is that it's far too easy to do the equivalent of p hacking and get your data to be significant by playing games with how you do the imputation. Garbage in, garbage out.

I think all of these methods should be abolished from the curriculum entirely. When I review papers in the ML/AI I automatically reject any paper or dataset that uses imputation.

This is all a consequence of the terrible statics used in most fields. Bayesian methods don't need to do this.

[jll29]

There are plenty of legit. articles that discuss/survey imputation in ML/AI: https://scholar.google.com/scholar?hl=de&as_sdt=0%2C5&q=%22m...

[parpfish]

I feel like multiple imputation is fine when you have data missing at random.

The problem is that data is never actually missing at random and there’s always some sort of interesting variable that confounds which pieces are missing

[DAGdug]

Maybe in academia, where sketchy incentives rule. In industry, p-hacking is great till you’re eventually caught for doing nonsense that isn’t driving real impact (still, the lead time is enough to mint money).

[aabaker99]

My intuition would be that there are certain conditions under which Bayesian inference for the missing data and multiple imputation lead to the same results.

What is the distinction?

The scenario described in the paper could be represented in a Bayesian method or not. “For a given missing value in one copy, randomly assign a guess from your distribution.” Here “my distribution” could be Bayesian or not but either way it’s still up to the statistician to make good choices about the model. The Bayesian can p hack here all the same.

[fn-mote]

Clearly you know your stuff. Are there any not-super-technical references where an argument against using imputation is clearly explained?

[karaterobot]

Does anyone else find it maddeningly difficult to read Quanta articles on desktop, because the nav bar keeps dancing around the screen? One of my least favorite web design things is the "let's move the bar up and down the screen depending on what direction he's scrolling, that'll really mess with him." I promise I can find the nav bar on my own when I need it.

[paulpauper]

why not use regression on the existing entries to infer what the missing ones should be?

[ivansavz]

That would push things towards the mean... not necessarily a bad thing, but presumably later steps of the analysis will be pooling/averaging data together so not that useful.

A more interesting approach, let's call it OPTION2, would be to sample from the predictive distribution of a regression (regression mean + noise), which would result in more variability in the imputations, although random so might not what you want.

The multiple imputation approach seems to be a resampling methods of obtaining OPTION2, w/o need to assume linear regression model.

[bgnn]

Why not interpolate the missing data points with similar patients data?

This must be about the confidence of the approach. Maybe interpolation would be overconfident too.

[SillyUsername]

Isn't this just Monte Carlo, or did I miss something?

[hatmatrix]

Monte Carlo is one way to implement multiple imputation.

[userbinator]

It reminds me somewhat of dithering in signal processing.

[a-dub]

life is nothing but shaped noise