[it_does_follow]

Kevin Murphy has done an incredible service to the ML (and Stats) community by producing such an encyclopedic work of contemporary views on ML. These books are really a much need update of the now outdated feeling "The Elements of Statistical Learning" and the logical continuation of Bishop's nearly perfect "Pattern Recognition and Machine Learning".

One thing I do find a bit surprising is that in the nearly 2000 pages covered between these two books there is almost no mention of understanding parameter variance. I get that in machine learning we typically don't care, but this is such an essential part of basic statistics I'm surprised it's not covered at all.

The closest we get is in the Inference section which is mostly interested in prediction variance. It's also surprising that in neither the section on Laplace Approximation or Fisher information does anyone call out the Cramér-Rao lower-bound which seems like a vital piece of information regarding uncertainty estimates.

This is of course a minor critique since virtual no ML books touch on these topics, it's just unfortunate that in a volume this massive we still see ML ignoring what is arguably the most useful part of what statistics has to offer to machine learning.

[dxbydt]

Do you really expect this situation to ever change ? The communities are vastly different in their goals despite some minor overlap in their theoretical foundations. Suppose you take rnorm(100) sample and find its variance. Then you ask the crowd the mean and variance of that sample variance. If your crowd is a 100 professional statisticians with a degree in Statistics, you should get the right answer atleast 90% of the time. If instead you have a 100 ML professionals with some sort of a degree in cs/vision/nlp, less than 10% would know how to go about computing the variance of sample variance, let alone what distribution that belongs to. The worst case is 100 self-taught Valley bros - not only will you get the wrong answer 100% of the time, they’ll pile on you for gatekeeping and computing useless statistical quantities by hand when you should be focused on the latest and greatest libraries in numpy that will magically do all these sorts of things if you invoke the right api. As a statistician, I feel quite sad. But classical stats has no place in what passes for ML these days. Folks can’t Rao Blackwellize for shit, how can you expect a Fisher Information matrix from them ?

[it_does_follow]

I think Bishop et al. WIP book Model-Based Machine Learning[0] is a nice step in the right direction. Honestly the most important thing missing from ML that stats has is the idea that your model is a model of something. That how you construct a problem mathematically says something about how you believe the world works. Then we can ask all sorts of detailed question about "how good is this model and what does it tell me?"

I'm not sure this will ever dominate. As much as I love Bayesian approaches I sort of feel there is a push to make them ever more byzantine, recreating all of the original critiques of where frequentist stats had gone wrong. So essentially we're just seeing a different orthodoxy dominant thinking with all of the same trapping of the previous orthodoxy.

0. https://www.mbmlbook.com/

[kuloku]

What would you advise to ML professionals to do to improve their knowledge of statistics? Some recommended books?

[jstx1]

Wait, what’s the problem with people not knowing things that they don’t need to know? This just comes across as being bitter that self taught people exist, or that other people are somehow encroaching on your field.

[yldedly]

To get the prediction variance in a Bayesian treatment, you integrate over the posterior of the parameters - surely computing or approximating the posterior counts as considering parameter variance?

[yellowcake0]

Although this is technically true, in practice probabilistic machine learning makes use of "un-priored" parameters all the time.

[barrenko]

Do you think this book is useful for someone just looking to get more into statistic and probability sans machine learning? How would I go about that?

Currently I have lined up - Math for programmers (No starch press), Practical Statistics for data scientists (O'Reily - the crab book), and Discovering Statistics using R.

Basically I'm trying to follow the theory from "Statistical Consequences of Fat Tails" by NNT.