[cfcf14]

Amazing post, agree with everything you've said. I've always felt that the problems with advanced MCMC methods (HMC, RM-MC, etc) are even more painful when one looks at approximate bayesian methods - ADVI (variational approximation), SGLD (langevin dynamics), and so on. My grad research was on ABC-SMC methods, probably the last resort of all last resorts.

There was a period a few years ago when it was all the rage to take arbitrary probabilistic programming models and just toss ADVI at it blindly with fancy tools like Pymc3 or stan. I feel like everybody eventually came to the conclusion that if the model was simple enough that you could guarantee that ADVI was actually correct, you didn't need it, and if it wasn't, you couldn't possibly verify you were approximating anything close to the true posterior.

At least with HMC it would generally explode when dealing with pathological geometry (multimodality, non-identifiability, whatever), whereas lots of the approximate methods will 'converge' to completely incorrect answers. I know there's been some work on determining if things have gone off the rails (https://arxiv.org/pdf/1802.02538.pdf), but I couldn't ever find a place where it felt safe to use this stuff blindly.