top | item 37665370

(no title)

davmre | 2 years ago

Prophet has gotten a lot of attention since being released in 2017, I think because the idea of a fully automatic solution is very appealing to people. One of the original developers, Sean Taylor, recently posted a nice retrospective on the project's successes and failures: https://medium.com/@seanjtaylor/a-personal-retrospective-on-... He quotes one of his earlier tweets:

  If I could build it again, I’d start with automating the evaluation of forecasts. It’s silly to build models if you’re not willing to commit to an evaluation procedure. I’d also probably remove most of the automation of the modeling. People should explicitly make these choices.
Having worked on similar Bayesian time-series forecasting tools at Google, this matches my experience (though I've never used Prophet seriously, so please don't take this as any direct judgement of it as a software package). There is a lot of value in a framework that lets you easily experiment with different model structures (our version of this was the structural time series tools in TensorFlow Probability, see, e.g., https://blog.tensorflow.org/2019/03/structural-time-series-m...). But if you're forecasting something you actually care about, it's usually worth the time to try to understand yourself what structure makes sense for your problem, and do a careful evaluation on held-out data with respect to whatever metric you're really trying to optimize. A fully automated search over model structures is cute, but even when it works, it mostly just ends up rediscovering properties of the data you could or should have already known (e.g., of course traffic to your work-related website will have a day-of-week effect), so the cases where it really adds practical value are harder to find than you might like.

Even in the age of deep learning, I do think these relatively classical Bayesian models have a lot of value for many applications. Time-series forecasting tends to be a case where:

- you don't have a ton of iid data points (often, only a single time series),

- you'd like forecasts with principled uncertainty estimates, e.g., credible intervals, giving you a range of scenarios to plan for,

- you often do have a pretty good idea of what features are relevant to the process you're predicting, and

- you want to understand in detail what features the forecast is accounting for (and what it might be missing),

all of which play to the strengths of more classical, structured statistical models, compared to more data-hungry black-box deep learning models. So the basic ideas in Prophet and similar tools do still have a lot of relevance going forward, IMHO.

discuss

order

esafak|2 years ago

You mention classical models but Bayesian deep learning is a thing too. One can even retrofit existing DL models to obtain uncertainty estimates, at the expense of increasing (possibly doubling) the number of model parameters.

The quality of the uncertainty estimates is a question though.