There's a little cautionary story I like to tell about predictions and probabilities
There is a man living near a volcano. He has put up a sign outside his house for travelers, proudly declaring: "Will not erupt today. Accuracy: 100%."
One night, after thirty years of this, the volcano erupts for a few hours at night, oozing magma on its opposite side.
The next morning, the man is grateful that his house is fine, but feels a pang of sadness as he replaces the famous sign with a new one: "Will not erupt today. Accuracy rate: 99.99%."
Yes, most people can predict weather in the dessert. But why do you claim this is what happened here? Or was it a joke? Because people took it serious.
Neither the article, nor the linked paper state that. But they have all the details on precision and condition.
>Figure 2 shows some of the forecast samples of GenCast for Typhoon Hagibis, shortly before it made landfall in Japan on 12 October 2019. Figure 2b–e,g,h–k,m shows that GenCast forecasts are sharp and have spherical harmonic power spectra that closely match the ERA5 ground truth at both 1- and 15-day lead times
Seriously. Let's see an accurate forecast for Cleveland, Ohio. Even local forecasters can barely get the next day correct on any sort of consistent basis.
This is great from a practical standpoint (being able to predict weather), but does it actually improve our understanding of the weather, or WHY those predictions are better?
That is my issue with some of these AI advances. With these, we won't have actually gotten better at understanding the weather patterns, since it's all just a bunch of weights which nobody really understands.
I’m by no means an expert in weather forecasting, but I have some familiarity with the methods. My understanding is that non-“AI” weather models basically subdivide the atmosphere into a 3d grid of cells that are on the order of hundreds to thousands of meters in each dimension, treat each cell as atomic/homogeneous at a given point in time, and then advance the relevant differential equations deterministically to forecast changes across the grid over time. This approach, again based on my limited understanding, is primarily held back by the sparse resolution and the computational resources needed to improve it, not by limitations of our understanding of the underlying physics. (Relatedly, I believe these models can be very sensitive to small changes in the initial conditions.) It’s not hard to imagine a neural net learning a more efficient way to encode and forecast the underlying physical patterns.
This has been the case for years now, way before the AI craze. We just used to call it machine leaning. The best performing predictive models are black boxes which can’t practically be interpreted by humans the same way you can say a linear regression model that gives easily digestible parameters as output. Boosted trees are a great example of very well performing models that quickly become impossible for humans to understand once they get big enough to be useful.
In Australia, meteorologists used to be deployed across the country to local offices and would receive computer-generated forecast models (and other raw data) whenever the supercomputer at headquarters had finished running a job. The local meteorologists would then be allowed to apply their local knowledge to adjust the computer-generated forecast.
This was (and still is) particularly important in situations such as:
* Fast moving weather systems of high volatility, such as fire weather systems coupled with severe thunderstorms.
* Rare meteorological conditions where a global model trained on historical data may not have enough observed data points to consider rare conditions with the necessary weighting.
* Accuracy of forecasts for "microclimates" such as alpine resorts at the top of a ultra-prominent peak. Global models tend to smooth over such as an anomaly in the landscape as if the landscape anomaly was never present.[1]
It'd perhaps be possible to build more local monitoring stations to collect training data and run many local climate models across a landscape and run more climate models of specific rare weather systems. But it is also possibly cheaper and adequate (or more accurate) to just hire a meteorologist with local knowledge instead?
A fisherman was relaxing on a sunny beach, enjoying the day with his fishing line in the water. A businessman, stressed from work, walked by and criticized him for not working harder.
"If you worked more, you'd catch more fish," the businessman said.
"And what would my reward be?" asked the fisherman with a smile.
"You could earn money, buy bigger nets, and catch even more fish!" the businessman replied.
"And then what?" the fisherman asked again.
"Then, you could buy a boat and catch even larger hauls!" said the businessman.
"And after that?"
"You could buy more boats, hire a crew, and eventually own a fleet, freeing you to relax forever!"
The fisherman, still smiling, replied, "But isn't that what I'm already doing?"
Define "our understanding". With complex / chaotic systems there sometimes are no higher level laws that govern them - all we have is just modeling and prediction.
I think this will be the next generation of science to some extent. The things we can understand and call explanations/reasons might be something involving 5 or 50 variables with not too many interactions between them. They have some unifying principles or equations we can fit in our head. I think many things inherently just involve far too many variables and complexity for us to understand in a neat theory and we are hitting those limits in biology & physics. Even so I'm sure we will develop better and better ways to interpret these models and get some level of understanding. Maybe we can understand them but not create them past a certain scale.
What is there that we don't already understand? Sure there are probably higher level patterns in the weather that the model might be exploiting to make the claimed better predictions, but normal models are just based on physics and people rarely cared to try and improve them based on these patterns before. Mainly because they are too complex.
In short those patterns are only useful because of AI.
We could train AI models to simplify models in ways that require much less "intelligence" in the given domain.
For instance, we could ask AI to simplify the "essence" of the problems it solves in a similar manner to how Einstein and Feynmann simplified laws of Physics. With train/elevator metaphors or representations like Feynmann diagrams.
Of course, such explanations don't give the depth of understanding required to actually do the tensor calculus needed to USE theories like General Relativity or Quantum Electrodynamics.
But it's enough to give us a fuzzy feeling that we understand at least some of it.
The REAL understanding, ie at a level where we can in principle repeat the predictions, may require intuitions so complex that our human brain wouldn't be able to wrap itself around it.
You will be mocked by the AI hype crazies for your very serious and important question during this time.
The reality is no ,we won't learn why and how something works. And it will get much worse in the future. We are trading human learning for machine learning. As time goes on humans will get more stupid and machines more intelligent and eventually we will have generations who know nothing about how anything works and depend on the machines to tell them how to function.
One could make the argument: hey... fifty years ago everyone knew intricately how a car worked because you had to. It broke down so often, you needed to be able to repair it yourself on the side of the road. Now people just press a button and if it doesn't work you have the 'shop' take care of it for you. AI advancement will be no different.
Problem is: the 'shop' today is still humans who designed and built the cars and know how a car works and how to repair one. AI advancement can lead to eventually no one knowing anything as models get so sophisticated we just don't know why A leads to Z.
I'm friends with a meteorologist and the 15+ day forecast is the bane of their existence because you can't accurately forecast beyond a week so I would love to know how they are measuring accuracy. The article doesn't say and I know the paper is going to go over my head.
Totally wrong. You cannot generalize such a statement because it depends on the micro and macro weather conditions. A very stable situation makes it very easy to forecast one week and beyond. On the other hand, there can be situations where you cannot accurately predict the next 12 hours (e.g. cold air pool).
I would guess that everyday they're comparing the current weather against the forecast from 15 days ago. Not a lot of data points to be sure, but perhaps enough to have confidence of very high accuracy.
The paper seems quite readable to me and I also lack the training. But this point is adressed in the beginning.
"The highly non-linear physics of weather means that small initial uncertainties and errors can rapidly grow into large uncertainties about the future. Making important decisions often requires knowing not just a single probable scenario but the range of possible scenarios and how likely they are to occur."
>> We use 2019 as our test period, and, following the protocol in ref. 2, we initialize ML models using ERA5 at 06 UTC and 18 UTC, as these benefit from only 3 h of look-ahead (with the exception of sea surface temperature, which in ERA5 is updated once per 24 h). This ensures ML models are not afforded an unfair advantage by initializing from states with longer look-ahead windows.
See Baselines section in the paper that explains the methodology in more depth. They basically feed the competing models with data from weather stations and predict the weather in a certain time period. Then they compare the prediction with the ground truth from that period.
Plot twist: they measure accuracy in predicting the weather 5 years in the past.
They can say what they want, but I get rained on by surprise rain more than I ever have in my life, now that I'm practically forced to use the built-in Google weather due to them and Apple catching and killing all the good weather apps.
It does seem like this is one of those domains where new AI models could thrive. From my understanding, the amount and variety of data necessary to make these models work is huge. In addition to historical data, you've got constant satellite data, weather stations on the ground for data collection, weather balloons going high into the atmosphere multiple times daily per location, Doppler radars tracking precipitation, data from ships and other devices in the ocean measuring temps and other info, and who knows what else.
It's incredible that we are able to predict anything this far into the future, but the complexity seems like it lends itself to this kind of black box approach.
*This is all speculation, so I'd be grateful if someone more knowledgeable could tell if if I'm mistaken about these assumptions. It's an interesting topic.
>> But DeepMind said GenCast surpassed the precision of the center's forecasts in more than 97 percent of the 1,320 real-world scenarios from 2019 which they were both tested on.
I don’t really understand why Google and other companies making similar models are able to train on existing modelled or reanalysis data sets and then claim further accuracy than the originals. Sure, stacks of convolutions with multimodal attention blocks should be able to tease apart all the of idiosyncratic correlations that the original models may not have seen. But it’s unclear to me that better models is the direction to go in as opposed to better data.
Would be really cool to convert it's predictive model into a computer program that predicts written in like python/C/rust/whatever, and I think that would better serve our ability to understand the world.
I recently read that the 10 day forecast now is as accurate as the 5 day forecast from 30 years ago when I was a kid.
This surprised me. I grew up in Ohio and now I live in the Bay Area and the forecast here seems to be accurate only 2-3 days out. It would be so helpful to have an accurate 10 day forecast.
[+] [-] Ankaios|1 year ago|reply
Which I can forecast 15 days out, too.
[+] [-] Terr_|1 year ago|reply
There is a man living near a volcano. He has put up a sign outside his house for travelers, proudly declaring: "Will not erupt today. Accuracy: 100%."
One night, after thirty years of this, the volcano erupts for a few hours at night, oozing magma on its opposite side.
The next morning, the man is grateful that his house is fine, but feels a pang of sadness as he replaces the famous sign with a new one: "Will not erupt today. Accuracy rate: 99.99%."
[+] [-] lukan|1 year ago|reply
Neither the article, nor the linked paper state that. But they have all the details on precision and condition.
https://www.nature.com/articles/s41586-024-08252-9
[+] [-] jefftk|1 year ago|reply
It was trained on global data, and makes global forecasts.
[+] [-] gcanyon|1 year ago|reply
Night and morning low clouds, burning off in the afternoon.
Source: lived in Los Angeles for a dozen years, saw the weather forecast a few times
[+] [-] pharrington|1 year ago|reply
>Figure 2 shows some of the forecast samples of GenCast for Typhoon Hagibis, shortly before it made landfall in Japan on 12 October 2019. Figure 2b–e,g,h–k,m shows that GenCast forecasts are sharp and have spherical harmonic power spectra that closely match the ERA5 ground truth at both 1- and 15-day lead times
[+] [-] gaoshan|1 year ago|reply
[+] [-] RicoElectrico|1 year ago|reply
[+] [-] simonebrunozzi|1 year ago|reply
[+] [-] genewitch|1 year ago|reply
"The easiest job in the world is the weatherperson in San Diego... And now, Ryan, what's the weather going to be like today?'
'uh, nice. back to you!'"
[+] [-] atonse|1 year ago|reply
That is my issue with some of these AI advances. With these, we won't have actually gotten better at understanding the weather patterns, since it's all just a bunch of weights which nobody really understands.
[+] [-] tfehring|1 year ago|reply
[+] [-] efxhoy|1 year ago|reply
[+] [-] dhx|1 year ago|reply
This was (and still is) particularly important in situations such as:
* Fast moving weather systems of high volatility, such as fire weather systems coupled with severe thunderstorms.
* Rare meteorological conditions where a global model trained on historical data may not have enough observed data points to consider rare conditions with the necessary weighting.
* Accuracy of forecasts for "microclimates" such as alpine resorts at the top of a ultra-prominent peak. Global models tend to smooth over such as an anomaly in the landscape as if the landscape anomaly was never present.[1]
It'd perhaps be possible to build more local monitoring stations to collect training data and run many local climate models across a landscape and run more climate models of specific rare weather systems. But it is also possibly cheaper and adequate (or more accurate) to just hire a meteorologist with local knowledge instead?
[1] Zanchi, M., Zapperi, S. & La Porta, C.A.M. Harnessing deep learning to forecast local microclimate using global climate data. Sci Rep 13, 21062 (2023). https://doi.org/10.1038/s41598-023-48028-1 https://www.nature.com/articles/s41598-023-48028-1
[+] [-] CabSauce|1 year ago|reply
[+] [-] raincole|1 year ago|reply
"If you worked more, you'd catch more fish," the businessman said.
"And what would my reward be?" asked the fisherman with a smile.
"You could earn money, buy bigger nets, and catch even more fish!" the businessman replied.
"And then what?" the fisherman asked again.
"Then, you could buy a boat and catch even larger hauls!" said the businessman.
"And after that?"
"You could buy more boats, hire a crew, and eventually own a fleet, freeing you to relax forever!"
The fisherman, still smiling, replied, "But isn't that what I'm already doing?"
[+] [-] chaos_emergent|1 year ago|reply
[+] [-] dartos|1 year ago|reply
But most people just need to know if it’s going to be storming, hot or, going to rain on a given day and that is where this shines.
[+] [-] kolinko|1 year ago|reply
[+] [-] zaptheimpaler|1 year ago|reply
[+] [-] bobthepanda|1 year ago|reply
[+] [-] casey2|1 year ago|reply
In short those patterns are only useful because of AI.
[+] [-] trashtester|1 year ago|reply
For instance, we could ask AI to simplify the "essence" of the problems it solves in a similar manner to how Einstein and Feynmann simplified laws of Physics. With train/elevator metaphors or representations like Feynmann diagrams.
Of course, such explanations don't give the depth of understanding required to actually do the tensor calculus needed to USE theories like General Relativity or Quantum Electrodynamics.
But it's enough to give us a fuzzy feeling that we understand at least some of it.
The REAL understanding, ie at a level where we can in principle repeat the predictions, may require intuitions so complex that our human brain wouldn't be able to wrap itself around it.
[+] [-] criddell|1 year ago|reply
[+] [-] gniv|1 year ago|reply
[+] [-] ulfw|1 year ago|reply
One could make the argument: hey... fifty years ago everyone knew intricately how a car worked because you had to. It broke down so often, you needed to be able to repair it yourself on the side of the road. Now people just press a button and if it doesn't work you have the 'shop' take care of it for you. AI advancement will be no different. Problem is: the 'shop' today is still humans who designed and built the cars and know how a car works and how to repair one. AI advancement can lead to eventually no one knowing anything as models get so sophisticated we just don't know why A leads to Z.
[+] [-] SpaceManNabs|1 year ago|reply
[+] [-] BadHumans|1 year ago|reply
[+] [-] beernet|1 year ago|reply
Totally wrong. You cannot generalize such a statement because it depends on the micro and macro weather conditions. A very stable situation makes it very easy to forecast one week and beyond. On the other hand, there can be situations where you cannot accurately predict the next 12 hours (e.g. cold air pool).
[+] [-] n4r9|1 year ago|reply
[+] [-] lukan|1 year ago|reply
"The highly non-linear physics of weather means that small initial uncertainties and errors can rapidly grow into large uncertainties about the future. Making important decisions often requires knowing not just a single probable scenario but the range of possible scenarios and how likely they are to occur."
[+] [-] DrBazza|1 year ago|reply
https://en.wikipedia.org/wiki/Weather_forecasting#Persistenc...
IIRC the Metoffice in the UK does/did pay bonuses to staff based on modelling exceeding that criteria in a calendar year.
Again, IIRC, in the UK the persistence forecast suggests something around 200-250 days of the year have the same weather as the previous day.
[+] [-] YeGoblynQueenne|1 year ago|reply
>> We use 2019 as our test period, and, following the protocol in ref. 2, we initialize ML models using ERA5 at 06 UTC and 18 UTC, as these benefit from only 3 h of look-ahead (with the exception of sea surface temperature, which in ERA5 is updated once per 24 h). This ensures ML models are not afforded an unfair advantage by initializing from states with longer look-ahead windows.
See Baselines section in the paper that explains the methodology in more depth. They basically feed the competing models with data from weather stations and predict the weather in a certain time period. Then they compare the prediction with the ground truth from that period.
Plot twist: they measure accuracy in predicting the weather 5 years in the past.
[+] [-] Always42|1 year ago|reply
[+] [-] queuebert|1 year ago|reply
[+] [-] xnx|1 year ago|reply
[+] [-] tokai|1 year ago|reply
[+] [-] choeger|1 year ago|reply
[+] [-] ericra|1 year ago|reply
It's incredible that we are able to predict anything this far into the future, but the complexity seems like it lends itself to this kind of black box approach.
*This is all speculation, so I'd be grateful if someone more knowledgeable could tell if if I'm mistaken about these assumptions. It's an interesting topic.
[+] [-] YeGoblynQueenne|1 year ago|reply
>> But DeepMind said GenCast surpassed the precision of the center's forecasts in more than 97 percent of the 1,320 real-world scenarios from 2019 which they were both tested on.
[+] [-] amelius|1 year ago|reply
[+] [-] mnky9800n|1 year ago|reply
[+] [-] mattxxx|1 year ago|reply
Would be really cool to convert it's predictive model into a computer program that predicts written in like python/C/rust/whatever, and I think that would better serve our ability to understand the world.
[+] [-] helsinki|1 year ago|reply
[+] [-] smartmic|1 year ago|reply
[+] [-] dumbfounder|1 year ago|reply
Did the DeepMind AI say this?
[+] [-] wslh|1 year ago|reply
[+] [-] encoderer|1 year ago|reply
This surprised me. I grew up in Ohio and now I live in the Bay Area and the forecast here seems to be accurate only 2-3 days out. It would be so helpful to have an accurate 10 day forecast.
[+] [-] BrawnyBadger53|1 year ago|reply
[+] [-] Aeroi|1 year ago|reply