Hi, hobbyist here! This is a huge area where government meteorologists and "Big Weather" differ and you can help close that gap!
For context:
The governments of the world provides these big weather companies (weather.com (cough IBM), Accuweather (cough IBM cough), etc) a metric shit ton of their data completely for free (by law) including data transfer. These are things like radar, satellite, ground station data, forecasts, composite models, etc. These companies profit substantially on it, as in billions of dollars. You as citizens also can get this data completely for free as well! MADIS is a system the government is working on to make that data access easier by bringing together many of these systems together and removing the bureaucratic redundancy and abstracting out the aging infrastructure. This is literally terabytes of data per day you can grab with almost no questions asked. That data is then processed privately and resold and repackaged to the end user, and you probably interact with this privatized data the most.
The frustration I have much of the additional "value" these weather data brokers provide is by linking up with each other with data contracts. These private companies have a much much higher detail on the ground than the government by being able to partner with companies that make common internet-connected personal radar stations and reselling that data to each other. The government doesn't have that privilege to buy limitless data. NOAA/NWS, for example, is extremely underfunded so if they had to privilege to buy it they probably couldn't come to an agreement to buy it. As a result, they can't use that data to improve the accuracy of alerts/warnings/forcasts, the same exact tools that the big weather companies make all their money from. It's a shit cycle and totally unfair IMO.
So please contribute if you can!!
Sidebar: I'm a founder of a self-bootstrapped startup to build a better weather data broker that doesn't cost an arm and a leg. If that's something you are passionate about solving, feel free to reach out :)
Hi, I am building an Open-Source Weather API aggregating open-data weather models from NOAA, ECMWF, DWD, MeteoFrance, JMA, CMA, CMCC and others. I agree that many weather companies basically redistribute NWS data at a premium. There is a free API service available on https://open-meteo.com and all databases are redistributed via an AWS Open-Data Sponsorship. Feel free to reach out if you need help building your weather data broker startup
As my own sidebar, I spent many years at a national lab working with distributed sensor networks (primarily ATC and other radar for detecting non-weather stuff :) ). I thought about using ADS-B as input for weather state and forecasts, but never got around to trying it. Now that I am working on my own startup (self-funded and without revenue so far), so this again will likely languish in my todo list. If someone wants to try it, great, and feel free to reach out as I can probably save you some time selecting and interpreting the right ADS-B fields:
We have a lot of aircraft blasting ADS-B reports whenever they fly. Most reports contain (1) accurate 3D position, including altitude and (2) barometric altitude measurements, which gives you (after some minimal work) air pressure. So you have millions freely available pressure reports not just on the ground, but throughout 0-40000 ft altitude band.
You also get measured airspeed and groundspeed, so in addition to pressure you get wind vectors at thousands of points in the air, updating in real time. I suspect this can provide some non-trivial information and I am not aware of anyone actually using it for this purpose.
> These private companies have a much much higher detail on the ground than the government by being able to partner with companies that make common internet-connected personal radar stations and reselling that data to each other.
I haven't heard of personal radar stations, and wasn't hitting anything in a quick web search. Are you able to provide an example of these systems?
> Sidebar: I'm a founder of a self-bootstrapped startup to build a better weather data broker that doesn't cost an arm and a leg. If that's something you are passionate about solving, feel free to reach out :)
Would love to hear more about this. I’m a researcher and a lot of my work revolves around machine learning applications to building energy modeling, and one of my projects actually revolves around the difference between using TMY vs AMY EPW files in automatic calibration of models. Would be great to chat more. What’s the name of your startup? I can shoot you an email at the official email.
I have always wanted to build a cheap ESP based weather station I can give to a friend, have them set up their WiFi and it would transmit data to the central server and I do not want to submit data to AccuWeather or wunderground or someone else.
Is there a Foss server around that I can set up on my own ?
I live in Portugal, and not only do I have a weather station, I’ve set up flood gauges with GSM uplinks along several local rivers - because I live in a watermill and I want to know when we’re about to have a bad time, after getting hit by a centenary flood.
Nobody wants the data, sadly - there used to be a national monitoring system on rivers up and down the country, but after 2008 it was abandoned, and now there’s only limited monitoring of major rivers, and none of tributaries. The IPMA and APA eventually responded and said thanks for your interest, but we have no means to ingest third party data.
I wish it could be useful for someone other than myself - but short of building my own platform and ingesting the APA’s copyrighted data, it ain’t happening.
It's scary how easily this happens. Even America briefly reduced funding to the USGS during the Trump years, among other agencies that saw budgets cut for anything that collects data on the environment. The impact was immediate and now certain important time series including stream gauges and ground water levels have years of missing data because those programs were shut down.
I want your API to not return 500 errors for hours at a time, multiple times a day...
Today it was down from 9AM to 2PM, and then from 9:30PM to 11PM.
Yesterday was even worse.
It isn't rate throttling because I only try to retrieve current conditions every 30 minutes.
Also, the load of the API might be a lot lower if it were possible to retrieve just the current measurements, but the response you get back for observations is insanely bloated with 12 hours back of each measurement, plus a bunch of station information that is heavily duplicative of information provided about the station.
Nothing about the station should be returned when you ask for the station's measurements should be returned except a link to API call for the station's information.
They are migrating the api to a new domain and browning out the old one to get folks to switch over and figure out if there are any issues. There is also another infrastructure change going on as well.
Normally those would be posted on https://www.weather.gov/notification/ but that’s having hiccups, but the email list for announcements still works.
I'm fairly sure weather.gov's api doesn't have an SLA of any kind. I highly recommend going through the push vs polling approach if possible and store the data as it comes in live in your own DB. AWS Open Data + MADIS are good sources that are... for reasons... far more stable
EDIT: sorry, I misread your commend. If it's only once every 30 secs, that may be a bit overkill for your needs. You may be able to get the appropriate text product directly and parse it out. Unfortunately probably the most stable, yet clunkiest, way
There used to be this amazing hobbyist weather station sharing site. Weather Underground.
But like so many they turned commercial and kicked off all the hobbyists that made them big. It was a really nasty move. I have never found a suitable replacement. I don't know what it is about weather that gives people so much greed.
I have sent my weather data with my Ambient station to wunderground for the last 7 years without issue. It has been used to track local weather patterns due to hurricanes and tropical storms too
This may be a dumb question but: What is a weather station exactly? How much does the hardware cost to install and then set up an interface to send data to this system?
I own this* one and it is set up on a pole off of my deck. I connect it to wunderground.com and anyone can look up data from my station. I'll be sharing with this service as well now that I know about it.
There are much cheaper options than that one if you're interested in hosting a weather station. I decided to install it since I live in a relatively rural area and there is a fairly large gap in coverage where I live. Now I can get more local data and my neighbors can benefit as well. If you live in an urban area odds are a lot of the stations you see on the coverage map are just like this. When I lived in Austin I had a lot nearby readings and if you dig into the info tab for each one you'll see they're personal stations people are sharing.
I’ve had a Weatherflow Tempest [1] for a few years and love its simplicity. No moving parts, solar and rechargeable battery powered.
Their platform is primarily mobile app based but they have a comprehensive API should you want to play around.
I have a Davis Vantage Vue. It's a nice station, but requires a proprietary data logger to get a serial connection which I loathe. There's tons of good weather station nerd info on wxforums[0].
weewx[1] software supports sending data to CWOP and has drivers for the Davis station among others.
Bonus: you can also send data to CWOP over the ham bands via aprs, and all your data shows up on aprs.fi[2]
The CWOP community likely has forums and users experienced in matching suggestions to new member questions.
Depends a lot on your skill | interest levels - the controlbyweb path is one option, and that's $450 pluse extra for sensors ( tempreture, wind speed | direction ), air pressure, etc.
All in ~ $1,100 (US) - controller + base sensors + shipping + some wiring and fitting at your site.
There are likely other products and various levels of kit build, solder your own, etc.
The San Diego Rowing Club has an Ambient Weather WS-2902 weather station and live camera feed [1] and the details of how we collect, process, and present the data. [2] If one clicks on the static image you'll get a live feed. The static image is updated frequently and is a better mobile experience. The AW API can be problematic at times so later this summer I'll reconfigure the station to do a GET request to a local device and process the data without a cloud dependency.
His has been working for 20 years now with no problems. There are cheaper ones no doubt, but the ability to add senors like a UV and solar wattage are nice. They also have sensors for soil moisture, and leaf transpiration, but it doesn't rain enough in norcal for those.
At some point, Davis changed the data logger, so I had to buy some USB adapter from some guy in Australia to unlock it, but it works fine. I don't remember the guy's name, or if it's still needed.
I use weewx and a raspberry pi to log everything. Integrated my PurpleAir into weewx and calculate AQIs as well.
I have an Ambient Weather WS-2902C feeding to CWOP out of the box. It also lets you configure custom endpoints (REST based), so I have the data feeding into my own system and logging into a db that I can use however I want.
Inside its esp8266 based. Currently sells for less than $200.
I have two KestrelMet 6000s, build quality seems very nice and I’ve been happy with them. The one stopped connecting for some reason for a little bit, but it’s been fine since, think the battery just got drained with so many cloudy days.
It would be great if this could be extended with air pollution data. Combined with weather data like wind speed and wind direction, the data could be of great use in forecasting for example air pollution from wildfires.
There are some platforms for air pollution like OpenAQ [1] or sensor.community [2] but as far as I know none really has such an open API like weather.gov and allows individual contributions.
As opposed to purchasing a personal weather station, is there anything in terms of building your own weather station (buying a raspberry pi and sensors)?
I wonder what the impact on accuracy will be. I know a few people who collected rainfall amounts and wanted to share that information with some org (can't remember who) only to be turned away because they didn't use the one approved collection device. The approved collection device was expensive and not attractive, but they were told that all the devices had to be identical to make sure the data collected was accurate and that was for something no more complicated than a measuring cup. What happens when the cheap Chinese weather station I get on amazon gives wildly different data from every other cheap Chinese weather station people got on amazon?
What a fascinating topic I knew nothing about. I am so grateful for HN users that share their knowledge so generously, unlike some weather companies. I've never heard anybody mention the obvious fact how s**y all the weather and pollen companies are.
One thing I didn't see anybody else mention, is your local Air Force unit does an expert job at delivering daily commercial-free weather reports.
Anyone with their own weather station, can you use it to improve forecasts, or just for more accurate 'actual' measurements? (if you somehow use it to enhance forecast accuracy, curious to hear how to you do it)
I have some remote forest land I would love to set some sensors up on. I just need to know exactly what to buy to gather the data, power it with solar, and send the data back via cellular signal.
Unfortunately it's too remote to have any sort of internet signal beyond cellular at the moment.
I'd appreciate any and all ideas folks have, assuming I'm willing to drop a few grand without blinking.
Additionally, it's in an area with wildfire smoke part of the year and id love to have AQI monitoring.
I've long had a theory that aggregating these weather sources to provide higher and more accurate models could create serious alpha for commodities and energy trading. Especially in nodal and ancillary markets.
I'm sure there are people here working at funds that leverage weather data to do exactly that. It would be a really cool job.
This is an absolutely fascinating topic I never knew about. I am grateful for HN users that share their knowledge so generously, unlike those weather companies.
One thing I didn't see discussed – your local Air Force unit does a very, very detailed and commercial-free weather reports daily.
Semi related, why does it seem like the forecast, current temp, and historical temp, are always 10c lower than reality in summer? Happens across a few locations I monitor, both urban and rural.
I’ll often read 41c outside, 35c in shade, but the weather says it’s currently 29c.
What's the chances home weather stations have some bias in their signal (e.g. likely to be mounted on a hot roof) and that these data points get mixed into a larger corpus and someone ends up drawing incorrect conclusions from that corpus? Hopefully not high?
[+] [-] mikeortman|1 year ago|reply
For context:
The governments of the world provides these big weather companies (weather.com (cough IBM), Accuweather (cough IBM cough), etc) a metric shit ton of their data completely for free (by law) including data transfer. These are things like radar, satellite, ground station data, forecasts, composite models, etc. These companies profit substantially on it, as in billions of dollars. You as citizens also can get this data completely for free as well! MADIS is a system the government is working on to make that data access easier by bringing together many of these systems together and removing the bureaucratic redundancy and abstracting out the aging infrastructure. This is literally terabytes of data per day you can grab with almost no questions asked. That data is then processed privately and resold and repackaged to the end user, and you probably interact with this privatized data the most.
The frustration I have much of the additional "value" these weather data brokers provide is by linking up with each other with data contracts. These private companies have a much much higher detail on the ground than the government by being able to partner with companies that make common internet-connected personal radar stations and reselling that data to each other. The government doesn't have that privilege to buy limitless data. NOAA/NWS, for example, is extremely underfunded so if they had to privilege to buy it they probably couldn't come to an agreement to buy it. As a result, they can't use that data to improve the accuracy of alerts/warnings/forcasts, the same exact tools that the big weather companies make all their money from. It's a shit cycle and totally unfair IMO.
So please contribute if you can!!
Sidebar: I'm a founder of a self-bootstrapped startup to build a better weather data broker that doesn't cost an arm and a leg. If that's something you are passionate about solving, feel free to reach out :)
[+] [-] open-meteo|1 year ago|reply
[+] [-] ptero|1 year ago|reply
As my own sidebar, I spent many years at a national lab working with distributed sensor networks (primarily ATC and other radar for detecting non-weather stuff :) ). I thought about using ADS-B as input for weather state and forecasts, but never got around to trying it. Now that I am working on my own startup (self-funded and without revenue so far), so this again will likely languish in my todo list. If someone wants to try it, great, and feel free to reach out as I can probably save you some time selecting and interpreting the right ADS-B fields:
We have a lot of aircraft blasting ADS-B reports whenever they fly. Most reports contain (1) accurate 3D position, including altitude and (2) barometric altitude measurements, which gives you (after some minimal work) air pressure. So you have millions freely available pressure reports not just on the ground, but throughout 0-40000 ft altitude band.
You also get measured airspeed and groundspeed, so in addition to pressure you get wind vectors at thousands of points in the air, updating in real time. I suspect this can provide some non-trivial information and I am not aware of anyone actually using it for this purpose.
[+] [-] HumblyTossed|1 year ago|reply
[+] [-] snowfield|1 year ago|reply
This is the Norwegian government weather service. It's global and free for everyone. Also has fully open apis
[+] [-] _xivi|1 year ago|reply
Didn't IBM sell weather.com (and all of the assets) to a private equity? The deal was announced last year and closed in Feb 2024
https://finance.yahoo.com/news/francisco-partners-completes-...
[+] [-] DavidPeiffer|1 year ago|reply
I haven't heard of personal radar stations, and wasn't hitting anything in a quick web search. Are you able to provide an example of these systems?
[+] [-] zorm|1 year ago|reply
NOAA has already made some contracts with Spire [1] and Saildrone [2]. I am sure there are more but these are the ones most familiar to me.
Your weather data broker startup sounds very interesting!
[1]: https://spire.com/press-release/spire-global-awarded-nationa... [2]: https://research.noaa.gov/2022/08/03/noaa-and-saildrone-team...
[+] [-] szvsw|1 year ago|reply
Would love to hear more about this. I’m a researcher and a lot of my work revolves around machine learning applications to building energy modeling, and one of my projects actually revolves around the difference between using TMY vs AMY EPW files in automatic calibration of models. Would be great to chat more. What’s the name of your startup? I can shoot you an email at the official email.
[+] [-] 2Gkashmiri|1 year ago|reply
Is there a Foss server around that I can set up on my own ?
[+] [-] madaxe_again|1 year ago|reply
Nobody wants the data, sadly - there used to be a national monitoring system on rivers up and down the country, but after 2008 it was abandoned, and now there’s only limited monitoring of major rivers, and none of tributaries. The IPMA and APA eventually responded and said thanks for your interest, but we have no means to ingest third party data.
I wish it could be useful for someone other than myself - but short of building my own platform and ingesting the APA’s copyrighted data, it ain’t happening.
[+] [-] jeffbee|1 year ago|reply
It's scary how easily this happens. Even America briefly reduced funding to the USGS during the Trump years, among other agencies that saw budgets cut for anything that collects data on the environment. The impact was immediate and now certain important time series including stream gauges and ground water levels have years of missing data because those programs were shut down.
[+] [-] KennyBlanken|1 year ago|reply
I want your API to not return 500 errors for hours at a time, multiple times a day...
Today it was down from 9AM to 2PM, and then from 9:30PM to 11PM.
Yesterday was even worse.
It isn't rate throttling because I only try to retrieve current conditions every 30 minutes.
Also, the load of the API might be a lot lower if it were possible to retrieve just the current measurements, but the response you get back for observations is insanely bloated with 12 hours back of each measurement, plus a bunch of station information that is heavily duplicative of information provided about the station.
Nothing about the station should be returned when you ask for the station's measurements should be returned except a link to API call for the station's information.
[+] [-] abkfenris|1 year ago|reply
Normally those would be posted on https://www.weather.gov/notification/ but that’s having hiccups, but the email list for announcements still works.
A few of the recent announcements: - https://www.weather.gov/media/notification/pdf_2023_24/scn24... - https://www.weather.gov/media/notification/pdf_2023_24/scn24... - https://www.weather.gov/media/notification/pdf_2023_24/scn24...
[+] [-] mikeortman|1 year ago|reply
EDIT: sorry, I misread your commend. If it's only once every 30 secs, that may be a bit overkill for your needs. You may be able to get the appropriate text product directly and parse it out. Unfortunately probably the most stable, yet clunkiest, way
[+] [-] zorm|1 year ago|reply
[+] [-] parkrrr|1 year ago|reply
/stations/{stationId}/observations/latest ?
[+] [-] renewiltord|1 year ago|reply
Strange that they have outage information via radio. What endpoints were failing? I'll email and see if they'll accept some help.
[+] [-] kmbfjr|1 year ago|reply
The problem however, is the 500 or target station readings returning null.
[+] [-] wkat4242|1 year ago|reply
But like so many they turned commercial and kicked off all the hobbyists that made them big. It was a really nasty move. I have never found a suitable replacement. I don't know what it is about weather that gives people so much greed.
[+] [-] Brybry|1 year ago|reply
The only "bad" thing Weather Underground did was they ended their free API for non-contributors. [1]
I believe personal weather station contributors still get free access to a different API that wunderground moved to. [2][3]
[1] https://archive.md/xe9B1
[2] https://docs.google.com/document/d/1eKCnKXI9xnoMGRRzOL1xPCBi...
[3] https://github.com/cytech/Home-Assistant-wundergroundpws/?ta...
[+] [-] screamingninja|1 year ago|reply
Yes, their acquisition by IBM really messed things up. I went from totally loving the app to uninstalling it and never looking back.
Definitely going to feed my local data to weather.gov.
[+] [-] firesteelrain|1 year ago|reply
[+] [-] hackernewds|1 year ago|reply
[+] [-] czhu12|1 year ago|reply
[+] [-] kyteland|1 year ago|reply
* https://ambientweather.com/ws-5000-ultrasonic-smart-weather-...
There are much cheaper options than that one if you're interested in hosting a weather station. I decided to install it since I live in a relatively rural area and there is a fairly large gap in coverage where I live. Now I can get more local data and my neighbors can benefit as well. If you live in an urban area odds are a lot of the stations you see on the coverage map are just like this. When I lived in Austin I had a lot nearby readings and if you dig into the info tab for each one you'll see they're personal stations people are sharing.
[+] [-] pavel_lishin|1 year ago|reply
[+] [-] rcdemski|1 year ago|reply
[1] https://tempest.earth/
[+] [-] thcipriani|1 year ago|reply
weewx[1] software supports sending data to CWOP and has drivers for the Davis station among others.
Bonus: you can also send data to CWOP over the ham bands via aprs, and all your data shows up on aprs.fi[2]
[0]: <https://www.wxforum.net/>
[1]: <https://weewx.com/>
[2]: <https://aprs.fi/>
[+] [-] defrost|1 year ago|reply
Citizen Weather Observer Program (CWOP) Information: http://www.wxqa.com/cwop_info.htm
If you're going digital, there are several suggested backbones, eg:
Legacy (oops): https://controlbyweb.com/legacy/x320m/
Current model ($450 US): https://controlbyweb.com/accessories/davis-suite-controller
which is a hub that uploads data from attached sensors, which in turn has details of compatable sensors:
Legacy manual: https://controlbyweb.com/wp-content/uploads/2024/01/X-320_um...
Current Integrated bunch of sensors on a stick ( $595 US ): https://controlbyweb.com/accessories/sensor-suite
The CWOP community likely has forums and users experienced in matching suggestions to new member questions.
Depends a lot on your skill | interest levels - the controlbyweb path is one option, and that's $450 pluse extra for sensors ( tempreture, wind speed | direction ), air pressure, etc.
All in ~ $1,100 (US) - controller + base sensors + shipping + some wiring and fitting at your site.
There are likely other products and various levels of kit build, solder your own, etc.
The Complete Tempest Weather System https://shop.tempest.earth/collections/featured/products/tem...
comes in at $339 (US) which is cheaper to get started and promises much - build quality, robustness, etc would have to be compared | asked about.
[+] [-] dano|1 year ago|reply
1. https://www.sandiegorowing.org/weather/
2. https://www.sandiegorowing.org/weather/tech/
[+] [-] jonathankoren|1 year ago|reply
https://www.davisinstruments.com/pages/vantage-pro2
His has been working for 20 years now with no problems. There are cheaper ones no doubt, but the ability to add senors like a UV and solar wattage are nice. They also have sensors for soil moisture, and leaf transpiration, but it doesn't rain enough in norcal for those.
At some point, Davis changed the data logger, so I had to buy some USB adapter from some guy in Australia to unlock it, but it works fine. I don't remember the guy's name, or if it's still needed.
I use weewx and a raspberry pi to log everything. Integrated my PurpleAir into weewx and calculate AQIs as well.
http://weewx.com/
[+] [-] kQq9oHeAz6wLLS|1 year ago|reply
Inside its esp8266 based. Currently sells for less than $200.
[+] [-] catgirlinspace|1 year ago|reply
[+] [-] subhro|1 year ago|reply
[+] [-] thallium205|1 year ago|reply
[+] [-] ahaucnx|1 year ago|reply
There are some platforms for air pollution like OpenAQ [1] or sensor.community [2] but as far as I know none really has such an open API like weather.gov and allows individual contributions.
[1] https://openaq.org/
[2] https://sensor.community/en/
[+] [-] ttymck|1 year ago|reply
[+] [-] autoexec|1 year ago|reply
[+] [-] missjuliekay|1 year ago|reply
One thing I didn't see anybody else mention, is your local Air Force unit does an expert job at delivering daily commercial-free weather reports.
[+] [-] nomilk|1 year ago|reply
[+] [-] singleshot_|1 year ago|reply
[+] [-] yareal|1 year ago|reply
Unfortunately it's too remote to have any sort of internet signal beyond cellular at the moment.
I'd appreciate any and all ideas folks have, assuming I'm willing to drop a few grand without blinking.
Additionally, it's in an area with wildfire smoke part of the year and id love to have AQI monitoring.
[+] [-] Aeroi|1 year ago|reply
I'm sure there are people here working at funds that leverage weather data to do exactly that. It would be a really cool job.
[+] [-] ulrischa|1 year ago|reply
[+] [-] gtzi|1 year ago|reply
[+] [-] zdw|1 year ago|reply
Most participants are located in Arizona (project is run by University of Arizona), and kind of makes sense because rain is important in the desert.
[+] [-] missjuliekay|1 year ago|reply
One thing I didn't see discussed – your local Air Force unit does a very, very detailed and commercial-free weather reports daily.
[+] [-] reustle|1 year ago|reply
I’ll often read 41c outside, 35c in shade, but the weather says it’s currently 29c.
[+] [-] edgineer|1 year ago|reply