"There are three parts to the problem: the system, the format, and the content. The system is actually quite amazing. The AFTN network connects every country in the world, and Notam information, once added, is immediately available to every user. Coupled with the internet, delivery is immediate.
The format is, at best, forgivable. It’s pretty awful. It’s a trip back in time to when Notams were introduced. You might think that was the 1960’s, or the 50’s. In fact, it’s 1924, when 5-bit ITA2 was introduced. The world shifted to ASCII in 1963, bring ing the Upper and Lower case format that every QWERTY key board uses today, but we didn’t follow – nope, we’ll stick with our 1924 format, thank you."
"The FAA is changing the format for Notice to Air Missions (NOTAM) to align with international standards. The transition to the new format will ensure U.S. NOTAMs are compliant with standards set by the International Civil Aviation Organization (ICAO) and promote further global harmonization among neighboring Air Navigation Service Providers (ANSPs)."
OpsGroup, the organisation behind that guide is also worth mentioning. It's a self-organised advocacy organization for flight operations personnel. Apparently they have a longstanding campaign to reform the NOTAM system (as explained in the guide).
Fun but deadly. They’re not wrong about the absolute avalanche of irrelevant data that will be used to show you “you should have known” - hence the proliferation of “interpretation” services.
This is hilarious. Really appreciate the examples of irrelevant clutter obscuring important information. Any UX designer can learn from this as an example of what not to do.
GROUND STOP ALL FLIGHTS / ALL DESTINATIONS EXCLUDES MILITARY AC AND
MEDEVAC FLIGHTS
DESTINATION AIRPORT; ALL
FACILITIES INCLUDED: ALL
GROUND STOP PERIOD: UNTIL 1430Z
REASON: EQUIPMENT OUTAGE
REMARKS: US NOTAMS SYSTEM DOWN
I'm surprised there's not more activity on this here on HN. I guess info is limited, so I found what I could from ATCSCC notices for today [1]. These are all the notices but portions only w/ redundant bits removed. See link in citation if you want full messages. Times here appear to be GMT:
004 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 00:47
THE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED TO ADDRESS THE EQUIPMENT
OUTAGE ISSUES FOR THE UNITED STATES NOTAM SYSTEM.
013 DCC 01/11/23 NOTAM SYSTEM EQUIPMENT OUTAGE_FYI 01/11/23 04:18
THE UNITED STATES NOTAM SYSTEM FAILED AT 2028Z. SINCE THEN NO NEW
NOTAMS OR AMENDMENTS HAVE BEEN PROCESSED. TECHNICIANS ARE CURRENTLY
WORKING TO RESTORE THE SYSTEM AND THERE IS NO ESTIMATE FOR
RESTORATION OF SERVICE AT THIS TIME.
014 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 04:19
HE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED TO ADDRESS THE EQUIPMENT
OUTAGE ISSUES...
015 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 07:42
THE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED TO ADDRESS THE
EQUIPMENT...
016 DCC 01/11/23 NOTAM SYSTEM EQUIPMENT OUTAGE_FYI 01/11/23 07:44
THE UNITED STATES NOTAM SYSTEM FAILED AT 2028Z. TECHNICIANS ARE
CURRENTLY WORKING TO RESTORE THE SYSTEM AND THERE IS NO ESTIMATE FOR
RESTORATION OF SERVICE AT THIS TIME. THERE IS CURRENTLY A HOTLINE IN
EFFECT WHICH HAS NAIMES/FAA FACILITIES/STAKEHOLDERS IN ATTENDANCE.
020 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 10:58
THE NOTAM OUTAGE HOTLINE IS BEING ACTIVATED...
030 DCC 01/11/23 NOTAM OUTAGE HOTLINE_FYI 01/11/23 12:37
THE NOTAM OUTAGE HOTLINE HAS BEEN TERMINATED.
In short, the terminology being used so far is "equipment failure".
I wanted to keep my above comment to the data only, so I'll post this here: "Equipment Failure" can cover just about anything. I think the "gut" reaction the public might have here when hearing that phrase is that it was physical equipment, but really it could have been software, network linkages, really anything. I hope the real cause here doesn't get buried somewhere in lack of follow up.
Also I really wish I'd been awake when the hotline was active (# was in the full notices linked) so I could call in and see what they were talking about as a silent observer. I can't tell from the context if that was a general status hotline, a failover service to provide the same data NOTAM would have done (though if that were the case then grounding flights wouldn't have been necessary) [1] or discussion of the actual problem and specific efforts to resolve.
[1] Well, only if it was a more benign & limited cause that wouldn't be expected to spill over into other systems. If the nature of the problem included the possibility of a malign actor then other vectors of attacks would not have been (and maybe still aren't, if that was the source) off the table.
The limited activity is probably because it's actually not that exciting from a technical perspective.
The answer is almost certainly something like "lack of funding forces FAA to continue using massively outdated systems and this is what happens when someone trips over the power cord of the mainframe in some basement".
It's much more interesting when companies like Google or Facebook go down, because they properly fund their infrastructure and there might be something for the HN crowd to learn.
> An FAA system outage is causing ground stops at AUS and other airports across the country. Arriving & departing passengers can expect delays this morning & through the day.
If you’re flying today, it’s gonna be a rough one.
" Normal air traffic operations are resuming gradually across the U.S. following an overnight outage to the Notice to Air Missions system that provides safety info to flight crews. The ground stop has been lifted...We continue to look into the cause of the initial problem"
Very entertaining to watch the local TV news this morning.
The reporter keeps pulling people aside as they arrive at the airport and asking them what they're going to do with all the planes grounded.
None of them know it's happening.
So far, they have all admitted that they either don't look at the news, or don't look at the local news, except for one woman who defiantly said, "It wasn't on Facebook," as if that meant that since the grounded flights weren't in her social media feed that it wasn't real.
REALLY IMPORTANT NOTAMS tend to alert pilots to RESTRICTED AIRSPACE or AREAS OF UNUSUAL AIR OPERATIONS. You can't fly anywhere near the President of the United States, the movements of which can change from day to day.
We get NOTAMS about military aircraft operations; often military airspace which is usually permitted may be shut down for a week or so.
Wildfire tanker operations or other natural emergencies can close an airspace.
It's quite dynamic, all on top of the scheduled flight plans and routes.
The NOTAM system is embarrassing. This just tops it off. Actually, our whole ATC system and the way the FAA operates is embarrassing. It all works kind of like waste management in NYC. If they were to add new tech and efficiency, fewer people would have jobs, so we keep doing it the old way to the detriment of everyone else.
It’s easy to say things are about jobs but do we have any evidence of that? There’s a modernization project underway at the FAA so I’d bet that if you looked there’s someone who’s been trying for years to get attention on it before something like this happened.
I've got a trip to take in a couple weeks. I have a choice: 4 hours of driving followed by two regional flights. Or just forget flying and drive 16 hours point to point. Between today's issues and all the headaches at YVR over the holidays, driving 16 hours seems more comfortable and less stressful than rolling the dice with regional airline travel.
I would rather be stuck driving my car for a day than spend a day sleeping on the floor of an airport lobby.
I used to need to be in a city every other week that was only a 1.5 hour flight.
I decided to see if driving might get me there in comparable time if I compared when I closed my front door to when I opened the door of my destination.
I was surprised to find that the time it took as almost the same. That's because flying requires:
* start drive to airport (45 minutes)
* park and get on a shuttle (15 minutes)
* get to gate (30 minutes)
* wait (60 minutes because I'm early by an hour)
* fly (1:45)
* taxi to gate (5-45 minutes)
* walk to light rail stop (10 minutes)
* ride light rail too stop (45 minutes)
* walk to destination (15 minutes)
If I'm understanding the situation correctly: the system stopped processing updates yesterday afternoon and it wasn't a big deal, then they tried restarting it this morning to fix it and it didn't come back up, so now older NOTAMs from before the issue started are not available. Is that right?
Do airlines/planes keep a cached copy of these? How large is the NOTAM database?
Not really. And for most commercial pilots this may come down to whether or not ForeFlight caches theirs or not. I'd actually be really curious to see unofficial commentary from ForeFlight developers about this because it's a pretty unique scenario in the age of electronic flight bags.
The problem isn't keeping a copy, it's sending updates that are very time critical (for example restricted areas around POTUS, any impactful event, etc.)
As of 06:24 US Eastern the appears to no longer be true. There is a NOTAM outage but no full ground stop is mentioned, just a few specific due to weather.
Edit: obviously a developing situation, see responses to this comment for current advisories.
I don't know how these things work, but the most recent advisory, 027, seems to imply that United Airlines has a full ground stop until 15:00 UTC / 10:00 Eastern.
Edit: Of course as I was typing that, Advisory 028 pops up with a nationwide ground stop until 9:30 Eastern: "GROUND STOP ALL FLIGHTS / ALL DESTINATIONS EXCLUDES MILITARY AC AND MEDEVAC FLIGHTS"
Wondering if this is just standard aging infrastructure embarrassment or could involve a cyberattack? Cause if I were a state actor with a beef this is exactly the sort of soft-but-expensive injury I'd want to inflict: extremely disruptive, but also hard to assess or explain to the public.
Q: Is there anything like a systems architecture diagram for how NOTAM works? I'm curious about which parts might have failed and what the moving pieces are.
>Update 5: Normal air traffic operations are resuming gradually across the U.S. following an overnight outage to the Notice to Air Missions system that provides safety info to flight crews. The ground stop has been lifted.
>We continue to look into the cause of the initial problem
> Update 4: The FAA is making progress in restoring its Notice to Air Missions system following an overnight outage. Departures are resuming at
@EWRairport
and
@ATLairport
due to air traffic congestion in those areas. We expect departures to resume at other airports at 9 a.m. ET.
> Flights to US airports have been groundstopped by the FAA until at least 09:30 ET (1430 UTC). This means any flight not in the air already will not be allowed to depart. Image: flights over US now compared to same time last week. About 700 fewer flights airborne now.
> Flights are being held as the FAA works to bring the NOTAM (NOtice To Air Missions) system back online.
> Update 3: The FAA is still working to fully restore the Notice to Air Missions system following an outage.
> The FAA has ordered airlines to pause all domestic departures until 9 a.m. Eastern Time to allow the agency to validate the integrity of flight and safety information.
> The President has been briefed by the Secretary of Transportation this morning on the FAA system outage. There is no evidence of a cyberattack at this point, but the President directed DOT to conduct a full investigation into the causes. The FAA will provide regular updates.
[+] [-] rwmj|3 years ago|reply
[+] [-] eternalban|3 years ago|reply
The format is, at best, forgivable. It’s pretty awful. It’s a trip back in time to when Notams were introduced. You might think that was the 1960’s, or the 50’s. In fact, it’s 1924, when 5-bit ITA2 was introduced. The world shifted to ASCII in 1963, bring ing the Upper and Lower case format that every QWERTY key board uses today, but we didn’t follow – nope, we’ll stick with our 1924 format, thank you."
https://web.archive.org/web/20230104130153/https://www.faa.g...
"The FAA is changing the format for Notice to Air Missions (NOTAM) to align with international standards. The transition to the new format will ensure U.S. NOTAMs are compliant with standards set by the International Civil Aviation Organization (ICAO) and promote further global harmonization among neighboring Air Navigation Service Providers (ANSPs)."
ICAO NOTAM format
https://vat-air.dk/files/ICAO%20NOTAM%20format.pdf
Example! (decoder ring not included)
"Q) EGTT/QMRXX/IV/NBO/A/000/999/5129N00028W005"
[+] [-] xg15|3 years ago|reply
They also have a summary of today's incident: https://ops.group/blog/us-flights-delayed-after-notam-system...
[+] [-] bombcar|3 years ago|reply
[+] [-] NelsonMinar|3 years ago|reply
[+] [-] dx034|3 years ago|reply
GROUND STOP ALL FLIGHTS / ALL DESTINATIONS EXCLUDES MILITARY AC AND MEDEVAC FLIGHTS DESTINATION AIRPORT; ALL FACILITIES INCLUDED: ALL GROUND STOP PERIOD: UNTIL 1430Z REASON: EQUIPMENT OUTAGE REMARKS: US NOTAMS SYSTEM DOWN
[+] [-] srmarm|3 years ago|reply
[+] [-] Tepix|3 years ago|reply
[+] [-] ineedasername|3 years ago|reply
[1] 1/11/2023 link to the day's messages w/ index numbering for each one per my above quotes" https://www.fly.faa.gov/adv/adv_list.jsp?WhichAdvisories=ATC...
[+] [-] ineedasername|3 years ago|reply
Also I really wish I'd been awake when the hotline was active (# was in the full notices linked) so I could call in and see what they were talking about as a silent observer. I can't tell from the context if that was a general status hotline, a failover service to provide the same data NOTAM would have done (though if that were the case then grounding flights wouldn't have been necessary) [1] or discussion of the actual problem and specific efforts to resolve.
[1] Well, only if it was a more benign & limited cause that wouldn't be expected to spill over into other systems. If the nature of the problem included the possibility of a malign actor then other vectors of attacks would not have been (and maybe still aren't, if that was the source) off the table.
[+] [-] sidfthec|3 years ago|reply
The answer is almost certainly something like "lack of funding forces FAA to continue using massively outdated systems and this is what happens when someone trips over the power cord of the mainframe in some basement".
It's much more interesting when companies like Google or Facebook go down, because they properly fund their infrastructure and there might be something for the HN crowd to learn.
[+] [-] zitterbewegung|3 years ago|reply
[+] [-] snake_doc|3 years ago|reply
https://www.oig.dot.gov/sites/default/files/FAA%20Actions%20...
[+] [-] controlller|3 years ago|reply
[+] [-] GalenErso|3 years ago|reply
[+] [-] ceejayoz|3 years ago|reply
> An FAA system outage is causing ground stops at AUS and other airports across the country. Arriving & departing passengers can expect delays this morning & through the day.
If you’re flying today, it’s gonna be a rough one.
edit: BBC now confirming a full ground stop. https://twitter.com/BBCBreaking/status/1613151796955856896
https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=28&adv_dat...
[+] [-] belter|3 years ago|reply
" Normal air traffic operations are resuming gradually across the U.S. following an overnight outage to the Notice to Air Missions system that provides safety info to flight crews. The ground stop has been lifted...We continue to look into the cause of the initial problem"
[+] [-] reaperducer|3 years ago|reply
The reporter keeps pulling people aside as they arrive at the airport and asking them what they're going to do with all the planes grounded.
None of them know it's happening.
So far, they have all admitted that they either don't look at the news, or don't look at the local news, except for one woman who defiantly said, "It wasn't on Facebook," as if that meant that since the grounded flights weren't in her social media feed that it wasn't real.
[+] [-] watersb|3 years ago|reply
We get NOTAMS about military aircraft operations; often military airspace which is usually permitted may be shut down for a week or so.
Wildfire tanker operations or other natural emergencies can close an airspace.
It's quite dynamic, all on top of the scheduled flight plans and routes.
[+] [-] ru552|3 years ago|reply
I used to operate a 550+ meter tower and when I had a tower light out, I had to file a NOTAM.
[+] [-] dreamcompiler|3 years ago|reply
Back when I was flying it was "Notice to Airmen." Wondered when they would get around to fixing that. Guess they finally did.
[+] [-] pookha|3 years ago|reply
What this tells me is a group of government busy-workers are playing around with complex systems that can dramatically impact millions of people.
[+] [-] wstuartcl|3 years ago|reply
[+] [-] unionemployee|3 years ago|reply
[+] [-] quantified|3 years ago|reply
This seems like the kind of critical infrastructure whose uptime requirements can make upgrading challenging once in place.
[+] [-] acdha|3 years ago|reply
[+] [-] kfrzcode|3 years ago|reply
[+] [-] dev_daftly|3 years ago|reply
[+] [-] sandworm101|3 years ago|reply
I would rather be stuck driving my car for a day than spend a day sleeping on the floor of an airport lobby.
[+] [-] toast0|3 years ago|reply
Probably less effort and time to fly, but a lot of waiting without agency.
[+] [-] buffington|3 years ago|reply
I decided to see if driving might get me there in comparable time if I compared when I closed my front door to when I opened the door of my destination.
I was surprised to find that the time it took as almost the same. That's because flying requires:
* start drive to airport (45 minutes) * park and get on a shuttle (15 minutes) * get to gate (30 minutes) * wait (60 minutes because I'm early by an hour) * fly (1:45) * taxi to gate (5-45 minutes) * walk to light rail stop (10 minutes) * ride light rail too stop (45 minutes) * walk to destination (15 minutes)
Door to door flying: about 6 hours
Driving:
* start driving (6 hours) * fuel stop (15 minutes) * park (15 minutes) * arrive
If I didn't obey speed limits, I could cut an hour or more off that 6 hours.
That said, I still preferred flying. Hard to take a nap or space out while driving for 6 hours.
[+] [-] outericky|3 years ago|reply
[+] [-] lost_tourist|3 years ago|reply
[+] [-] testplzignore|3 years ago|reply
Do airlines/planes keep a cached copy of these? How large is the NOTAM database?
[+] [-] stevehawk|3 years ago|reply
[+] [-] p_l|3 years ago|reply
[+] [-] zrail|3 years ago|reply
Edit: obviously a developing situation, see responses to this comment for current advisories.
[+] [-] callahad|3 years ago|reply
Edit: Of course as I was typing that, Advisory 028 pops up with a nationwide ground stop until 9:30 Eastern: "GROUND STOP ALL FLIGHTS / ALL DESTINATIONS EXCLUDES MILITARY AC AND MEDEVAC FLIGHTS"
https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=28&adv_dat...
[+] [-] emn13|3 years ago|reply
Clearly still developing; we'll see.
[+] [-] perihelions|3 years ago|reply
https://www.bbc.co.uk/news/world-us-canada-64236047
[+] [-] nkcmr|3 years ago|reply
It feels like not a single month can pass these days without some "Airline X has major outage and ruins tons of people's days!" story on the news.
What is the tech behind these frail systems?
[+] [-] anigbrowl|3 years ago|reply
[+] [-] jxf|3 years ago|reply
[+] [-] tyingq|3 years ago|reply
https://www.faa.gov/air_traffic/technology/swim/users_forum/...
https://www.faa.gov/sites/faa.gov/files/2021-10/NOTAM%20Mod%...
[+] [-] steponlego|3 years ago|reply
[+] [-] ceejayoz|3 years ago|reply
[+] [-] harrego|3 years ago|reply
[+] [-] TecoAndJix|3 years ago|reply
[+] [-] samwillis|3 years ago|reply
>Update 5: Normal air traffic operations are resuming gradually across the U.S. following an overnight outage to the Notice to Air Missions system that provides safety info to flight crews. The ground stop has been lifted.
>We continue to look into the cause of the initial problem
https://mobile.twitter.com/FAANews/status/161317140080139468...
---
Edit - Update from FAA:
> Update 4: The FAA is making progress in restoring its Notice to Air Missions system following an overnight outage. Departures are resuming at @EWRairport and @ATLairport due to air traffic congestion in those areas. We expect departures to resume at other airports at 9 a.m. ET.
https://mobile.twitter.com/FAANews/status/161316273572174643...
---
Flightradar24 are reporting:
> Flights to US airports have been groundstopped by the FAA until at least 09:30 ET (1430 UTC). This means any flight not in the air already will not be allowed to depart. Image: flights over US now compared to same time last week. About 700 fewer flights airborne now.
> Flights are being held as the FAA works to bring the NOTAM (NOtice To Air Missions) system back online.
https://mobile.twitter.com/flightradar24/status/161315347921...
Latest FAA update:
> Update 3: The FAA is still working to fully restore the Notice to Air Missions system following an outage.
> The FAA has ordered airlines to pause all domestic departures until 9 a.m. Eastern Time to allow the agency to validate the integrity of flight and safety information.
https://mobile.twitter.com/FAANews/status/161314857927045939...
Statement from the white house:
> The President has been briefed by the Secretary of Transportation this morning on the FAA system outage. There is no evidence of a cyberattack at this point, but the President directed DOT to conduct a full investigation into the causes. The FAA will provide regular updates.
https://mobile.twitter.com/PressSec/status/16131535612899328...
I believe this means that nothing can take off, but aircraft already in the air can continue to their planned destinations.
[+] [-] dx034|3 years ago|reply
https://www.fly.faa.gov/adv/adv_otherdis.jsp?advn=31&adv_dat...