I've not seen a detailed breakdown on how the NOTAM system works, but we do know three things
NOTAMs are issued from a database
The FAA said preliminary indications “traced the outage to a damaged database file.”
This system is really old
When the Transportation Department made its annual budget request to Congress last year for the FAA, it sought $29.4 million to accelerate development of a program that would let the agency eliminate what it called “failing vintage hardware” that currently supports the Notam alert system. The system has failed at least two other times since early 2021, industry officials said, though in those cases the FAA was able to avoid widespread impacts to flights.
Possibly 30 years old
The Federal Aviation Administration software that failed Wednesday causing thousands of flight delays and cancellations is 30 years old and at least six years away from being updated, a government source familiar with the situation tells CNN.
An employee replaced files, causing the outage
The issue apparently occurred during routine scheduled systems maintenance, a senior official briefed on the internal review told ABC News. An engineer "replaced one file with another," the official said, not realizing the mistake was being made. As the NOTAM system began showing problems and ultimately failed, FAA staff feverishly tried to figure out what had gone wrong. Engineers and IT teams are now working to keep the system from crashing again, while also trying to figure out if there are any similar systems that could fail without redundancies.
What are they using?
The odd part is the mention of "replacing files". Now, a failed drive can cause all sorts of issues (sooner or later you have to write data to disk), but generally you don't do relational database maintenance by "replacing files". What this suggests is that they are not using a relational database at all, but an older system known as a flat file database
A flat-file database is a database stored in a file called a flat file. Records follow a uniform format, and there are no structures for indexing or recognizing relationships between records. The file is simple. A flat file can be a plain text file (e.g. csv, txt or tsv), or a binary file. Relationships can be inferred from the data in the database, but the database format itself does not make those relationships explicit.
In other words, these files are essentially stripped-down spreadsheets (i.e. comma or tab separated columns). This would better mesh with the statement about "failing vintage hardware". If this system is really old, it may be running on something written in an older language like pure COBOL or ADA. In a flat file system, you would be fiddling with files, and any errors in the file could mess the whole system up. This would also mesh well with a system in use since the early 1990s, when RDBMS systems were not yet as widely used as they are today.
This article from 2008 contains this
The system itself is called NADIN (National Aerospace Data Interchange Network). It was designed by North American Philips for the FAA in the early 1980s. The two Philips DS714/81 mainframes became operational in January 1988. The company went out of business later that year, and the FAA bought out the entire parts inventory.
1988 would put this around 35 years old. This article has this note
Isaksen uses two Philips DS714 mainframe computers as his message-switching network. The DS714 was originally manufactured in 1968 and upgraded with new processors in 1981. Since then, they have been getting increasingly harder to maintain, support and write code for.
"These machines don't really have an operating system," says Isaksen. "This was in the days when the assembly language was written directly to the machine. We have hardware that is no longer supportable."
This system was running the FAA traffic system, but this blog suggests it might have also been running NOTAM
According to this story, the system launched in 1988 on a pair of Philips DS714 mainframe computers. Philips quit the computer business shortly after these machines were installed. This article from 2005 talks about the FAA’s plan to replace the Philips mainframes and to “go online early next year [2006]”, but apparently they haven’t completed the porting project. The volume of traffic is larger than you’d expect, with 1.5 million messages per day (about 20 per second; roughly what a pizza-box Web server might be expected to handle), apparently because the network handles weather and NOTAM data as well as flight plan data.
This would reinforce my suspicions that it's a giant flat file system, since such a machine would be unlikely to run Oracle, and highly unlikely to run IBM's DB2 (which IBM sold hardware for).
