2
$\begingroup$

After a Jan 11 2023 NOTAM outage, FAA and news outlets have reported there was a "damaged database file" and a problem with the backup. A government contractor, Spatial Front, was connected with the outage, and more detail was given:

after contracting personnel “unintentionally deleted files” while working to correct synchronization between a live primary database and a backup database for the IT system.

There's been a lot of public discussion explaining what the NOTAM system is and why it's important, but I haven't been able to find technical details on the database and system. Not just "a database file".

For instance, it might be PICK/Basic on Solaris, or DB2 via Cobol on z/OS, or MS Access on Windows NT 3.51. That's the level of detail I'm curious about.

$\endgroup$
2
  • $\begingroup$ (would this be a better fit on stackoverflow? I came here because of the domain specific knowledge) $\endgroup$ Jan 31, 2023 at 5:31
  • 2
    $\begingroup$ I think this site is better. SO is for questions about programming, while this question is about how a database related to aviation is configured. $\endgroup$
    – Someone
    Jan 31, 2023 at 6:48

1 Answer 1

4
$\begingroup$

I've not seen a detailed breakdown on how the NOTAM system works, but we do know three things

NOTAMs are issued from a database

The FAA said preliminary indications “traced the outage to a damaged database file.”

This system is really old

When the Transportation Department made its annual budget request to Congress last year for the FAA, it sought $29.4 million to accelerate development of a program that would let the agency eliminate what it called “failing vintage hardware” that currently supports the Notam alert system. The system has failed at least two other times since early 2021, industry officials said, though in those cases the FAA was able to avoid widespread impacts to flights.

Possibly 30 years old

The Federal Aviation Administration software that failed Wednesday causing thousands of flight delays and cancellations is 30 years old and at least six years away from being updated, a government source familiar with the situation tells CNN.

An employee replaced files, causing the outage

The issue apparently occurred during routine scheduled systems maintenance, a senior official briefed on the internal review told ABC News. An engineer "replaced one file with another," the official said, not realizing the mistake was being made. As the NOTAM system began showing problems and ultimately failed, FAA staff feverishly tried to figure out what had gone wrong. Engineers and IT teams are now working to keep the system from crashing again, while also trying to figure out if there are any similar systems that could fail without redundancies.

What are they using?

The odd part is the mention of "replacing files". Now, a failed drive can cause all sorts of issues (sooner or later you have to write data to disk), but generally you don't do relational database maintenance by "replacing files". What this suggests is that they are not using a relational database at all, but an older system known as a flat file database

A flat-file database is a database stored in a file called a flat file. Records follow a uniform format, and there are no structures for indexing or recognizing relationships between records. The file is simple. A flat file can be a plain text file (e.g. csv, txt or tsv), or a binary file. Relationships can be inferred from the data in the database, but the database format itself does not make those relationships explicit.

In other words, these files are essentially stripped-down spreadsheets (i.e. comma or tab separated columns). This would better mesh with the statement about "failing vintage hardware". If this system is really old, it may be running on something written in an older language like pure COBOL or ADA. In a flat file system, you would be fiddling with files, and any errors in the file could mess the whole system up. This would also mesh well with a system in use since the early 1990s, when RDBMS systems were not yet as widely used as they are today.

This article from 2008 contains this

The system itself is called NADIN (National Aerospace Data Interchange Network). It was designed by North American Philips for the FAA in the early 1980s. The two Philips DS714/81 mainframes became operational in January 1988. The company went out of business later that year, and the FAA bought out the entire parts inventory.

1988 would put this around 35 years old. This article has this note

Isaksen uses two Philips DS714 mainframe computers as his message-switching network. The DS714 was originally manufactured in 1968 and upgraded with new processors in 1981. Since then, they have been getting increasingly harder to maintain, support and write code for.

"These machines don't really have an operating system," says Isaksen. "This was in the days when the assembly language was written directly to the machine. We have hardware that is no longer supportable."

This system was running the FAA traffic system, but this blog suggests it might have also been running NOTAM

According to this story, the system launched in 1988 on a pair of Philips DS714 mainframe computers. Philips quit the computer business shortly after these machines were installed. This article from 2005 talks about the FAA’s plan to replace the Philips mainframes and to “go online early next year [2006]”, but apparently they haven’t completed the porting project. The volume of traffic is larger than you’d expect, with 1.5 million messages per day (about 20 per second; roughly what a pizza-box Web server might be expected to handle), apparently because the network handles weather and NOTAM data as well as flight plan data.

This would reinforce my suspicions that it's a giant flat file system, since such a machine would be unlikely to run Oracle, and highly unlikely to run IBM's DB2 (which IBM sold hardware for).

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .