Is there (some form of) penetration testing in aviation?

Question

I just listened to the first Miles to go podcast about a near miss at SFO where an Air Canada flight nearly landed on a taxiway where 4 aircrafts were waiting to take off. The incoming Air Canada flight wrongly aligned to land on the taxiway, apparently because it got confused by the lighting. Thankfully, someone noticed something was off at the last second, and the plane went around without damage.

The way I understand the podcast, it was a series of edge cases that lined up to lead to the situation. In short (please edit for accuracy if needed): - one of the runways had its lighting off - the aircraft had slightly old navigation equipment - the air traffic controller was alone, and on the phone - ...

Post incident, the inflight recorder wasn't turned off, so the critical part of the conversation was apparently recorded over, etc...

This all looks very similar to how computer security problems happen: if you input a specific value in this place at that specific moment, under this particular set of circumstances, you can get access to some confidential data.

The software industry has developed penetration testing as a method to find out such problems, where essentially you pay people to try to break into your systems, and possibly reward them every time they find a security problem. It seems like the situation above could have been identified by a similar method.

Does the aviation industry use similar practices? How does it work and what is it called? And if not, what prevents it?

Answer 1

There is lots of testing in aviation, in every step of the process from paper designs to an actual airframe. For situation based training the aviation industry turns to the tried and true method of

Simulation

This is similar to software in many regards (and often uses computer based simulations). Simulator tech has come a long way in the past 35 years and full motion simulators are a great way to get very very close to the real thing. If you want to test out a situation where you knock out some random approach lights, throw in low visibility and some wind shear there is no reason to risk an actual aircraft, and a pilots life. Just queue it up in the sim and see how they do.

When it comes to the physical testing of say an airframe the FAA (and other bodies) also tend to take the "...before we let it out there approach" to testing and certification. In other words before an airframe is certified for production it will undergo extensive flight testing that physically pushes the airframe and exposes it to situations it should never encounter in normal flight. Omega Tau does a great episode with an Airbus test pilot that goes deep into whats involved in this, its worth a listen.

The biggest difference in aviation is that you have control over the end to end situation. In computer security you simply cant tell people "please be careful, if you use this encryption it may break in the following cases and we dont know how to fix it". On the contrary in aviation you have the ability to train pilots and more importantly you have the ability to make (or enforce) a no-go call. If we know an approach system only works down to certain minimums you simply don't fly if the weather is below them, unlike in computer security where the system is expected to always work and always be secure and in a lot of instances, always be up.

Answer 2

There isn't true penetration testing for the scenarios you're describing in aviation, from my understanding. Penetration testing and cyber security is about how leniency and bugs in the system allow deliberate bad actors to compromise confidentiality, integrity, or availability (along with a few other things like privacy and non-repudiation). Typically penetration testers are limited from doing real damage and can only do things like obtain access to a restricted system or view confidential data (under strict privacy and NDA agreements).

The deliberateness is important- hackers often utilize small amounts of information gleaned or small errors to create big problems, and headline-worthy hacks involve the hacker taking full control of his target. The basic principle is that if thousands of people are attempting to break your security, you should have a few guys on the inside testing the same thing.

The kind of scenario you're talking about is different for several reasons:

• It wasn't deliberate
• Testing out the scenario in the real world (to capture real issues like distracted controllers) would risk a lot more than confidential data being visible to contracted testers
• There were some preventative systems which had issues and violated security assumptions like the Airport Surface Surveillance Capability system, but the bugs or ways to get around these systems don't seem to be the main factor in this incident.
• This is about a potential crash, not a security issue. It's not someone getting information they shouldn't, controlling something they shouldn't have access to, or (not really) about someone going somewhere they shouldn't.

There is risk and hazard analysis, human factors analysis, and worst-case scenario simulation, but those aren't the same as penetration testing. While there are a small number of cases of deliberate bad actors in aviation (e.g. the Horizon plane stolen in August 2018, GermanWings Flight 9525, 9/11 attacks, etc.), and there is increasing concern for cyber security in connectivity hardware, the scenario you describe is not one of them.

Answer 3

The closest it comes to penetration testing would be military exercises against OPFOR equipment. This would appear lackluster compared to the pen-testing done in cyber security. Throughout the Cold War, Soviet aircraft were simulated by old and wrong Western ones, even when it was known that the substitution was inadequate.

An example of ignoring the threat model is the West's tunnel vision on X-band RF in pre-2010s design despite the Soviets extensively using C, S, L bands and IR systems, and on Soviet end, their blindness to LO technology even when it was an open secret. In contrast, IT security pen-testing employs the tools and methods as close as possible to those used by actual threat actors, occasionally complemented by personal experience from heel-face turned black hats.

There isn't much to do pen-testing in civil aviation on, except the cockpit door locks and airport security. And each time the latter gets bypassed, it comes as a complete surprise to everyone, confirming that the principle of penetration testing is quite alien to the industry.

Pen-testing is not an effective method for preventing accidents as opposed to deliberate actions. Most accidents involve a compounding chain of small errors and bad decisions. In contrast, deliberate attacks focus on finding a short, reproducible chain of compromises that breaks the system.

Modern aviation has near-zero resistance to deliberate destructive actions by the pilots or even mechanics, or anyone with modest resources and a will to die. The aviation "pen-testing" equivalent would be for a pilot to wrestle the flight control from normal law into direct law - a much easier task than breaking into even the most vulnerable system.

On the other hand, it has very good resistant to accidents - much better than common for IT security. Defining inf-sec failure as data leaks, "accidents" would include confidential data on USB sticks, unlocked computers, passwords on post-it notes, shared accounts, documents in the trash. That's right, the norm. Not in aviation.

If aviation safety was run like IT security, planes would take an iris scan and OTP to get in, but then routinely take off with two blown tires, 3 out of 4 engines, one of them a legacy model, the PIC would go for a drink in first class right after rotation, and you'd only call ATC when you forgot your destination navpoint. Needless to say, after verifying your identity on an encrypted channel, ATC would promptly ask you in a strong Bangladesh accent to please hold and try relighting your engines.

If IT security was run like aviation safety, there would be no passwords, but each computer would only ever run job-specific software, which would have twice the learning curve of SAP R/3 with all the vowels removed.

It's not that one is worse or better - safety and security are just different.