Why are critical flight computers redundant?

Question

At least in airliners, the truly critical computers are redundant. Typically three identical copies of the autopilot computers run in parallel and compare results; if one computer disagrees with the other two, its output is ignored. The system allows some processors to be faulty while maintaining the operation of the overall system.

But why? I've never heard of microprocessors suddenly failing. Sure, there could be manufacturing errors, but those would have been caught at the factory. Perhaps the program (and its proof) is wrong, but it would be wrong in the same way across the processors. Similarly, bad input would cause bad output across all three computers. What kind of errors does this redundancy protect against? Do microprocessors sometimes just do math wrong?

If a microprocessor is overheated or overloaded and spontaneously fails, I would expect it to stop doing anything and produce no output. To deal with this kind of failure, you'd want to have a backup processor, but you wouldn't need to compare the outputs of three computers—any output produced would be assumed correct, so you'd be happy to directly use the output of any processor that was producing output.

Related: The answer to What is the purpose of multiple autopilots? simply says "redundancy" before going into how this is achieved.

Answer 1

Failure modes to consider:

• Overheating. This changes the chip's timing properties and eventually results in error. This can manifest as single-bit errors in the middle of seemingly normal operation; it will eventually crash, but may output bad data first.
• Water damage. Manifests as a parasitic resistance on the board and may cause you to misinterpret bits as high or low. May be leaky housings, condensation, etc.
• Electromagnetic interference. (The system is supposed to be resistant to this, but it's still worth thinking about).
• Physical connection problems. Either during construction (soldering faults) or induced afterwards (heat, vibration). Microscopic cracks in boards or joints can pass QA but result in intermittent faults. Again this can lose you a single bit at a time. This is related to the Xbox "red ring of death" issue.
• Failure of other components. Capacitors are the usual suspect; electrolytic, tantalum, ceramic all have different failure modes. Again this may result in a system that mostly works but is prone to misinterpreting marginal values or suffers from timing drift.
• Weird materials science ("Purple plague", tin whiskers due to lead-free solder)
• Part QA may not be up to the standards you expect (suppliers shipping inferior parts with a bogus "aerospace grade" label). Hard to detect even after it's happened.

It's important to recognize that in high-speed digital systems you don't get nice clean "one" and "zero", you get a series of rising and falling edges that are smeared out by the parasitic capacitance and inductance of the wiring. This is inherently vulnerable to being misinterpreted in marginal electrical conditions.

Answer 2

As other answer pointed out: A CPU can fail. Either partially (giving erroneous answers), or totally.

Moreover all computer are subject to cosmic radiations that can once in a while flip a bit in memory (in addition to other sources of error like short circuit, ...). That's why scientific experiments and long running servers use ECC memory. Spaceships also use specific hardened CPUs to limit this effect as they are less shielded from such interferences. Planes fly at high altitudes and are subject to more of this interferences than your earth-bound computer.

Even if the event of this happening is very uncommon (but not unheard of), you MUST ensure that the results are 100% accurate. One bit flipped could change the behavior of your plane in unpredictable ways, like inverting the controls, inverting the flight envelop protection law, ...

Answer 3

I know this question has already recieved a handful of answers, but none of them seem to address the issue of why there are three systems in the redundant set, rather than just two.

First off, as was pointed out by Simon, Jan Hudec and RedGrittyBrick, the designs aren't identical at all. Indeed, they are often completely different for very good reason: the probability that any given problem will affect all redundant systems, and especially affect all redundant systems in the same way, goes from "small" to "utterly miniscule bordering on nonexistent". Compare How dissimilar are redundant flight control computers?

Second, as to why there are three systems in each redundant setup. When everything is working fine and the aircraft is in steady flight, for some value and some given set of inputs, all systems report that a correction of 0 (of whatever unit) is needed. At this point there is no problem, and the computers just serve to maintain the present state. Now, one of the component systems fail to do its job properly for any reason, and start reporting that a correction of +50 units is required. That is, the set of responses changes from [0,0,0] to [0,0,+50]. Two systems agree and the third reports something else, so we can likely safely disregard the outlier and go with the two systems that report the same: treat the result as [0,0,incorrect] and ignore the incorrect result while logging technical details and displaying some sort of prominent warning that the systems need to be looked over ASAP. But what if we had only two systems to begin with, and one of the two fails in the same way? The determined correction needed goes from [0,0] to [0,+50]. Quick now: which value is correct? Should you maintain state, or correct by +50?

At that point, there is no way to know whether correcting by 0 or +50 is the proper course of action. You could take an average, but using an average of two numbers (one of which is likely to be incorrect) could actually be worse than either value by itself.

By adding a third system to the redundant set, you add a tie-breaker for the situation where there is one malfunctioning system. Only if two of the three systems start to malfunction at the same time do you have an actual problem, and if the aircraft is having such problems that two out of three redundant systems are giving erroneous outputs then you likely have some serious trouble to begin with.

Answer 4

Why are critical flight computers redundant?

Software

One point that has been missed is that the redundant systems are often independent designs, especially the software. This guards against design faults (or software bugs) that may otherwise cause problems under rarely occurring combinations of circumstances.

Hardware

Even if a microprocessor is highly reliable, there are a number of factors that may be relevant

• aircraft fly at high altitude where the atmosphere provides less protection against cosmic rays. This not only affects crew health but has the potential to interfere with electronic devices.
• Avionics systems are composed of more than just microprocessors, there are surely also other, more failure-prone, devices - such as capacitors. There are innumerable ways electronics can fail, e.g. vibration induced failure of grounding leading to interference on data lines (e.g. from analogue sensors).

---

I've never heard of microprocessors suddenly failing.

Reliability ≠ Safety

• Many accidents occur without any component “failure”
  
  • Caused by equipment operation outside parameters and time limits upon which reliability analyses are based.
  • Caused by interactions of components all operating according to specification.
  • Highly reliable components are not necessarily safe

^{From Nancy Leveson, MIT, via UCSD}

Answer 5

Most of the answers have revolved around the potential for computer hardware errors and things of that nature. While all that's true, no one has mentioned what the computers are actually looking at.

Let's say you're on approach, getting ready to do a CAT III autoland, and you only have two computer systems. Both computer systems compare the #1 and #2 radio altimeter systems. Only there is a malfunction with one of the radio altimeter systems causing a discrepancy of some arbitrary value that is not within limits.

How does the computer know which one is wrong? One computer looks at radio altimeter system #1 and sees 500 ft. The other looks at system #2 and see 1000 ft. Which one is right and which one is wrong? How could the computer possibly make that decision?

Enter the third computer. If the value of what it sees is commiserate with that of any of the other two computers, it can effectively "vote" the invalid reading "off the island."

I should note that most of these computers have anywhere between two and four processors all comparing their own results. That is the INTERNAL redundancy to avoid hardware failure, but having numerous cross comparisons of external systems is largely the reason a third system exists.

Note: As an A&P mechanic, 9 times out of 10 it's one of the external systems having failed (radio altimeter, MMR/ILS miscompare, etc...) that causes a degradation in capabilities - NOT the computer itself.

Answer 6

Computers fail, spontaneously, all the time. You're not used to that because you've not used many computers. But consider somebody like Google, who runs massive data centres containing thousands of computers. The software that runs Google is designed around the explicit assumption that computers do fail because it happens multiple times per day. Now, an aircraft doesn't contain very many computers but the ones it does contain are safety-critical. So they're duplicated to make sure their failure doesn't cause a problem.

Answer 7

If we look at this from a strictly engineering standpoint, microprocessors like anything have a cycle lifetime. Generally speaking it's very long, and the PC you are posting this from most likely will be out dated long before it hits is cycle life time. Although a microprocessor has no moving parts it does take input from various sensors. I can only assume that the inputs are fused in some way, but that does not mean that a spikes are completely eliminated and isolated if they do occur. For what it's worth, even relatively small surges will fry a microprocessor. Keeping that in mind, to err on the side of caution multiple systems are used. With the ever shrinking size of technology it has gotten easier and cheaper to carry a spare so from a sales standpoint the peace of mind is there. Again it's better to have it and not use it than to not have it when you need it.

To directly address your question, I have been around microprocessors, micro-controllers and the like for a long time. In that time I have had maybe two or three fail spontaneously, usually related to heat. In a plane this may not seem like an issue but actually extreme cold can cause issues as well as extreme heat when it comes to electronics. That being said I have toasted countless units by hitting them with overloaded inputs. Let's say your plane got hit by lighting (I know that modern airlines are protected against this) but for arguments sake let's say a ground was bad: this would easily toast a unit.

Side note: It is more common for memory chips/drives to fail these days. This is something you may never know as most modern computers can deal with dead memory be it in the disk or in system memory.

Answer 8

On specific redundancy, the installation environment of these systems is probably the biggest factor. Not only are many systems crammed closely together in tight spaces, but airflow is often very limited in there. Heat is a great destroyer of many a microprocessor. Airplanes also vibrate, a lot, due to spinning engines, in-flight turbulence, and simply landing. Poor solder joints and sub-par crimp jobs or a loose connector backshell, and you got a bad connection, or worse, an intermittent one.

On redundancy in general, if you experience a BSOD at work, comparatively it's no big deal. You might have lost the document you were working on, but that's about it. If aircraft systems go out, you have a real problem. It's difficult to achieve, but the redundancy is there because the lives of hundreds of people are relying upon it.

Answer 9

The chance of processor failure is very low indeed, but not zero. Upon failure of the processor, what happens during the transition time between failure, and full functionality after re-boot? With a rare event like this, could we ever build up enough experience to be sure that we've tested in all circumstances? We're talking about < $10^{-9}$ numbers here.

Backups are prone to hidden failures. The backup is normally not used and only switched on when needed - but has something rusted through, or has a funky mould nestled in a comfortable warm spot and cause a short upon activation? Murphy still haunts aerospace applications. The backup can be tested before take-off, but what if it shakes loose and the main processor fails? The chances are slim, but all major accidents nowadays are caused by unlikely strings of events like this.

Redundancy is useful because it continuously demonstrates that the main devices are functioning properly, and it is used for flight critical circumstances. Back-up systems may be used if you can do without the main device, or if it is guaranteed that the back-up will always work, like manual actuation of flight controls.

An autopilot in cruise is not flight critical, and may be disconnected without grave consequences. In a CAT III landing where the runway can only be observed once you're driving on it, they are absolutely vital. You don't want the autopilot to disconnect 10 metres above the runway, no visibility, gusty side wind - there is no time to engage the back-up.

Answer 10

If a microprocessor is overheated or overloaded and spontaneously fails, I would expect it to stop doing anything and produce no output.

Have you ever overclocked a CPU, or watched an old piece of hardware die? You can get all kinds of weird artifacts while the cpu is still running.

Answer 11

In an airplane, safety is more important than any other factor (after that comes optimum weight for fuel efficiency, and overall cost is third). If airplane would not be safe, not enough people would fly, and the airline industry would collapse. That's why there are FAA regulations, and that's why there are so many rules for the airlines. (the airport security check is another subject, related to national/political security with immigration, etc, so when we say 'safety' of the airplane, I mean engineering-wise)

Critical systems on board (i.e. systems that are required to fly the airplane) will need redundancy. Like the burner in the jet engine has 2 igniter, even though one is enough. Also, if an engine fails, the other engine can fly the airplane, and the computer will compensate for the left/right force imbalance. Many systems in the airplane rely on the computer so it needs to have a 'plan b' (redundancy is one of the 'plan b')

Answer 12

Because, despite the theory being good, the reality is that not all computer components are equal.

A specific example from the early 90s: Intel produced the 486/33 CPU (it was quite cutting edge and blazingly fast for the day). Most left the factory just fine, but some had an esoteric bug in the FPU that would generate incorrect answers. The magazines of the day were filled with calculations you could put into your spreadsheet that would produce X if your CPU was good, or Y if it had the bug.

If your plane happens to be running with one of the faulty CPUs in it, and just the right set of data happens to be collected and fed into any of the flight control programs and runs into this FPU bug, you'll be happy that the other two CPUs calculated the correct value and kicked the errant one out of the loop.