19
$\begingroup$

At least in airliners, the truly critical computers are redundant. Typically three identical copies of the autopilot computers run in parallel and compare results; if one computer disagrees with the other two, its output is ignored. The system allows some processors to be faulty while maintaining the operation of the overall system.

But why? I've never heard of microprocessors suddenly failing. Sure, there could be manufacturing errors, but those would have been caught at the factory. Perhaps the program (and its proof) is wrong, but it would be wrong in the same way across the processors. Similarly, bad input would cause bad output across all three computers. What kind of errors does this redundancy protect against? Do microprocessors sometimes just do math wrong?

If a microprocessor is overheated or overloaded and spontaneously fails, I would expect it to stop doing anything and produce no output. To deal with this kind of failure, you'd want to have a backup processor, but you wouldn't need to compare the outputs of three computers—any output produced would be assumed correct, so you'd be happy to directly use the output of any processor that was producing output.

Related: The answer to What is the purpose of multiple autopilots? simply says "redundancy" before going into how this is achieved.

$\endgroup$
  • 7
    $\begingroup$ I will wait for authoritative answers, but on the systems I have been involved with, the 3 computers ran different software, produced by independent teams and proven to generate the same outputs for the same inputs. $\endgroup$ – Simon Mar 24 '15 at 22:00
  • 6
    $\begingroup$ It might be, I've been out of the loop for about 20 years. BTW, I'm a software engineer now and have seen processors fail and, more commonly, RAM chips fail. $\endgroup$ – Simon Mar 24 '15 at 22:05
  • 14
    $\begingroup$ @raptortech97: Autopilot is not that critical; the plane can be flown by hand. The really critical systems are fly-by-wire controls. In Airbus they run on pairs of dissimilar boards (i386 and m68k) with independently written software that cross-check each other, these pairs are multiplied for fail-over and there is independent set for primary flight controls (elevator and aileron) and another for alternate (spoilers and horizontal stabilizer), so if either fails the other can still control both pitch and roll. I believe Boeing system on 777 and 787 is similar. $\endgroup$ – Jan Hudec Mar 25 '15 at 6:14
  • 3
    $\begingroup$ I find the choice of using 3 to be a bit odd. In order to deal with one of them failing in an arbitrary way you need byzantine resilience, which cannot be achieved with less than 4. $\endgroup$ – kasperd Mar 25 '15 at 12:35
  • 4
    $\begingroup$ "I've never heard of microprocessors suddenly failing". That's because you are not familiar with electronics. Ask here and you'll be enlightened. In addition a flight computer is not made only of a CPU/MCU. Keyboard, display, connectors, memory, clock, other chips, other electronic components, power unit... name it. $\endgroup$ – mins Mar 27 '15 at 2:03

12 Answers 12

29
$\begingroup$

Failure modes to consider:

  • Overheating. This changes the chip's timing properties and eventually results in error. This can manifest as single-bit errors in the middle of seemingly normal operation; it will eventually crash, but may output bad data first.
  • Water damage. Manifests as a parasitic resistance on the board and may cause you to misinterpret bits as high or low. May be leaky housings, condensation, etc.
  • Electromagnetic interference. (The system is supposed to be resistant to this, but it's still worth thinking about).
  • Physical connection problems. Either during construction (soldering faults) or induced afterwards (heat, vibration). Microscopic cracks in boards or joints can pass QA but result in intermittent faults. Again this can lose you a single bit at a time. This is related to the Xbox "red ring of death" issue.
  • Failure of other components. Capacitors are the usual suspect; electrolytic, tantalum, ceramic all have different failure modes. Again this may result in a system that mostly works but is prone to misinterpreting marginal values or suffers from timing drift.
  • Weird materials science ("Purple plague", tin whiskers due to lead-free solder)
  • Part QA may not be up to the standards you expect (suppliers shipping inferior parts with a bogus "aerospace grade" label). Hard to detect even after it's happened.

It's important to recognize that in high-speed digital systems you don't get nice clean "one" and "zero", you get a series of rising and falling edges that are smeared out by the parasitic capacitance and inductance of the wiring. This is inherently vulnerable to being misinterpreted in marginal electrical conditions.

$\endgroup$
  • $\begingroup$ Wow, thanks for the great detail! I do want to note that I find it hard to believe that the parts would not have been up to standard. In aviation, the original design and every spare part had to get FAA-certified, and there's a large paper trail tracking everything. The bogus spare parts industry is rather small nowadays. $\endgroup$ – raptortech97 Mar 25 '15 at 12:00
  • 1
    $\begingroup$ I'm coming from the electronics rather than the aviation side here, so I'm not familiar with how well the paper trail works, but you might find this interesting if you like detail: bunniestudios.com/blog/?page_id=1022 $\endgroup$ – pjc50 Mar 25 '15 at 12:08
  • 2
    $\begingroup$ (Anecdotally I think the majority of "worked in QA fails early in production" faults are mechanical failures of solder joints; I believe aviation/MILSPEC used to prefer wire-wrap for exactly this purpose, but that's no longer practical with tiny packages) $\endgroup$ – pjc50 Mar 25 '15 at 12:10
  • 2
    $\begingroup$ This is the best answer. The simple fact is that CPUs can produce wrong answers and often do when operated outside of their specified voltage/temperature/etc. ranges because it messes with the timings. A timing difference of picoseconds can cause you to read the wrong value off of a line. Similarly, a very small voltage difference can cause a value to be read as 1 instead of 0 or vice versa. This is why when people are testing overclocks, they run burn-in tests that do math as quickly as possible for hours on end and watch for wrong answers. $\endgroup$ – reirab Mar 25 '15 at 18:30
  • 3
    $\begingroup$ As far as saying it will eventually crash... it might eventually crash, or it might just keep on producing bad output. When CPUs get too hot, for instance, they often continue happily running their code all day long while producing wrong answers, especially if one or more ALUs or FPUs are operating out of spec, but the instruction decoder is not (which is not a terribly uncommon failure mode.) $\endgroup$ – reirab Mar 25 '15 at 18:32
27
$\begingroup$

As other answer pointed out: A CPU can fail. Either partially (giving erroneous answers), or totally.

Moreover all computer are subject to cosmic radiations that can once in a while flip a bit in memory (in addition to other sources of error like short circuit, ...). That's why scientific experiments and long running servers use ECC memory. Spaceships also use specific hardened CPUs to limit this effect as they are less shielded from such interferences. Planes fly at high altitudes and are subject to more of this interferences than your earth-bound computer.

Even if the event of this happening is very uncommon (but not unheard of), you MUST ensure that the results are 100% accurate. One bit flipped could change the behavior of your plane in unpredictable ways, like inverting the controls, inverting the flight envelop protection law, ...

$\endgroup$
  • 8
    $\begingroup$ Not only can cosmic radiation flip a bit in memory, it can also flip a bit in the CPU. Now ECC for memory still makes sense as memory has far more bits, but the risk still exists for CPU's. $\endgroup$ – MSalters Mar 25 '15 at 11:41
  • $\begingroup$ Yep, that was there is ECC memory AND Hardened CPUs :) $\endgroup$ – Antzi Mar 25 '15 at 11:48
  • 1
    $\begingroup$ I think this is the most important reason. If bad memory were the only concern, we could just triplicate the RAM but have a single CPU. But if a cosmic ray flips the wrong bit in the CPU and the CPU isn't triplicated, the result could be catastrophic. $\endgroup$ – raptortech97 Mar 25 '15 at 11:50
  • 2
    $\begingroup$ As comparison, there is more background radiation in a typical passenjer jet at cruise altitude than there is today at ground zero in hiroshima. $\endgroup$ – J... Mar 26 '15 at 9:32
  • 1
    $\begingroup$ This is called a Single Event Upset. There will often be 3 copies of the same data stored in memory as well as another unit to take over in the event of a critical fault. $\endgroup$ – scotty3785 Jun 5 '17 at 13:53
11
$\begingroup$

Why are critical flight computers redundant?

Software

One point that has been missed is that the redundant systems are often independent designs, especially the software. This guards against design faults (or software bugs) that may otherwise cause problems under rarely occurring combinations of circumstances.

Hardware

Even if a microprocessor is highly reliable, there are a number of factors that may be relevant

  • aircraft fly at high altitude where the atmosphere provides less protection against cosmic rays. This not only affects crew health but has the potential to interfere with electronic devices.
  • Avionics systems are composed of more than just microprocessors, there are surely also other, more failure-prone, devices - such as capacitors. There are innumerable ways electronics can fail, e.g. vibration induced failure of grounding leading to interference on data lines (e.g. from analogue sensors).

I've never heard of microprocessors suddenly failing.

Reliability ≠ Safety

  • Many accidents occur without any component “failure”
    • Caused by equipment operation outside parameters and time limits upon which reliability analyses are based.
    • Caused by interactions of components all operating according to specification.
    • Highly reliable components are not necessarily safe

From Nancy Leveson, MIT, via UCSD

$\endgroup$
  • 2
    $\begingroup$ "I've never heard of microprocessors suddenly failing." I had one fail in a desktop once. I'm typing away at whatever, and the screen just went black. After the usual testing we sent it back, they said we needed a new CPU. $\endgroup$ – paul Mar 25 '15 at 13:34
  • 6
    $\begingroup$ People who've never heard of CPUs failing have never gone through the AMD Athlon/Pentium4 days where frying a CPU wasn't uncommon $\endgroup$ – slebetman Mar 25 '15 at 13:49
  • $\begingroup$ Thanks for mentioning that indeed the software is not identical, but different teams write for different hardware, but to the same specs. The original question is misleading, to say the least. $\endgroup$ – Peter Kämpf Mar 25 '15 at 14:11
11
$\begingroup$

I know this question has already recieved a handful of answers, but none of them seem to address the issue of why there are three systems in the redundant set, rather than just two.

First off, as was pointed out by Simon, Jan Hudec and RedGrittyBrick, the designs aren't identical at all. Indeed, they are often completely different for very good reason: the probability that any given problem will affect all redundant systems, and especially affect all redundant systems in the same way, goes from "small" to "utterly miniscule bordering on nonexistent". Compare How dissimilar are redundant flight control computers?

Second, as to why there are three systems in each redundant setup. When everything is working fine and the aircraft is in steady flight, for some value and some given set of inputs, all systems report that a correction of 0 (of whatever unit) is needed. At this point there is no problem, and the computers just serve to maintain the present state. Now, one of the component systems fail to do its job properly for any reason, and start reporting that a correction of +50 units is required. That is, the set of responses changes from [0,0,0] to [0,0,+50]. Two systems agree and the third reports something else, so we can likely safely disregard the outlier and go with the two systems that report the same: treat the result as [0,0,incorrect] and ignore the incorrect result while logging technical details and displaying some sort of prominent warning that the systems need to be looked over ASAP. But what if we had only two systems to begin with, and one of the two fails in the same way? The determined correction needed goes from [0,0] to [0,+50]. Quick now: which value is correct? Should you maintain state, or correct by +50?

At that point, there is no way to know whether correcting by 0 or +50 is the proper course of action. You could take an average, but using an average of two numbers (one of which is likely to be incorrect) could actually be worse than either value by itself.

By adding a third system to the redundant set, you add a tie-breaker for the situation where there is one malfunctioning system. Only if two of the three systems start to malfunction at the same time do you have an actual problem, and if the aircraft is having such problems that two out of three redundant systems are giving erroneous outputs then you likely have some serious trouble to begin with.

$\endgroup$
5
$\begingroup$

Most of the answers have revolved around the potential for computer hardware errors and things of that nature. While all that's true, no one has mentioned what the computers are actually looking at.

Let's say you're on approach, getting ready to do a CAT III autoland, and you only have two computer systems. Both computer systems compare the #1 and #2 radio altimeter systems. Only there is a malfunction with one of the radio altimeter systems causing a discrepancy of some arbitrary value that is not within limits.

How does the computer know which one is wrong? One computer looks at radio altimeter system #1 and sees 500 ft. The other looks at system #2 and see 1000 ft. Which one is right and which one is wrong? How could the computer possibly make that decision?

Enter the third computer. If the value of what it sees is commiserate with that of any of the other two computers, it can effectively "vote" the invalid reading "off the island."

I should note that most of these computers have anywhere between two and four processors all comparing their own results. That is the INTERNAL redundancy to avoid hardware failure, but having numerous cross comparisons of external systems is largely the reason a third system exists.

Note: As an A&P mechanic, 9 times out of 10 it's one of the external systems having failed (radio altimeter, MMR/ILS miscompare, etc...) that causes a degradation in capabilities - NOT the computer itself.

$\endgroup$
4
$\begingroup$

Computers fail, spontaneously, all the time. You're not used to that because you've not used many computers. But consider somebody like Google, who runs massive data centres containing thousands of computers. The software that runs Google is designed around the explicit assumption that computers do fail because it happens multiple times per day. Now, an aircraft doesn't contain very many computers but the ones it does contain are safety-critical. So they're duplicated to make sure their failure doesn't cause a problem.

$\endgroup$
  • $\begingroup$ I think the OP was especially asking about the fact that there are three which compare instead of just two in the case the hardware dies. $\endgroup$ – David Mulder Mar 25 '15 at 11:02
  • $\begingroup$ From everything I've heard about Google's systems, they're designed to handle the total failure of any computer, not computers that are misbehaving $\endgroup$ – raptortech97 Mar 25 '15 at 11:44
  • $\begingroup$ Hmm. I think dealing with malfunction rather than outright failure is implicit in my answer but you're both right that it's not very well tailored to the question. I'll have a think and either improve or delete it. $\endgroup$ – David Richerby Mar 25 '15 at 11:46
  • 1
    $\begingroup$ Google's systems are built around fail-and-retry as that's built into all the Internet protocols. Other large systems are similar and even embrace "crash-only" operation (see Netflix "chaos monkey"). This isn't appropriate for safety-critical real-time systems. It's an interesting contrast between a cheap system which in practice works nearly all the time versus a much harder to develop and more expensive system which can offer design guarantees. $\endgroup$ – pjc50 Mar 25 '15 at 15:04
2
$\begingroup$

If we look at this from a strictly engineering standpoint, microprocessors like anything have a cycle lifetime. Generally speaking it's very long, and the PC you are posting this from most likely will be out dated long before it hits is cycle life time. Although a microprocessor has no moving parts it does take input from various sensors. I can only assume that the inputs are fused in some way, but that does not mean that a spikes are completely eliminated and isolated if they do occur. For what it's worth, even relatively small surges will fry a microprocessor. Keeping that in mind, to err on the side of caution multiple systems are used. With the ever shrinking size of technology it has gotten easier and cheaper to carry a spare so from a sales standpoint the peace of mind is there. Again it's better to have it and not use it than to not have it when you need it.

To directly address your question, I have been around microprocessors, micro-controllers and the like for a long time. In that time I have had maybe two or three fail spontaneously, usually related to heat. In a plane this may not seem like an issue but actually extreme cold can cause issues as well as extreme heat when it comes to electronics. That being said I have toasted countless units by hitting them with overloaded inputs. Let's say your plane got hit by lighting (I know that modern airlines are protected against this) but for arguments sake let's say a ground was bad: this would easily toast a unit.

Side note: It is more common for memory chips/drives to fail these days. This is something you may never know as most modern computers can deal with dead memory be it in the disk or in system memory.

$\endgroup$
  • 1
    $\begingroup$ The avionics will generally be in the avionics bay below the cockpit, or in the cockpit itself, so the extreme cold won't be an issue, but with inadequate cooling they could overheat. What I didn't clarify was that if a microprocessor spontaneously fails, it should be possible to detect that and switch to a backup. The system in use compares the three outputs, implying that a microcontroller could produce an incorrect output, as opposed to simply failing to produce any output. $\endgroup$ – raptortech97 Mar 25 '15 at 0:41
  • 1
    $\begingroup$ I find the statement that "most modern computers can deal with dead memory" dubious at best. Maybe in the strict sense that "if a memory cell goes bad, it won't immediately cause the computer to crash", but it will cause erratic operation as soon as that memory is actually used for something. The saving grace in a way is that in modern PCs, the lion's share of RAM is not actively used; it gets used for cache. Which can be equally bad if you have no way of detecting a problem (which in practice means you're using non-ECC RAM, which most desktop systems don't). $\endgroup$ – a CVn Mar 25 '15 at 9:17
  • $\begingroup$ That is incorrect, PC's are capable of skipping over and ignoring bad sectors on disk see here en.wikipedia.org/wiki/Bad_sector likewise the linux kernel now supports badram and badmem which are capable of telling the system to skip over corrupt addresses, which is what I was referencing. You are however correct that bad memory can cause erratic behavior, however it is not always the case. $\endgroup$ – Dave Mar 25 '15 at 13:11
  • $\begingroup$ @user16230: And who tells the kernel to exclude that memory? certainly not the program that is currently running off from that memory. This is a purely offline last resort measure that is done after detecting that a memory bit is defective by other means than a program currently running on it. Even for bad sectors the result is data loss. Nothing that can be fixed mid flight by reflashing the system. Also bits might misbehave under the weirdest conditions only, that can not be detected by even the best memory test programs. Or why do you think rowhammer was only discovered recently? $\endgroup$ – PlasmaHH Mar 25 '15 at 13:19
  • $\begingroup$ Again I agree however I was merely pointing out that it is possible to deal with bad memory (not that I advise doing it in flight). $\endgroup$ – Dave Mar 25 '15 at 13:22
2
$\begingroup$

On specific redundancy, the installation environment of these systems is probably the biggest factor. Not only are many systems crammed closely together in tight spaces, but airflow is often very limited in there. Heat is a great destroyer of many a microprocessor. Airplanes also vibrate, a lot, due to spinning engines, in-flight turbulence, and simply landing. Poor solder joints and sub-par crimp jobs or a loose connector backshell, and you got a bad connection, or worse, an intermittent one.

On redundancy in general, if you experience a BSOD at work, comparatively it's no big deal. You might have lost the document you were working on, but that's about it. If aircraft systems go out, you have a real problem. It's difficult to achieve, but the redundancy is there because the lives of hundreds of people are relying upon it.

$\endgroup$
2
$\begingroup$

The chance of processor failure is very low indeed, but not zero. Upon failure of the processor, what happens during the transition time between failure, and full functionality after re-boot? With a rare event like this, could we ever build up enough experience to be sure that we've tested in all circumstances? We're talking about < $10^{-9}$ numbers here.

Backups are prone to hidden failures. The backup is normally not used and only switched on when needed - but has something rusted through, or has a funky mould nestled in a comfortable warm spot and cause a short upon activation? Murphy still haunts aerospace applications. The backup can be tested before take-off, but what if it shakes loose and the main processor fails? The chances are slim, but all major accidents nowadays are caused by unlikely strings of events like this.

Redundancy is useful because it continuously demonstrates that the main devices are functioning properly, and it is used for flight critical circumstances. Back-up systems may be used if you can do without the main device, or if it is guaranteed that the back-up will always work, like manual actuation of flight controls.

An autopilot in cruise is not flight critical, and may be disconnected without grave consequences. In a CAT III landing where the runway can only be observed once you're driving on it, they are absolutely vital. You don't want the autopilot to disconnect 10 metres above the runway, no visibility, gusty side wind - there is no time to engage the back-up.

$\endgroup$
  • $\begingroup$ I appreciate the answer. Note that I use they/them pronouns, not he/him. $\endgroup$ – raptortech97 Jun 6 '17 at 13:28
  • 1
    $\begingroup$ I've noticed and amended. $\endgroup$ – Koyovis Jun 6 '17 at 14:00
  • $\begingroup$ "An autopilot in cruise is not flight critical, and may be disconnected without grave consequences", indeed true in theory, but has been once lethal when coupled with airspeed indication failure. $\endgroup$ – mins Oct 4 '17 at 19:31
  • 1
    $\begingroup$ @mins In reality each event like autopilot disengagement during cruise is given a safety level, and has to meet that safety level. The level of redundancy and rigor is adjusted to meet the safety level required. Autopilot disengagement during cruise is serious, yes, but not "catastrophic" and so is marked during safety analysis as only a "minor" or "major" problem. The same safety analysis may mark autopilot disconnect and airspeed indication failure as a joint issue with higher consequences if there's a significant interaction between the two. $\endgroup$ – Cody P Oct 5 '17 at 20:00
1
$\begingroup$

If a microprocessor is overheated or overloaded and spontaneously fails, I would expect it to stop doing anything and produce no output.

Have you ever overclocked a CPU, or watched an old piece of hardware die? You can get all kinds of weird artifacts while the cpu is still running.

$\endgroup$
  • $\begingroup$ CPUs used in safety critical tasks are paired with a watch dog, a (simple) system similar to railway industry dead man switch. This stops/resets the CPU as soon as it is not able to do a predefined handshake (e.g. to discharge a capacity before it is fully charged) $\endgroup$ – mins Sep 4 at 18:51
1
$\begingroup$

In an airplane, safety is more important than any other factor (after that comes optimum weight for fuel efficiency, and overall cost is third). If airplane would not be safe, not enough people would fly, and the airline industry would collapse. That's why there are FAA regulations, and that's why there are so many rules for the airlines. (the airport security check is another subject, related to national/political security with immigration, etc, so when we say 'safety' of the airplane, I mean engineering-wise)

Critical systems on board (i.e. systems that are required to fly the airplane) will need redundancy. Like the burner in the jet engine has 2 igniter, even though one is enough. Also, if an engine fails, the other engine can fly the airplane, and the computer will compensate for the left/right force imbalance. Many systems in the airplane rely on the computer so it needs to have a 'plan b' (redundancy is one of the 'plan b')

$\endgroup$
  • $\begingroup$ This doesn't answer the full question. I know that most things in aviation are redundant, but all of that redundancy has a specific reason - some mode of failure that could make the redundancy useful. I was asking what mode of failure chip redundancy protects against. $\endgroup$ – raptortech97 Apr 8 '15 at 21:06
-1
$\begingroup$

Because, despite the theory being good, the reality is that not all computer components are equal.

A specific example from the early 90s: Intel produced the 486/33 CPU (it was quite cutting edge and blazingly fast for the day). Most left the factory just fine, but some had an esoteric bug in the FPU that would generate incorrect answers. The magazines of the day were filled with calculations you could put into your spreadsheet that would produce X if your CPU was good, or Y if it had the bug.

If your plane happens to be running with one of the faulty CPUs in it, and just the right set of data happens to be collected and fed into any of the flight control programs and runs into this FPU bug, you'll be happy that the other two CPUs calculated the correct value and kicked the errant one out of the loop.

$\endgroup$
  • 4
    $\begingroup$ I think your example is misleading. First, I think you're actually talking about the Pentium FDIV bug. That was a design error, not a random manufacturing error. Every processor that was shipped had the bug, until the design was corrected. In that case, redundancy woudln't have helped: you'd just get multiple copies of the same wrong answer. $\endgroup$ – David Richerby Mar 25 '15 at 8:43
  • 2
    $\begingroup$ It's either the Pentium FDIV bug as mentioned by @DavidRicherby, or something like the 80386 double-sigma issue. I don't think the 486 ever had the sort of problems described in this answer. $\endgroup$ – a CVn Mar 25 '15 at 9:20
  • 1
    $\begingroup$ @MichaelKjörling It sounds a lot like a conflation of the 386 double-sigma (affecting a seemingly random sample of manufactured units) and the Pentium FDIV bug (floating point and lots of public attention). $\endgroup$ – David Richerby Mar 25 '15 at 9:46
  • $\begingroup$ Regardless, it'd be be cheaper to thoroughly test every processor than to triplicate them. $\endgroup$ – raptortech97 Mar 25 '15 at 11:47
  • 3
    $\begingroup$ @raptortech97 Tell that to NASA. They don't know about it. $\endgroup$ – a CVn Mar 25 '15 at 12:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.