The short answer is that no safety-critical avionics systems that I'm aware of use Linux, and the highest criticality systems often don't use a commercial operating system at all. However, Linux is used in other safety-critical applicaitons like the Space X Falcon 9 and medical applications. A more detailed explanation is difficult to do without going into too much depth, since the question is kind of like asking "are modern airframes made with nanomaterials?", where a detailed explanation would have to cover the pros and cons of the material, places where its use makes the most and least sense, differences in manufacturers, and an overview of what's used instead, etc. I'll try to cover all those points succinctly with links to further information.
According to this FAA report from 2001, some of the major operating systems for certified avionics were VRTX, LynxOS, PSOS, VxWorks, and Enea's OSE, although non-critical avionics sometimes use other systems like Windows NT. (Yes, LynxOS is based on Unix and Linux is considered "Unix-like", but there are many differences between the two, just like how a Linux isn't similar to the BSD-based Mac OS X). However, the most critical systems don't use a commercial OS at all. They note "With the evidence available today, in general, it can be stated that COTS products typically do not meet the requirements for level A criticality software." In layman's terms, that means most avionics developed with third-party software, from VxWorks to Linux, weren't being tested and analyzed enough to be put in something critical like a landing system, TCAS unit, or critical display unit. That's likely changed in the 10+ years since the report was written, however. Here's a more recent article on real-time operating system usage in avionics.
What do RTOS and COTS mean?
An important concept here is a real-time operating system, or RTOS. A RTOS in a nutshell provides guarantees that software won't run out of scheduled computing time, messages between parts of the software are passed in a small amount of time, memory won't run out, and other important tasks are guaranteed instead of working well under normal conditions then breaking under unusual conditions. Normal operating systems can't make such guarantees. These guarantees, or acceptable substitutes, are required for certification of most avionics.
Another important concept is the difference between in-house and commerical off-the-shelf (COTS) systems. Commercial off-the-shelf systems are publicly available and are not customized much for each manufacturer. This article provides a good summary of pro's and cons of commercial and in-house operating systems. Many high-criticality avionics still have the core software developed in-house due to the advantages involved.
Does Linux meet certification requirements?
Yes, Linux is not "FAA certified" but actually no RTOS is "FAA certified" or "meets DO-178C". DO-178C provides objectives that the avionics system engineers have to fulfill in order to certify their whole avionics software (the objectives cover audits, testing, safety analysis, and requirement writing, among other things). So the best the OS supplier can do is provide "DO-178C ready" software or follow DO-178C guidelines. Note that DO-178C recognizes various safety levels, so what's certifiable under DO-178C for a maintenance system may not be certifiable under DO-178C for a critical display unit. The problem with Linux isn't that Linux isn't "certified under DO-178C", it's that it's difficult to certify due to challenges noted below. It's possible to certify a system using Linux, but doing so could be prohibitively difficult due to the extra analysis, safety measures, and documentation that would be needed, especially for the most critical avionics. Some of these challenges and alternate methods of compliance are outlined in this FAA report.
Advantages and disadvantages of Linux
There are advantages to using Linux, which are partly shared with other commercial systems. More employees will already have expertise with Linux, and the OS was designed with more expertise and time than most organizations have available. Commercial systems also have a much lower sticker price than in-house software. Commercial systems are also more adaptable and allow changes to hardware and room for future growth without going back to the drawing board. Tooling to work with the software is more developed, and they may also have a longer service history that provides confidence in the software.
Here are some of the issues Linux faces in an avionics system when competing with a RTOS, from this FAA report:
partitioning: If two programs are supposed to be running independently, there needs to be evidence that one program can't interfere with another through shared memory structure, caches, board support and firmware, errors, interrupts, etc.
worst case execution time: In desktop computing it's ok if I bog down my PC with too many requests or unusual conditions at once and the system slows down or skips some program steps. In avionics this is not acceptable for obvious reasons. The supplier would need to verify this either through models of the CPU, memory, etc. or rigorous lab tests and timing analysis. Both are more difficult the more complex your hardware and software is.
- MCDC coverage: Tests on the most critical avionics (Level A) requires running enough lines of code and decision conditions to reach the MCDC code coverage standard, which falls between "run every line of code" and exhaustive tests of every combination of conditions. Some operating systems like Linux can be extremely difficult to test thoroughly enough to meet this standard.
- documentation: Rigorous analysis and stress test requires a lot of documentation on inner workings of the OS
- security considerations: Linux by design has more security problems than a lean, in-house application. These security issues are becoming
increasingly important, especially in the military
- dead and unreachable code: Carefully disabling the many unused functions of Linux and guaranteeing
they can't interfere in your software is usually required for
certification
Other safety-critical industries
What about similar safety-critical industries? The Space X Falcon does use Linux in some of its flight computers (sources). Based on this interview, Space X seems to prioritize future growth, availability, short cycle times, and expertise over the simplicity and robustness of in-house development in this area. Note that the flight control computers on the Space X Falcon aren't directly analogous to an LRU in typical modern avionics, so extra work or an unusual architecture would likely be required to get Linux to work for typical avionics applications.
Linux is also used in many safety-critical medical applications, but not without problems similar to those faced in aviation. I'd recommend reading Wind River's "Choosing Linux for Medical Devices", or this article on the disadvantages of Linux in safety-critical medical applications.
I'm assuming you're talking about the principal avionics like flight guidance, autopilot, fly-by-wire, or displays. Other airplane systems could loosely be considered to be safety-critical but aren't typical examples of safety-critical devices, like certified electronic flight bags, connectivity solutions, and maintenance software. These are sometimes better suited for a Linux application due to high complexity and less rigorous safety considerations.
Note that I'm not an expert in this field and my advice does not replace the advice of a certification specialist.