The Wankel rotary engine seems to have several advantages over traditional reciprocating cylinder-and-piston designs. In general, they have a much higher power-to-weight ratio, less vibration, and drastically reduced rate of catastrophic failure, among many other benefits. I can only find disadvantages related to emissions and fuel efficiency. So my question is, why hasn’t it seen more use in aviation even though it seems very well suited to this application?
They do not have drastically lower rates of failure than comparable piston engines. Their manifest unreliability in automotive applications is one reason they failed so badly in the marketplace. My college materials science TA owned one and had it rebuilt four times. His record for the shortest distance before failure on a zero-timed Mazda rotary engine was 150 miles.
The breakdown mode was failure of the apex seals at the tips of the rotor due to poor lubrication, leading to excessive wear and scoring and the complete destruction of the seals. According to my TA, Mazda went through more than six fundamental redesigns of the apex seal system but never could completely solve the basic problem.
As another answer points out, the main problem is poor reliability.
In a conventional piston engine, the sliding contact of the piston against the cylinder wall is mostly against relatively cool metal and is constantly lubricated by oil from underneath. The contact is often maintained tightly by piston rings, which have a natural outward spring.
In the Wankel type the sliding contact of the rotor tip seal against the chamber outer wall is frequently against much hotter metal from recent combustion and is difficult to lubricate. A separate spring mechanism is required to hold the seal closed against the firing pressure, so the whole thing is more complicated. The result is that the tip seals are extremely demanding and tend to wear out or crack.
I seem to recall that Norton reckoned to have solved the problems but their bikes and UAVs failed to gain significant business. I don't know if there were other issues affecting that.
I think most answers on this question are fueled by lack of knowledge on rotaries, these engines do have inherent flaws relating to apex seals but 90% of their "unreliability" claims come from pure ignorant owners (at least in the automotive world) and a complete lack of understanding on how to maintain one. Most rx8's can do 100,000 kms before needing a rebuild, (average running time for that many kilometers is 2000 hours) so this is relatively in line with most overhaul windows (thanks google). And least us not forget the rotary engine won Le-mans in 1991 and apex sealing technology has come a long way since then. However to actually answer the question the real reason is due to the effort in getting the engine certified and the costs involved.
Wankel RCE is still in use today for Ultralight, Parapente, and other uses. Aixro, Wankel Supertec, Wankel GMBH, AIE.uk, are offering Wankel Rotary Engines for sale, research exists in India. John Deere, also Curtiss-Wright, developed a line of Wankel prototypes for Aviation, in several power ranges. The John Deere engine had an SFC around 240 gr/HP/hr, good enough, if you consider that it can work with low grade gasolines, heavy fuel versions of Wankel existed also.
Light weight, compactness, faults being not that total and catastrophic as in reciprocating engines, are Wankel advantages, a 2-Stroke engine, with sudden seizing, plus torsional vibration problems in shaft from engine to propeller killed the BD-5 and some pilots.