What causes spiral mode instability in a GA plane and how do you minimize it in the first place, instead of trying to dampen it with dihedral (which is reducing efficiency by reducing vertical lift)?
1 Answer
One of the primary causes of spiral instability is the fact that once a turn begins, the outboard wingtip is moving along a larger circumference and therefore moving faster, and tending to produce more lift, than the inboard wingtip. So the bank angle tends to increase. Keeping the wingspan small will minimize this effect. Obviously, this is a trade-off and other desirable qualities are sacrificed.
Re dihedral-- the inefficiency due to more wing area being needed to get the same amount of vertical lift component is rather minimal. The more significant negative effects of dihedral include a excessive slip-roll coupling that creates excessive roll torque during an intentional slip (eg crosswind landing), bad effects in gusty crosswinds, poor roll response to aileron-only roll inputs, and a tendency toward "Dutch roll" yaw-roll oscillations.
Keeping the vertical fin small will help allow enough sideslip to allow whatever dihedral is present work as effectively as possible to help minimize spiral instability, but obviously this is a trade-off as well.
For more on the relationship between sideslip and dihedral see the related answers in Is this paragraph about the dihedral effect in the FAA's Glider Flying Handbook correct? or How does the dihedral angle work? .