Mainly, the optimum cruise altitude is where thrust and lift requirements for both take-off and cruise balance well. An additional benefit is the colder air which increases the efficiency of heat engines. Since this helpful drop in temperature ceases once the aircraft climbs above the tropopause, the benefits of flying higher increase most below the tropopause.
With increasing flight altitude, the airliner needs:
- Bigger engines to create the needed thrust in thinner air
- Bigger wings to create the needed lift
With the wings, the size of the tailplanes will also grow; this effect alone likely will weigh more than the beefing up of the fuselage structure for the increased cabin pressure. Flying higher will make almost all parts bigger and heavier.
Note that Mach 0.85 is a hard limit for efficient flight; airliners cannot compensate for lower density by flying faster. The only way to allow higher flight levels is to attach bigger wings and tails.
Another consideration is Breguet's formula: Jet aircraft have their optimum cruise lift coefficient at a value of $c_L = \sqrt{0.6\cdot c_{D0}\cdot\pi\cdot AR\cdot\epsilon}$, if we assume the thrust of high-bypass-ratio engines to vary with speed proportional to $v^{-0.5}$, which is a reasonable assumption. This means the airliner cannot fly higher by flying at a higher lift coefficient: This would decrease efficiency.
(Nomenclature: $c_{D0}$ = zero-lift drag, $AR$ = wing aspect ratio, $\epsilon$ = span efficiency)
With the wing size and the engines needed for flight at Mach 0.82 in the tropopause (Mach 0.85 is really not as efficient; follow the link to find out why this is the quoted cruise speed for long-range airliners), the take-off distance is quite reasonable and approximately matches the airports which had been defined by NATO during the cold war. Flying any higher into the stratosphere would increase the aircraft's mass due to bigger engines and wings, but would not incur the efficiency gains of increasing cruise altitude in the troposphere, where temperature drops with altitude.
Conversely, picking a lower design cruise altitude would allow to make both wings and engines smaller, but this would translate into:
- Higher take-off and landing speeds, and critical speeds during take-off due to the smaller wing,
- Lower take-off acceleration due to smaller engines,
- For twins: Not enough thrust during take-off when one engine fails,
- Lower climb speeds, so it would take longer to reach cruise altitude, and
- Not fully taking advantage of the cold air up in the tropopause.
Designing for a lower cruise altitude would translate into much longer runways and less efficient flight overall.
Designing for cruise in the tropopause is simply the sweet spot for airliner designers where all conditions match well and produce a balanced outcome.