It really comes down to cost/benefit.
Let's consider two aircraft that were built out of stainless steel, and performed very well: the XB-70 and X15. Stainless steel was used to resist the heat of Mach 3+ flight, which would weaken aluminum to a failure point. At the time the XB70 was originally designed, titanium wasn't available in sufficient quantity to consider a large (100+ airframe) production run.
The X15 did achieve a speed of Mach 6.7, where skin temperature would be quite high. However, it was a pure research rocket powered aircraft, so it doesn't make as appropriate a contrast, and that speed was maintained only briefly.
By the early 1960's, titanium was available in enough quantity (albeit by using shell corporations to purchase it from the Soviets) to build the smaller and less produced A12/SR71. Like the XB-70, it was designed to sustain Mach 3+ for over an hour, and the subsequent high skin temperatures that preclude the use of aluminum.
The key issue to understand here is sustained Mach 3 flight: for over an hour. Aluminum interceptors such as the MIG25 have a brief Mach 3 dash capability, five or ten minutes, where the skin temperature won't rise as much. After ten minutes at Mach 3, the MIG25 is very low on fuel and it's engines are ruined.
To keep weight low, the XB70 used a complex honeycomb design that is expensive to manufacture, and the resulting aircraft still was heavier than an aluminum aircraft of similar size. The XB70 made up for this by using compression lift - literally riding it's own supersonic shock wave - for very efficient Mach 3 flight. The A12/SR71 used the shock wave in it's engines to achieve a similar level of efficiency.
Note that the B-58 also used a honeycomb construction to keep weight low, but was made from aluminum, as it's design speed was Mach 2. The Concorde was made from aluminum, but it cruised at Mach 2.2. Beyond that speed, the heat generated from skin friction rises dramatically, to a level where aluminum would be critically (and fatally) weakened.
Presumably, a commercial aircraft made from stainless steel would also require the complex honeycomb design to achieve a reasonably efficient weight. It would still not be as efficient as an aluminum aircraft. As the primary reason not to use aluminum was to resist sustained Mach 3 heat, a speed that only the SR71 and XB70 can maintain for any period of time, the extra expensive of honeycomb steel (or titanium) and the greater weight isn't really justified for a subsonic airliner.
The XB-70 was canceled as it was thought to be vulnerable to the SAM2 missile the Soviets had developed. Ironically, the SR71 proved this not to be the case: being shot at over 1000 times (including many SAM2's) and never hit.
Yes, you can build an aircraft out of stainless steel. But, the honeycomb construction is expensive, and the result less efficient than an aluminum aircraft. The only reason steel was used in those two aircraft was to sustain very high skin temperatures. You could build one out of titanium, and it would be even more expensive.
While there have been airframe failures from aluminum fatigue, countermeasures far less expensive and lighter than moving to stainless steel have been developed.