The answer is actually Yes, Boeing explored both 5- and 6-engine variants during the design phase of the 747-100 series. So, your first rationale was the correct one.
I read about it in an older book on the 747, written when the plane was only a few years in service. I can't remember the author or title at the moment, I'm afraid, but it had extensive quotes from Joe Sutter and other engineers, and discussed the early teething troubles in detail.
In the early 747 development, one of the biggest problems was the JT9D engine, one of the most powerful engines conceived at the time. The initial prototype version, the JT9D-1, had insufficient thrust even when it was working, which wasn't often. Pratt & Whitney weren't sure how much more power they could eke out of it in time for EIS, and the wear rates were abysmal.
So, uncertain that the plane would have sufficient thrust at the required MTOW for the intended passenger, baggage, and fuel load, Boeing and P&W considered two potential alternatives.
- One option was a 5th engine (another JT9D-1) buried in the rear fuselage and fed through an S-duct intake. This was a desperate crutch, since it would increase fuel burn, increase maintenance, and decrease dispatch reliability, but it would guarantee enough power. See note below.
- The second option was stranger. They considered installing two JT3D engines alongside the rear fuselage, with a T-tail design. The JT3D was the first-generation turbofan used on the 707, DC-8, and C-141, so just imagine two 707 turbofans mounted like the engines of a DC-9, and you've got the idea. I'm not sure they would have been used in cruise flight.
In the end, the "improved" (but still hideously unreliable and finicky) JT9D-3 engine provided enough power to (just) meet the requirements, and the 5- and 6-engine options were shelved. From there, the -3/-3A variants were replaced by the more powerful and actually reliable JT9D-7 series, and the rest was history. For growth projects after that point, they focused on increasing power -- first with the JT9D-23 planned for the Phase B program, then on with PW4000, etc.
There were several other non-standard engine configurations pondered, too.
Note: The center-engine placement for the 5-engine concept came back twice in the 747's history, both in the context of 3-engine 747s. It came up first early in the 747's history, even before EIS. Boeing proposed three-engined variants to compete with the DC-10 and L-1011, as mid-range alternatives, which they dubbed the 747-300 (sometimes, 747-300B) the so-called 747-300B. Different inlet and outlet configurations were considered (the 747-305B and -306B were two of them, plus another with a boundary-layer annular inlet). This was abandoned due to lack of interest. A few years later, with the 747 actually in service, Boeing tried again with the 747-3, the idea being that airlines might now like the fleet commonality it allowed. The 747-3 centered around a proposal for for United Airlines. It would have used an almost straight-through center engine built atop a redesigned aft fuselage. There's no trijet that really resembles it. Most of the trijet proposals were for shorter-body 747s, but unlike the SP, these were designed for shorter-range, domestic and transcontinental routes. The 747-3, for example, would have been less than 200 feet long (and lost 20 feet of wingspan), carried 301 passengers, and only flown 2,600 nautical miles.
In appearance, the trijets varied. Some proposals had T-tails, others had cruciform-style tails like the Sud Caravelle, and some had fairly conventional tails (like the 747-3). One "747-300" proposal even had a stretched upper deck for 44 people, even though it had a shortened body just 174 ft. long and spanning 160 ft. (about 57 ft. shorter and 25 ft. narrower than the 747-100) [FI, 1968-03-28].
I've even seen claims some 747SP development ideas included a trijet with an asymmetrical configuration: twinned engines on one pylon on one wing, a single nacelle on the other. I have no idea how that would have worked safely on takeoff, or even if it's true; that's an internet-forum claim.
Early in the 747 development, there were musings from Boeing about a "big twin" version. This was actually called the 747-200 in some initial articles in Flight International and other publications. There's not a ton of public info to be had, but the references talk about 1,200 - 1,800 n. mi. ranges, 200-240 passengers, but still using JT9D engines. This was conceived as a competitor more to the original Airbus A-300 concept, before the actual A300 [FI, 1968-05-02].
Finally, in the unusual engines department, Boeing considered several reduced-power 747s for US domestic operators. One, a 747-100B with a stretched upper deck (SUD), was used in Japan, but Boeing considered a version powered by four JT9D-7R4E engines (the 767 configuration).
Then, still in 1982, they considered a down-rated 747SP for transcontinental routes. This would have had four engines, too, but they would have been PW2037 or RB211-535E4 models, the same engines as the 757.
So, that summarizes pretty much all (known to me, anyway) 747 projects that weren't quadjets, plus the weaker quads. For everything else, Boeing felt that four engines would be powerful enough, if the engine manufacturers developed increased-thrust variants.
The heaviest 747 proposals would have had (or do have) far more power.
- 1970s stretch studies: JT9D-23, 60,000 lb. thrust range
- 747 Super, 747-500, 747X: GE90 derated variant (GE90-68?), GE uprated CF6-80E1, PW ADP, PW4168, Rolls-Royce Trent 700, 70,000-75,000 lb. thrust
- 747-500X and 747-600X: Rolls-Royce Trent 900 or Engine Alliance GP7176, 76,000-80,000 lb. thrust
- 747X and 747X Stretch: Rolls-Royce Trent 600 or Engine Alliance GP7168 / GP7172, 68,000-72,000 lb. thrust
- 747-8 Intercontinental: GEnx 66,500 lb. thrust
So, when you think about the change from a 43,500 lb. thrust JT9D-3A, the 747-8I is carrying six engines worth of thrust!