Why does the second floor of the Boeing 747 occupy only part of the plane, while the second floor of the Airbus a380 occupies the entire space of the plane?
The basic design of the Boeing 747 was originally developed to for the US military's CX-HLS program for a large cargo aircraft. One of the main requirements of the program was for cargo to be loaded from the nose, and placing the cockpit above the main cabin made this possible. The Lockheed C-5 Galaxy was selected, but Boeing carried the same features over to the 747.
In the 1960s when the 747 was being developed, some believed that supersonic aircraft would be taking over the long-haul travel market. The relatively short "hump" was retained not for extra passenger capacity, but so that if the passenger market dried up, they could keep producing the aircraft for the cargo market.
Boeing did extend the "hump" further back to add capacity on passenger versions, first on the 747-300 (and later retrofitted to a few -100 and -200 planes), and again on the 747-8I. Extending the second deck all the way back would create too many major changes to the structure and aerodynamics of the airplane, essentially making it a completely new model. Boeing did study a full double-deck model to compete with the A380, but opted to focus on a smaller, more fuel-efficient market with the 787 instead.
According to Joe Sutter, the chief designer of the 747, his team came up with the concept of a wide body as an alternative to a double decker. More efficient use of the available volume, and faster emergency egress. It took some selling, and several mockups, to persuade Pan Am to accept this new and radical design.
The 747 was initially conceived a stopgap effort in the mid 1960's, expected to be overshadowed by the 2707 SST under development. As such, it was modified for use as a cargo aircraft, by moving the cockpit above the main cabin, to facilitate a front opening nose for rapid cargo load/unload. The thinking was - when the 2707 came out, they could still use the 747 for cargo, where high speed wasn't required.
The fairing was added behind the cockpit for aerodynamics, and the extra space used for passengers. Originally, Sutter's plan was to have crew quarters there, to which Juan Trippe of Pan Am replied - Hell no. We're putting paying passengers there.
When the SST was abandoned due largely to skyrocketing fuel costs, the 747 was front and center as the first large airliner to make use of the more efficient high bypass turbofans, lowering fuel usage at a time when fuel costs had gone through the roof.
Why didn't they extend the upper cabin through the entire fuselage, as the A380 did?
The requirement for 400+ passengers from Pan Am was met by the widebody design.
Remember that when the 747 debuted, the typical international airliner (707, DC8) carried 150 passengers. So it was already a huge leap. Even more would have stretched the engineering challenges considerably, when the customer wasn't asking for that.
The 747's cockpit was raised for a different aviation market. The "hump" was extended aft to improve its aerodynamics. Back then a dual-deck airliner was being considered to maximize passengers, but it would have been too heavy. But that was no longer the case for the A380.