In a nutshell: The four engined jumbo jet as a airliner was killed by smaller ETOPS-certified twinjets with extended ranges. The smaller twins are just more economical for the airlines and while slightly cheaper to buy new than a 747 or an A-380, they really beat the jumbos in terms of maintenance costs and airfield operation.
The 3- and 4-engined jumbo jets were the kings of the 1970s-1990s as engine technology during that period had not advanced to the point that reliability for extended overwater flights demanded an airplane with more than two engines onboard, justifying the extra operating costs associated with that design. So as an operator faced with rapidly growing international air travel market and a choice between buying a 707 or DC-8 vs a 747, DC-10, or L-1011 and have the increased range and decreased operating cost per passenger mile combined with government regulation of the airlines to soften the financial blow in case the gamble didn't pay off for them, it made perfect sense to upgrade to the jumbo for their intercontinental operations.
In order to make its owner money, an airliner must be in the air with a full load of paying passengers going places. Can't fill the seats because of travel demands? Can't operate your jet from shorter airfields where your travelers are or want to go to nonstop? Too bad. You're going to lose money - and quite often a LOT of money on a failed gamble like that. If you own or operate a jumbo jet you have to fly it like this round the clock, stopping only for passenger transfers, refueling, re-servicing and maintenance. Periods of time where the aircraft is not doing this - even short ones - can cost the airlines literally millions of dollars in lost revenue with is why AOG is such a dreaded term.
With the dawn of the new millennium, air travel has changed. Customer demands for long range, non-stop travel is more fluid and there is a greater profit to be made if you can provide the service from smaller satellite airports as opposed to limiting this to just the major hubs. This combined with improvements in engine technology which made for the ETOPS-certification of twins for intercontinental routes and medium sized, extended range, twin wide body aircraft like Boeing's 767-300, 787 and the Airbus A350 fit much better into that niche, cost less both to buy and for the major budget buster of the airlines, the operating costs.
The operating costs of a 747 or an A-380 are similar to the costs of operating two A-330s or two 787s, except that its a single aircraft and cannot be rerouted depending on passenger loads. The massive size of weight of the superjumbo jet (over 1.2 million lbs MTOW for an A-380!) requires long runways, wide taxiways with good clearance and parking ramps and gates which can accommodate the aircraft while the smaller jets fit better with established infrastructure at smaller airports, which gives a lot more flexibility for operations by the airlines to match passenger trends. If travel demands change, simply re-route one or both aircraft to different fields by schedule. Simple as that.
Bigger airplanes need more power, which means more fuel and maintenance per flight hour, mitigated somewhat by passenger loads. Having a four engine aircraft adds a LOT more in maintenance costs, effectively doubling the op costs over a twin. The dawn of the ETOPS twin made this apparent to the airlines and they quickly went over to medium-large twin engine aircraft for international travel. This is strikingly obvious if you look at the orders for the A380's ancestor, the A340. Sales for this airplane were quite strong in the 1990's but had decreased to a trickle by 2010 and beyond; I would be surprised if this aircraft was still in production by 2020.
There is one area where the superjumbo remains king of the air and that's in hauling freight. Freight doesn't care much about hanging around airport terminals, changing planes and gates, flight delays, etc., so the hub-and-spoke and op cost per pound per mile advantages that a large aircraft like a 747 or A380 offers. When I worked for Boeing as an intern in the summer of 2000, I only saw one passenger 747-400 on the lines; the rest were 747-400F freighters. If you have ever watch the IMAX film 'Living in the Age of Airplanes', it quite convincingly advocates that our modern way of life would simply not be possible without these mechanical giants moving parts and parcels around the globe.