There are several angles to this question, so I'll try to address what I think is important. My so-far short career has so far involved working almost exclusively with 747 derivatives, so I don't have as much to say about the A380, but I'll touch on it as I can.
In summary, the decision to modify a commercial aircraft for military use comes down to cost, schedule, performance, and politics. The platform needs to be the right one for the mission, be available in time to execute the mission, be worth the money for its capabilities, and be politically appealing. These four factors are best addressed on a case-by-case basis. The 747 has a much longer and more storied history than the A380, and I am not aware of any thoughts to militarize the latter; however, the 747 has been adapted for military use in several circumstances and there have been even more preliminary studies about doing so that never made it to production. So here is a brief summary of those programs. Hopefully looking at each will satisfy your curiosity and give a picture that answers the question.
Civil Reserve Air Fleet
Adapting an aircraft for military use does not necessarily involve modifying it in any way. The United States has the Civil Reserve Air Fleet (CRAF), a program through which domestic airlines pledge the availability of some of their aircraft to the Department of Defense "to augment the Air Mobility Command's long-range intertheater C-5s and C-17s during periods of increased airlift needs." Currently, none of these aircraft is permanently modified. Perhaps the interior of a passenger model gets reconfigured for a CRAF flight, but they are the same airplanes that members of the public fly on regularly. Back when Delta and United flew 747s, nearly all of them were available to the CRAF; now it is only 747 freighters that are available. The cost-schedule-performance analysis of this setup is favorable. The DOD only pays for the airplanes when it needs them, and is legally allowed to use them any time it needs them. Airlines get compensated at above-market rates when their aircraft are used. And unless you need to land on an austere runway, fly through contested airspace, perform an airdrop, or load/unload extremely quickly on the ground, a 747 can do almost anything that a C-5 or C-17 can, and usually more efficiently.
However, in the late 1980s and early 1990s, about two dozen early-model 747s were structurally modified specifically to support the CRAF, including N739PA:
The fuselage structure of N739PA differed from the majority of Boeing 747 aircraft in that it had been modified to carry special purpose freight containers on the main deck, in place of seats...The effect of this modification on the structure of the fuselage was mainly to replace the existing main deck floor beams with beams of more substantial cross-section than those generally found in passenger carrying Boeing 747 aircraft. A large side loading door, generally known as the CRAF door, was also incorporated on the left side of the main deck aft of the wing.
The CRAF door can be clearly seen in this photo:
Again, the analysis is favorable. The modifications were relatively simple structural ones and the DOD paid an ongoing subsidy to operators of these aircraft to offset the cost of the extra weight, which was much less expensive than maintaining a fleet of military-only aircraft of comparable capability.
I do not know what capabilities there are for NATO or other European countries to press civilian aircraft into service, but they do not seem extensive. This is where I think you'd most likely see an A380 used in a defense role.
Airborne Aircraft Carrier
Here's a fun one that didn't make it very far. The idea was to use a 747 as a flying platform for the launch, recovery, and refueling of small fighter aircraft:
Boeing's report sung the praises of this idea, of course, but my feeling is that it was something of a solution in need of a problem. The USAF had experimented with the idea before and found it to be technically feasible but operationally difficult, which was likely again the case here. Although the proposed schedule was favorable, the performance would not have matched a mission, and the cost of putting the system together and deploying it would likely have been huge.
Advanced Tanker Cargo Aircraft
The 747 was indeed converted for use as a refueling tanker, albeit in small numbers. The KC-10 won over a 747 variant mainly due to lower costs and better overall performance. The KC-10 could operate out of smaller airfields and had an all-new boom, as opposed to the 747 contender's barely-modified KC-135 boom with poorer handling qualities and less clearance away from the aircraft. There were a few other technical aspects favorable to the KC-10 as well, not to mention the inherent ongoing cost savings of flying a trijet over a quad. Additionally, McDonnell Douglas at the time was in need of a lifeline for the DC-10 program and thus applied more political pressure. It should also be noted that the 747-based tanker design was better suited operationally for refueling long-range bombers and transports than fighters. As the other answers and some comments point out, an enormous tanker has some inherent deficiencies when directly supporting combat operations. My opinion is that Boeing focused too much on the long-range mission when trying to sell their solution, but this might be the best example of the 747's size as a direct and insurmountable detriment, whereas in most of its other military roles its size is its greatest benefit.
Despite the US not adopting the 747 solution, Iran purchased a couple of them, and at least one still flies:
For Iran, the cost-schedule-performance triangle was beneficial. My guess is that Boeing offered a discount to Iran to get some tankers operationally deployed to help boost their case with the USAF, they were available almost immediately, and Iran's military operations were more concentrated and less reliant on situations that would put such enormous links in peril.
And just because it's interesting, it was the very first 747, RA001, that was modified to test this refueling concept, as seen here:
RA001 still boasts most of that equipment (the boom is long gone, though), but it should be noted that there were no modifications to the fuel system to allow it to pass fuel—the hardware was for testing tanker-receiver contacts and boom dynamics only. You can see RA001 up close for yourself at The Museum of Flight. And if you do stop by, say hi—I'm one of the volunteers on the restoration crew for this airplane.
Cruise Missile Carrier Aircraft
Another interesting one that never made it past the drawing board, although it got farther along than it usually gets credit for. The idea was patented by Boeing during the B-1 program as a way to supplement that aircraft and ease the B-52 into retirement.
It was eventually canceled in favor of the B-2 and upgrades to the B-52. History has questioned that decision, but at the time costs were deemed too high given commitments to the aforementioned platforms, and the considerable performance was still not found to fit well into the contemporary operational concept.
E-4
The E-4 was the first 747 derivative to enter military service. There is not much to say about it technically that Wikipedia doesn't, but it is worthwhile to note that, in the view of the Air Force, having four engines is a must on this type of platform. In the early 1970s, there wasn't much by way of twins, so that was taken care of at the time of aquisition. Redundancy and survivability are primary design drivers, so although its systems could probably have been transferred to a smaller, cheaper airframe, there is no way to match the mission assurance of four engines. Along with that, it would be tough to find a way to generate enough electricity on the airplane to power all its systems without four engines. Additionally, the first two airframes were offered to the government at a discount due to their being built for a canceled order.
Non-Developmental Airlift Aircraft
More a case study than anything, this might actually be the best example for how cost, schedule, and performance are balanced. I highly encourage you to read this retrospective:
...[the NDAA] was originally developed in the early nineties due to poor cost and schedule performance during phase two...of the C-17 acquisition. Forty C-17s were being built for phase two; and due to problems, alternatives were sought with respect to the remaining 80 aircraft. Attempting to leverage commercial industry, the NDAA promised a low-cost alternative to the DOD acquisition process. However, since this was an alternative for the C-17, DOD stipulated that the NDAA would have to be a jumbo cargo aircraft capable of carrying outsize cargo. The NDAA study focused on a minimally modified Boeing 747 cargo aircraft. These “minimal” modifications included hardened decks and a flipup nose and ramp system for ease of straight-in loading versus the sidemounted-cargo-door style loading of the commercial industry. Consequently, the price for the NDAA alternative increased from under \$150 to about \$200 million per aircraft. Nevertheless, after examining several options, the most cost-effective solution was an 86/30 mix of C-17 and NDAA aircraft. This mix, however, did not allow for a full strategic brigade airdrop nor was it optimized for tactical airlift requirements and lesser regional contingencies in support of peace enforcement scenarios. These requirements were better met by the military-specific C-17. Ultimately, the Defense Acquisition Board decided to procure the fleet of 120 C-17s and no NDAA.
Airborne Laser
The YAL-1 ultimately suffered from cost, schedule, and performance problems. The idea was to fly around a huge laser that could heat up and destroy ballistic missiles from afar. The 747 platform was chosen because it was the only airframe large enough to contain the huge tanks of chemicals needed to generate a laser strong enough to have any effect. But after almost two decades and more than five billion dollars, it still needed a laser many times more powerful to be operationally viable, so it was canceled. One prototype flew for over 10 years, but was unfortunately scrapped.
VC-25A
The VC-25A is generally what people think of as Air Force One, although technically that designation applies to any USAF aircraft with the president onboard. Again, the number of engines was a major design driver, and in this case the size and iconography of the 747 helped it tremendously. The DC-10 was competed as an alternative, although it never was a serious contender and was reportedly only included to get Boeing to lower its price. As with the E-4, survivability, redundancy, and power requirements support a four-engine airplane; the floor space of a 747 also supports a lot of functions that couldn't otherwise coexist. Even by the time the VC-25A was introduced, the 747 had an enduring and recognizable preeminence in the history of aviation and Unites States prestige that the government found attractive to its branding as the president travels the world. Commonality with the E-4 airframe was also a consideration. In this case, politics and performance played an outsized role in selection of the aiframe, with cost and schedule secondary. Obviously there was much willingness to spend the big money and wait some extra time to fulfill all the requirements.
VC-25B
Once again, we have power and redundancy requirements driving the selection of the airframe for the next presidential aircraft. There was some early talk about perhaps pursuing an A380 option, but politics quickly put that idea to bed in favor of the 747-8. The VC-25B will have modifications similar to those of the VC-25A, and will probably look similar:
The program has been much in the news due to cost, schedule, performance, and politics, but is overall proceeding with similar priorities as the VC-25A program did.
Conclusions
I count about 30 747 airframes that have been modified and flown for military purposes. Obviously that's not a huge number, but it is actually fairly respectable for an aircraft of its era, although 707 variants are clearly more ubiquitous (it's important to note, however, that the C-135 family is not a derivative of the 707). Nowadays, with the KC-46 and P-8 leading the way, commercial derivatives are becoming more attractive to the armed forces. But with production of the 747 winding down, we are unlikely to see any more variants beyond the VC-25B: parts will become more difficult to obtain and the labor force skilled with the airframe will dry up. The 747's size and capabilities have both helped and hindered it along the way, and it warrants a case-by-case look at each program to see how cost, schedule, performance, and politics have played a role. The A380 is a somewhat different case, being more recent and less revolutionary in its era than the 747. It has not found a military mission that cannot be accomplished with smaller, cheaper airplanes, and it has no history of modification to help it win over the decision-makers. I am doubtful that we'll ever see it take on a military role. A lot more could be said about any of my examples and the overall picture, but I hope I've dispelled the idea that the 747 is not used militarily, and I hope I've provided some insight as to why it is not used more.
