Has there ever been a competitor that is a four-engine plane from Boeing that rivals the A380?
The premise of the question is flawed: the A380 competed unsuccessfully against the 747.
The 747 was produced for about 55 years. The A380 was in production for about one third as long, about 18 years. Over 1,500 747's were built, just over 250 A380's, so one sixth as many airframes sold. The 747 was still in production a year after the last 380 was delivered.
The market spoke, and while all large 4-engine jets are fading away in light of the power and reliability of the engines like those on the 777, A330, A350, and 787, the 747 was vastly more successful economically than the A380.
Boeing did the same market studies that Airbus did and came up with quite different conclusions.
Airbus concluded there was a massive market for very large aircraft operated on trunk routes to eventually replace existing 747s. They designed the A380.
Boeing concluded that the future market was for a larger number of smaller aircraft with shorter turnaround times, and designed the 787 (originally they tried for the Sonic Cruiser, adding increased speed into the mix as well, which idea they couldn't get to work).
So no, Boeing didn't try to compete with the A380 because it wasn't in their vision for the future of civil aviation.
Actually Boeing and Airbus cooperated for a couple of years at the beginning of the project:
In January 1993, Boeing and several companies in the Airbus consortium started a joint feasibility study of a Very Large Commercial Transport (VLCT), aiming to form a partnership to share the limited market. In June 1994, Airbus announced its plan to develop its own very large airliner, designated as A3XX. Airbus considered several designs, including an unusual side-by-side combination of two fuselages from its A340, the largest Airbus jet at the time. The A3XX was pitted against the VLCT study and Boeing's own New Large Aircraft successor to the 747. In July 1995, the joint study with Boeing was abandoned, as Boeing's interest had declined due to analysis that such a product was unlikely to cover the projected $15 billion development cost.
Boeing actually knew about the A380 project for some time. Boeing, however, chose to develop the 787 Dreamliner and 737-MAX instead. Boeing figured the market was headed away from the hub-and-spoke model that the 747 had dominated for decades. While there are still airlines that heavily rely on that model, they are getting rarer. (2013 article)
The airline's stance reflects the broader aviation picture. The purchase of an A380 or 787 is far less a decision between manufacturers than a question of competing visions of the economics and future of air travel. Airlines carrying high volumes of passengers on major routes favour the A380, and Emirates - by far the largest customer - has refitted its Dubai hub to service its fleet, which is set to reach 140 planes. Five depart Heathrow for Dubai daily. At crowded airports where slots are at a premium, putting more customers on each plane taking off can make real sense.
But if the A380 is, as an Airbus spokesman put it, a "congestion buster", the 787 is better known as a "hub buster" – one that rewrites traditional models of funnelling connecting passengers onto the jumbos that make long-haul flying viable. With fewer seats, its fuel efficiency and range are enabling the likes of a direct Heathrow-Austin connection – BA's next new route for 2014 – even with less passenger demand than established routes.
One reason for that is that in the days of the 747's launch, very few lesser cities had commercial airports and air travel was expensive. So if you wanted to travel to Europe, you'd fly to JFK in New York and connect in a wide body to another hub in Europe. Nowadays, you can get direct flights from many places and skip the hubs entirely. The airlines fly smaller jets to accommodate the demand. This is what contributed heavily to the doom of the A380 line