Further to what's been said above, there's also the weight and maintenance concerns. Having variable sweep wings adds a lot of weight to the primary structure, plus there is a lot of required maintenance required for the sweep machinery. Thus you would always try to avoid having variable sweep, unless it is absolutely necessary.
If we look at the two major US fighter aircraft with swing wings, they both had requirements which drove the design in different directions, leading to the need to adopt swing wings as a solution.
F111: The F111 needed to excel in the high speed, low level penetration role, but also be able to takeoff and land from short unprepared runways near the frontline. Short runways requires high-aspect ratio, unswept wings to lower the takeoff/landing speeds. At high speed, low level, the required lift coefficients are very low, as is the drag, due to the high velocity and air density. To reduce the drag, you need to reduce the frontal area and the wetted surface area, with very low aspect ratio wings. To meet both of these requirements the only way forward is to have variable sweep.
F14: Carrier take-off, and the fleet defense requirement, carrying the (very large) AIM-54 Phoenix a long way at high speed to defend carrier battle groups from Soviet Bombers with anti ship missiles. The carrier takeoff requirement is the same as the short runway, so you want high aspect ratio, low sweep angle. High speed means high M, so you want high sweep angle on the wings. As the AIM-54 is large, and needs a large radar and 2nd crewmember, the aircraft will also be large. Getting enough lift for takeoff/landing at low speed with a highly swept wing was not feasible, and therefore variable sweep wings were required.
More modern aircraft have generally not required the same set of capabilities, and we have a better understanding of the complexities and cost of swept wing aircraft. The aerodynamic understanding applied to the F14 and F15 was almost exactly the same, as they were designed at similar times, it's more that the F15's design mission ('not a pound for air to ground') meant that it was designed to be excellent in air to air combat rather than having short runway requirements or anything like that.