Why does airfoil used by modern subsonic aircraft (this is true for transonic airliners too) vary from plane to plane?
Haven't we figured out a single best airfoil shape, with the highest lift to drag ratio?
For the same reason why we do not have one single type of aircraft flying all commercial and military missions worldwide: flight has many variables, and there is no single optimum solution.
First off, you mention transonic flight in your question, which usually involves sonic flow at over at least part of the airfoil, which significantly alters the desired properties of the airfoil. To follow with this example, have a look at supercritical airfoils. They exhibit good qualities for their typical flight regime, albeit at substantial development and manufacture cost. A simpler NACA airfoil would be more cost-effective for applications that do not require high-subsonic cruise efficiency as one of their main points.
Additionally, there are more specific situations, like flying wings, that usually use reflex camber to deal with the lack of horizontal stabilizer. Again, this modification is rendered into pointless weight and cost if used outside this particular scenario.
Now, to specifically address your Lift-to-Drag requirement: that is simply not a good way to decide "the best airfoil for subsonic flight". High efficiency is important for aircraft with a mission that emphasizes endurance. This category is populated by gliders, and is indeed full of highly optimized designs ($C_L / C_d \approx 50$ is not unheard of) with little differences, mostly due to structural considerations.
Now take another subsonic aircraft with an emphasis on endurance: the maritime patrol aircraft, the best known example here is probably the P-3 Orion. A glider airfoil would be terrible because of the need for thickness to accommodate a spar that can carry the engines without breaking under a bad gust. Instead it uses a NACA 0014-1.10 at the root, transitioning to a NACA 0012-1.10 at the tip, meaning the designers had to compromise between two airfoils even within the same wing.
In aircraft, size does matter.
Smaller aircraft flying at the same speed, air temperature and altitude than a larger aircraft have a smaller Reynolds number which characterises the boundary layer flow. A smaller Reynolds number allows for more laminar flow but demands a less steep pressure rise in order to avoid early separation. So there is a different airfoil for every design Reynolds number, even at the same lift coefficient. Very fast subsonic designs have to deal with pockets of supersonic flow which put very different demands on the airfoil's shape.
Next, airplanes come with a big variety of wing loadings, from 40 kg/m² in gliders up to the nearly 1200 kg/m² of the Rockwell B-1B. This translates into a very different range of speeds, so the variation in Reynolds numbers is much wider than the size alone implies. Heavier aircraft tend to have higher wing loadings and need to add high-lift devices, which put their own demands on airfoil shape.
Then consider aspect ratio: Depending on the purpose, the optimum aspect ratio of subsonic airplanes is anywhere between 4.5 and 50. The smaller value is for flying fast and high g loads, like in aerobatics, and the higher value is typical for high performance gliders. The short and stubby wing will need a different airfoil than the long and sleek one.
Also, ideally you should vary the airfoil within one wing, depending where along the span you look. The root will benefit from a thicker airfoil than the outer wing, where a wide angle of attack range from aileron deflections and roll speeds needs to be tolerated.
And then there are designs wich care less about L/D but need to have ideal stalling characteristics and no camber: Aerobatic aircraft have quite different airfoils from all the others, and with good reason.
This only scratched the surface of the issue, but I hope I could get the general idea across: Even when optimum L/D is the design goal, the best airfoil shape differs with the other parameters of the aircraft.
Airfoil performance will depend on air speed and density. A Cessna 172 and a DeHavilland Dash 8 are both operating at subsonic speeds, but in very different speed and density regimes. Jet aircraft and propeller aircraft mostly operate in different speed regimes as well. The details of how the wing is laid out will also affect performance. The chord length will change the Reynolds number that the airfoil experiences. The wingspan and sweep angle will also affect the performance of the wing. Different wing geometry in different flow conditions will drive different choices in airfoil.
All of these differences are because different aircraft are designed for different kinds of flying. Gliders, for instance, are designed for a very high lift to drag ratio for minimal drag. Bush planes will place a high emphasis on low speed performance to reduce takeoff and landing distance. Passenger aircraft are willing to sacrifice some of the aerodynamic efficiency to fly faster to reduce trip times. Structural considerations as well as infrastructure will limit wingspan and influence thickness.
Airfoils will also have different characteristics in situations such as stall. For most aircraft, you don't want the stall to be sudden and severe. But for aircraft designed for performance, such as aerobatics or racing, they are willing to sacrifice this for lower drag.
You will see the same trends in nature, as there are many different types of wings. Small wings are very different from large ones. Wings designed for soaring will be different from wings designed for speed.
Generally, we know the best airfoil shape, but we do not know the best speed.
Lift is proportional to the SQUARE of speed. The Cessna 172 is barely above stall at 50 knots, but putting in large amounts of down trim at 100 knots.
But the airliners have it figured out. They can fly in a huge speed envelope of around 150 knots to 550 knots by changing wing configuration, more specifically changing camber.
The highest lift wings are heavily cambered with a concave underside. These "thin undercambered wings" were popular in the early days of flight, but fell out of favor as speeds increased. A thin wing with less camber produces adequate lift at higher speeds but with MUCH LESS DRAG.
So the airliner reconfigures to a thinner less cambered wing for high speed cruise, but returns to the early days to greatly reduce landing speeds. Perhaps we will see a Fiesler Storch with retractable slats.