I read a 2016 article that said Air India broke a record for the longest flight (distance) by flying from New Delhi to San Francisco over the Pacific rather than a polar/Atlantic route. The total flight time was shorter because of the jet stream.
Why didn't they always do this?
(gcmap.com) Great circle route.
The polar route is mostly over land and is in close proximity to airports and navigational aids. Flying over long stretches over water requires (among other things) enhanced navigation, using specific communication protocols, and adhering to criteria set by the countries overseeing the region. Picking the new Pacific route is not easy (authorization, training, and the associated costs).
The Pacific communication, navigation, and surveillance requirements also allow the utilization of flex routes to adapt to the changing jet stream, as the linked document below explains. (The flight's history also shows the different routes.)
The relevant operational circular—Operational Authorization Process for Pacific Operations—can be found on the DGCA website (.pdf). The document covers all the requirements, of which, the carrier seeking the authorization may need to demonstrate to the authorities a flight or more:
(...) the final step of the approval process may require a validation flight through Pacific airspace by a DGCA Flight Operations Inspector to verify that all relevant procedures are applied effectively.
In short, when an airline selects a new challenging route, it must prove it is prepared for it, and to set up new training procedures. It's like a pilot who has never flown over the Atlantic, when that time comes, they undergo special training for it. In this case, flying over the Pacific was new grounds for Air India.
The requirements (published August 2016) were put in place after Air India sought this route. Air India has been undergoing major changes recently due to large amounts of debt, so they've been streamlining their operations.
Related (question about same flight): Could winds of up to 150 km/h impact the structural loads on a Boeing 777?
In addition to the other answers, it's worth pointing out that the jet stream does not stay in a constant location above the Earth, but shifts and meanders over the course of weeks and months. Here's a great video from NASA showing these shifts over a 30-day period:
The Pacific route would only be advantageous if the jet stream was relatively far south in its meanderings.
The great circle (shortest path) between DEL and SFO goes nearly straight north then nearly straight south (slightly westerly). It's super close to a route where you would go straight north, then straight south. For instance, I plotted DEL to DEN (Denver-CO), and there we have a straight north route.
The fact is, often times the shorter distance (great circle) actually beats the Pacific route, and the jet stream isn't straight from the west to the east. It does shift to the north and south in many places.
It might be just better to plot a DEL-SFO route where the aircraft goes mostly north but picks up a northeast jetstream in the beginning and then picks up a southeast jetstream in the end. I've seen jets flying in the northern USA on a heading 040 (more north than south) with a 100+ knots tailwind.
The polar route is actually a shorter distance. So, in the absence of winds (and in the absence of political conditions preventing you from overflying certain regions,) the polar route is faster.
The jet stream moves around and its average location and strength vary seasonally. Even within a season, though, its position and strength can vary significantly even from one day to the next.
Taking the Pacific route instead of the polar route adds quite a significant amount of distance to the DEL-SFO flight. As such, it only makes sense to take that route when the jet stream is strong and located conveniently for where you're trying to go. If the jet stream is weaker and/or heading the wrong way for your route, taking the polar route will be faster.
In the absence of wind, the only downside to taking the polar route over the Pacific route is that, if you have to divert mid-flight, you'll end up in some frozen wasteland in Russia, Canada, Svalbard, or Greenland, rather than some tropical island in the Pacific.
