(gcmap.com) Great circle route.
The polar route is mostly over land and is in close proximity to airports and navigational aids. Flying over long stretches over water requires (among other things) enhanced navigation, using specific communication protocols, and adhering to criteria set by the countries overseeing the region. Picking the new Pacific route is not easy (authorization, training, and the associated costs).
The Pacific communication, navigation, and surveillance requirements also allow the utilization of flex routes to adapt to the changing jet stream, as the linked document below explains. (The flight's history also shows the different routes.)
The relevant operational circular—Operational Authorization Process for Pacific Operations—can be found on the DGCA website (.pdf). The document covers all the requirements, of which, the carrier seeking the authorization may need to demonstrate to the authorities a flight or more:
(...) the final step of the approval process may require a validation flight through Pacific airspace by a DGCA Flight Operations Inspector to verify that all relevant procedures are applied effectively.
In short, when an airline selects a new challenging route, it must prove it is prepared for it, and to set up new training procedures. It's like a pilot who has never flown over the Atlantic, when that time comes, they undergo special training for it. In this case, flying over the Pacific was new grounds for Air India.
The requirements (published August 2016) were put in place after Air India sought this route. Air India has been undergoing major changes recently due to large amounts of debt, so they've been streamlining their operations.
Related (question about same flight): Could winds of up to 150 km/h impact the structural loads on a Boeing 777?