When entering waypoints into an FMS, the FMS calculates bearing and distance between them.
Does it use "normal" spherical trigonometry equations, or go for the more accurate Vincenty algorithm which takes the earth to be an oblate spheroid?
Aeronautical data (fixes, waypoints, navaids, instrument procedures) are delivered to aircraft operators and avionics manufacturers referenced to the WGS-84 ellipsoid.
To make is simple the ellipsoid is the volume enclosed by the revolution of an ellipse which best approximates the shape of the Earth, generally in a given region only. WGS ellipsoid is a global one which approximates the whole Earth shape.
FAA provides rules for designing procedures, and algorithms to use to compute various elements, including the distance between two fixes and their relative azimuths. This is described in Order 8260.54 The United States Standard for Area Navigation (RNAV), Appendix 2. TERPS Standard Formulas for Geodetic Calculations:
The ellipsoidal formulas contained in this document must be used in determining RNAV flight path (GPS, RNP, WAAS, LAAS) fixes, courses, and distance between fixes. Notes: Algorithms and methods are described for calculating geodetic locations (latitudes and longitudes) on the World Geodetic System of 1984 (WGS-84) ellipsoid, resulting from intersections of geodesic and non-geodesic paths. These algorithms utilize existing distance and azimuth calculation methods to compute intersections and tangent points needed for area navigation procedure construction. The methods apply corrections to an initial spherical approximation until the error is less than the maximum allowable error, as specified by the user.
3.4 Direct and Inverse Algorithms.
The Direct and Inverse cases utilize formulae from T. Vincenty’s, Survey Review XXIII, No. 176, April 1975: Direct and Inverse Solutions of Geodesics on the Ellipsoid with Application of Nested Equations.
FMS use ellipsoid geometry to compute distances and angles, in order to match aeronautical data.
A visible consequence is that, except for meridians and equator, prlongating a geodesic on an ellipsoid doesn't draw a closed curve, as if it were a great circle:
Detailed answer: Need for WGS-84
New modes of navigation have emerged with a significantly better accuracy than using VOR-DME and inertial:
At the same time these new direct modes of navigation (compared to a navaid-to-navaid paths) can define paths extending to thousands of NM, and commonly crossing different geodetic reference systems where latitude, longitude, height and vertical can have significantly different values.
As an illustration, the measure of "height" used to be referenced to local MSL (mean sea level):
But a GPS receiver, which is now the main instrument to locate a point, can only provide the height above an ellipsoidal simplified Earth model. Latest receivers also provide the height above the simplified sea level model (geoid), but only as the result of a transformation of the ellipsoidal height, etc.
As visible above, the direction of the local vertical is impacted by the transformation, creating another problem for airborne vehicles navigation.
This can quickly become a nightmare with differences for the same point up to 100m depending on the reference used:
For these reasons, it appeared very early that all elements involved in mapping, navigation and ATC must use the same datum, in all countries. ICAO has selected WGS-84.
The Council [...] on 28 February 1994, adopted Amendment 28 to Annex 15, introducing the provisions concerning the promulgation of WGS-84-related geographical coordinates. [...]
WGS-84 on Wikipedia:
The WGS 84 datum surface is an oblate spheroid (ellipsoid) with major (equatorial) radius a = 6378137 m at the equator and flattening f = 1/298.257223563. The polar semi-minor axis b then equals a times (1−f), or 6356752.3142 m.
Current country compliance with ICAO selected datum: See Jeppesen WGS-84 Status Report.
FMS uses aeronautical data (fixes, procedures) in the ARINC 424 format, where legs are WGS-84 paths.
Processing of aeronautical data
I've had difficulty to get the exact authoritative document making mandatory the use of ellipsoid geometry in a FMS, however, from FAA Order 8260.52 related to RNP, FMS are part of the so called GPN means which are referenced to WGS-84:
Geographic Positioning Navigation (GPN).
Navigation based on geodetic calculation of geographic position referenced to the WGS-84 ellipsoid. Global positioning system (GPS), wide area augentation system (WAAS), local area augmentation system (LAAS), flight management system (FMS), RNP, and area navigation (RNAV) are examples of GPN.
When calculating distances between fixes, or angles, a FMS computes geodesics which are shortest paths between fixes. On a sphere this would be a great circle. On an ellipsoid this has to be adjusted, Vincenty's formulas are one of the methods that can be used for approximation.
Additional details on how geodesic calculations are done can be found in FAA Order 8260.58 related to Performance Based Navigation.