Which trigonometric algorithm does an FMS use to determine distance between waypoints?

When entering waypoints into an FMS, the FMS calculates bearing and distance between them.

Does it use "normal" spherical trigonometry equations, or go for the more accurate Vincenty algorithm which takes the earth to be an oblate spheroid?

• Welcome to Aviation.SE. I have to admit I am torn on this one, I am not sure this might be on topic here, but I would wait also for someone else's opinion. – Federico Oct 6 '15 at 12:14
• I think it is on topic. I remember seeing spherical trigonometry equations in FMS technical docs, but nothing I have nothing I can refer to. – DeltaLima Oct 6 '15 at 12:29
• (I'm no expert) but far from the poles, the difference is too little to play much role. Imprecisions in windspeed are more significant than that. So IMHO, it doesn't matter. – yo' Oct 6 '15 at 13:03
• No need for spherical trigonometry when a sphere is concerend. Simple linear algebra is sufficient to determine distance along a great circle route. Less CPU cycles required too. But I have no idea if it's used in an FMS. – Steve H Oct 6 '15 at 14:27
• Given that a \$50 handheld GPS will use a WGS84 spheroid, and happily convert between alternative spheroids, I'd be disappointed if an expensive FMS used a cruder model. – RedGrittyBrick Oct 6 '15 at 17:57

Aeronautical data (fixes, waypoints, navaids, instrument procedures) are delivered to aircraft operators and avionics manufacturers referenced to the WGS-84 ellipsoid.

To make is simple the ellipsoid is the volume enclosed by the revolution of an ellipse which best approximates the shape of the Earth, generally in a given region only. WGS ellipsoid is a global one which approximates the whole Earth shape.

FAA provides rules for designing procedures, and algorithms to use to compute various elements, including the distance between two fixes and their relative azimuths. This is described in Order 8260.54 The United States Standard for Area Navigation (RNAV), Appendix 2. TERPS Standard Formulas for Geodetic Calculations:

1.0 Purpose

The ellipsoidal formulas contained in this document must be used in determining RNAV flight path (GPS, RNP, WAAS, LAAS) fixes, courses, and distance between fixes. Notes: Algorithms and methods are described for calculating geodetic locations (latitudes and longitudes) on the World Geodetic System of 1984 (WGS-84) ellipsoid, resulting from intersections of geodesic and non-geodesic paths. These algorithms utilize existing distance and azimuth calculation methods to compute intersections and tangent points needed for area navigation procedure construction. The methods apply corrections to an initial spherical approximation until the error is less than the maximum allowable error, as specified by the user.

[...]

3.4 Direct and Inverse Algorithms.

The Direct and Inverse cases utilize formulae from T. Vincenty’s, Survey Review XXIII, No. 176, April 1975: Direct and Inverse Solutions of Geodesics on the Ellipsoid with Application of Nested Equations.

FMS use ellipsoid geometry to compute distances and angles, in order to match aeronautical data.

A visible consequence is that, except for meridians and equator, prlongating a geodesic on an ellipsoid doesn't draw a closed curve, as if it were a great circle:

New modes of navigation have emerged with a significantly better accuracy than using VOR-DME and inertial:

• RNAV (area navigation) generally uses GNSS (global navigation satellite system, one GNSS being the US GPS),
• and more generally performance-based navigation (PBN).

At the same time these new direct modes of navigation (compared to a navaid-to-navaid paths) can define paths extending to thousands of NM, and commonly crossing different geodetic reference systems where latitude, longitude, height and vertical can have significantly different values.

As an illustration, the measure of "height" used to be referenced to local MSL (mean sea level):

(source)

But a GPS receiver, which is now the main instrument to locate a point, can only provide the height above an ellipsoidal simplified Earth model. Latest receivers also provide the height above the simplified sea level model (geoid), but only as the result of a transformation of the ellipsoidal height, etc.

(source)

As visible above, the direction of the local vertical is impacted by the transformation, creating another problem for airborne vehicles navigation.

This can quickly become a nightmare with differences for the same point up to 100m depending on the reference used:

For these reasons, it appeared very early that all elements involved in mapping, navigation and ATC must use the same datum, in all countries. ICAO has selected WGS-84.

The Council [...] on 28 February 1994, adopted Amendment 28 to Annex 15, introducing the provisions concerning the promulgation of WGS-84-related geographical coordinates. [...]

WGS-84 on Wikipedia:

The WGS 84 datum surface is an oblate spheroid (ellipsoid) with major (equatorial) radius a = 6378137 m at the equator and flattening f = 1/298.257223563. The polar semi-minor axis b then equals a times (1−f), or 6356752.3142 m.

Current country compliance with ICAO selected datum: See Jeppesen WGS-84 Status Report.

Aeronautical databases

FMS uses aeronautical data (fixes, procedures) in the ARINC 424 format, where legs are WGS-84 paths.

Processing of aeronautical data

I've had difficulty to get the exact authoritative document making mandatory the use of ellipsoid geometry in a FMS, however, from FAA Order 8260.52 related to RNP, FMS are part of the so called GPN means which are referenced to WGS-84: