When entering waypoints into an FMS, the FMS calculates bearing and distance between them.

Does it use "normal" spherical trigonometry equations, or go for the more accurate Vincenty algorithm which takes the earth to be an oblate spheroid?

  • $\begingroup$ Welcome to Aviation.SE. I have to admit I am torn on this one, I am not sure this might be on topic here, but I would wait also for someone else's opinion. $\endgroup$
    – Federico
    Oct 6, 2015 at 12:14
  • 8
    $\begingroup$ I think it is on topic. I remember seeing spherical trigonometry equations in FMS technical docs, but nothing I have nothing I can refer to. $\endgroup$
    – DeltaLima
    Oct 6, 2015 at 12:29
  • $\begingroup$ (I'm no expert) but far from the poles, the difference is too little to play much role. Imprecisions in windspeed are more significant than that. So IMHO, it doesn't matter. $\endgroup$
    – yo'
    Oct 6, 2015 at 13:03
  • $\begingroup$ No need for spherical trigonometry when a sphere is concerend. Simple linear algebra is sufficient to determine distance along a great circle route. Less CPU cycles required too. But I have no idea if it's used in an FMS. $\endgroup$
    – Steve H
    Oct 6, 2015 at 14:27
  • 5
    $\begingroup$ Given that a $50 handheld GPS will use a WGS84 spheroid, and happily convert between alternative spheroids, I'd be disappointed if an expensive FMS used a cruder model. $\endgroup$ Oct 6, 2015 at 17:57

1 Answer 1


Short answer

Aeronautical data (fixes, waypoints, navaids, instrument procedures) are delivered to aircraft operators and avionics manufacturers referenced to the WGS-84 ellipsoid.

To make is simple the ellipsoid is the volume enclosed by the revolution of an ellipse which best approximates the shape of the Earth, generally in a given region only. WGS ellipsoid is a global one which approximates the whole Earth shape.

enter image description here

FAA provides rules for designing procedures, and algorithms to use to compute various elements, including the distance between two fixes and their relative azimuths. This is described in Order 8260.54 The United States Standard for Area Navigation (RNAV), Appendix 2. TERPS Standard Formulas for Geodetic Calculations:

1.0 Purpose

The ellipsoidal formulas contained in this document must be used in determining RNAV flight path (GPS, RNP, WAAS, LAAS) fixes, courses, and distance between fixes. Notes: Algorithms and methods are described for calculating geodetic locations (latitudes and longitudes) on the World Geodetic System of 1984 (WGS-84) ellipsoid, resulting from intersections of geodesic and non-geodesic paths. These algorithms utilize existing distance and azimuth calculation methods to compute intersections and tangent points needed for area navigation procedure construction. The methods apply corrections to an initial spherical approximation until the error is less than the maximum allowable error, as specified by the user.


3.4 Direct and Inverse Algorithms.

The Direct and Inverse cases utilize formulae from T. Vincenty’s, Survey Review XXIII, No. 176, April 1975: Direct and Inverse Solutions of Geodesics on the Ellipsoid with Application of Nested Equations.

FMS use ellipsoid geometry to compute distances and angles, in order to match aeronautical data.

A visible consequence is that, except for meridians and equator, prlongating a geodesic on an ellipsoid doesn't draw a closed curve, as if it were a great circle:

enter image description here

Detailed answer: Need for WGS-84

New modes of navigation have emerged with a significantly better accuracy than using VOR-DME and inertial:

  • RNAV (area navigation) generally uses GNSS (global navigation satellite system, one GNSS being the US GPS),
  • RNP (required navigation performance),
  • and more generally performance-based navigation (PBN).

At the same time these new direct modes of navigation (compared to a navaid-to-navaid paths) can define paths extending to thousands of NM, and commonly crossing different geodetic reference systems where latitude, longitude, height and vertical can have significantly different values.

As an illustration, the measure of "height" used to be referenced to local MSL (mean sea level):

enter image description here

But a GPS receiver, which is now the main instrument to locate a point, can only provide the height above an ellipsoidal simplified Earth model. Latest receivers also provide the height above the simplified sea level model (geoid), but only as the result of a transformation of the ellipsoidal height, etc.

enter image description here

As visible above, the direction of the local vertical is impacted by the transformation, creating another problem for airborne vehicles navigation.

This can quickly become a nightmare with differences for the same point up to 100m depending on the reference used:

For these reasons, it appeared very early that all elements involved in mapping, navigation and ATC must use the same datum, in all countries. ICAO has selected WGS-84.

ICAO Doc9674:

The Council [...] on 28 February 1994, adopted Amendment 28 to Annex 15, introducing the provisions concerning the promulgation of WGS-84-related geographical coordinates. [...]

WGS-84 on Wikipedia:

The WGS 84 datum surface is an oblate spheroid (ellipsoid) with major (equatorial) radius a = 6378137 m at the equator and flattening f = 1/298.257223563. The polar semi-minor axis b then equals a times (1−f), or 6356752.3142 m.

Current country compliance with ICAO selected datum: See Jeppesen WGS-84 Status Report.

Aeronautical databases

FMS uses aeronautical data (fixes, procedures) in the ARINC 424 format, where legs are WGS-84 paths.

enter image description here

Processing of aeronautical data

I've had difficulty to get the exact authoritative document making mandatory the use of ellipsoid geometry in a FMS, however, from FAA Order 8260.52 related to RNP, FMS are part of the so called GPN means which are referenced to WGS-84:

Geographic Positioning Navigation (GPN).

Navigation based on geodetic calculation of geographic position referenced to the WGS-84 ellipsoid. Global positioning system (GPS), wide area augentation system (WAAS), local area augmentation system (LAAS), flight management system (FMS), RNP, and area navigation (RNAV) are examples of GPN.

When calculating distances between fixes, or angles, a FMS computes geodesics which are shortest paths between fixes. On a sphere this would be a great circle. On an ellipsoid this has to be adjusted, Vincenty's formulas are one of the methods that can be used for approximation.

Additional details on how geodesic calculations are done can be found in FAA Order 8260.58 related to Performance Based Navigation.

  • $\begingroup$ If that's the short answer, I am not going to attempt to read the long answer!!! $\endgroup$
    – FreeMan
    Oct 27, 2015 at 15:58
  • 2
    $\begingroup$ @FreeMan: LoL. If you read all of the post and don't want the long answer, then the only possibility is to try to clear it from your memory :-) because they are both in the post (but it was not said where one finishes and the other begins). That's a trap I use, because nobody wants to read my long answers. $\endgroup$
    – mins
    Oct 27, 2015 at 20:23
  • 1
    $\begingroup$ Actually, I skimmed through, went "ooohhh pretty pictures!" and wrote a snarky comment. Just like I did in calculus class. $\endgroup$
    – FreeMan
    Oct 27, 2015 at 20:39
  • $\begingroup$ The answer is very good for content however more would be better, not exactly a topic that is approachable with short answers. As tend to happen when involving geodetics and its application via models and processes. But really good to try condense it. $\endgroup$
    – jCisco
    Oct 27, 2015 at 20:58
  • $\begingroup$ A spectacular answer. Incidentally so is the FAA Order 8260.54 (link included above) - it contains calculators for some of the functions AND detailed algorithms for easy implementation in a programming language. $\endgroup$
    – Ugo
    Oct 31, 2015 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.