I would like to learn about the flight distance calculation and positional accuracy requirement in aviation.

In practice, great circle distance is calculated between two airports but earth is more accurately represented with an ellipsoid and in this context, global positioning is based on WGS84 ellipsoid.

Is it useful to calculate geodesic (shortest path on the ellipsoid) rather than great circle on the sphere for more accurate estimation of the distance?

When thinking that flight routes are 3D, are 3D distances calculated in flight planning? In addition, are these 3D distances calculated with elliposidal cartesian (ECEF) coordinates (X,Y,Z) regarding ellipsoid height (height(MSL)+geoid height (undulation))?.

Does it make sense to calculate the flight distance in "meter" against "km" precision? If it is in nautical mile, what is the precision? e.g. 2394 nm or 2394.xx nm?

My another question is how accurate the 3D position of an aircraft is determined? What is the future perspective in this context together with satellite-based navigation and augmentation systems?

  • 2
    $\begingroup$ Is this a homework question? $\endgroup$
    – Steve V.
    Mar 28, 2017 at 0:58
  • $\begingroup$ Positional accuracy is tied to the position source, for WAAS GPS, this is around 7.6m worst case and about 1m on average. I don't see any need to calculate flight distance sub meter, great circle is accurate enough unless you are landing delivery drones on somebody's doorstep. $\endgroup$
    – Ron Beyer
    Mar 28, 2017 at 1:28
  • 1
    $\begingroup$ Every distance calculation I've ever seen uses WGS84 or NAD83. When they say "great circle" I'm sure calculations are actually using an ellipsoid model. $\endgroup$
    – TomMcW
    Mar 28, 2017 at 2:16
  • $\begingroup$ It would very much help to know what context you want information about. Airlines? The military launching a cruise missile? Me flying the Cherokee to a dirt strip somewhere? $\endgroup$
    – jamesqf
    Mar 28, 2017 at 18:05
  • $\begingroup$ The context is civil aviation. $\endgroup$
    – K. Johnson
    Mar 28, 2017 at 23:30

3 Answers 3


The aviation industry is in the process of moving to Performance Based Navigation (PBN) within Required Navigation Performance (RNP) airspace. Using this structure the primary performance is measured by the Total System Error (TSE) which must be less than 2 times the RNP value 95% of the time. Example: In RNP 1 airspace the aircraft must maintain positional accuracy within 2.0 NM of its desired path 95% of the time. The standard for navigational equipment (FMS) is defined in RTCA document DO-283B, Minimum Operational Performance Standards for Required Navigation Performance for Area Navigation.

The TSE for an aircraft is broken down into three components; path definition error (PDE), position estimation error (PEE), and path steering error (PSE). Your question directly relates to PDE. The accuracy of PDE depends on several components, the primary one being the accuracy of the positional (fix) information published by the regulatory agency (FAA in the US) contained within the navigational database. For enroute and terminal airspace, the accuracy of fix information is less than 0.01 minutes of latitude and longitude or about 17 meters.

Using the data from the database and the WGS84 earth model, the FMS will compute lateral path data. Lateral paths are computed as a geodesic on the surface of the WGS84 ellipsoid.

DO-283 specifies the following data (among other) resolutions for display and entry:

Distance: 0.1 NM for values <10 NM, 1 NM for values >=10 NM
Fix latitude/longitude: 0.01 min
RNP: 0.01 NM for values <1.0 NM, 0.1 NM for values >=1.0, and <10 NM, 1 NM for values >=10 NM
EPU: 0.01 NM for values <1.0 NM, 0.1 NM for values >=1.0, and <10 NM, 1 NM for values >=10 NM
Present Position latitude/longitude: 0.1 min

DO-283 also states rounding of displayed data is desired. For that reason internal FMS computations normally are performed to 1 decimal place more than the displayed resolution and then rounded.

From the above you can see that the displayed distance of a leg should be within 0.1 NM for distances less than 10 NM and 1 NM for longer distances.

As for your 3-D question, lateral (LNAV) and vertical (VNAV) navigation are computed separately. VNAV is still predominately barometric based except for precision (GBAS) and near precision (SBAS) approaches which use WGS84 geometric height.

For positional data, along with the present position, the system must calculate an estimated positional uncertainty (EPU) as shown in the table. The EPU calculation is quite complex and varies based on the source of the positional data (GPS, IRS, etc.) This value when added to the PSE (cross-track deviation + errors) should be held within the RNP value. As you can see form the structure of the EPU requirement, it is the most significant component of the system error.


Calculating a geodesic path is required by the MOPS. Great circle is not acceptable.

Paths are calculated using lat/lon and baro altitude coordinates. Distances are output in nautical miles; not feet, meters, or km. Internally, the systems can use what ever they want. Altitude is in feet or meters, depending on airspace rules. The 3-D path is used to predict time of arrival at each fix along the route.

Positional accuracy (EPU) is calculated based on the selected nav sensor(s) using a number of statistical methods including Kalmann filters.

  • $\begingroup$ Really thank you for the detailed information. Could you please explain the distance precision for whole path? e.g. aaaa nm or aaaa.aa nm? $\endgroup$
    – K. Johnson
    Mar 29, 2017 at 1:38
  • $\begingroup$ It's important that we separate internal accuracy from display accuracy. Obviously, the internal accuracy has to be greater than or equal to the display accuracy. Internally, all distances are computed to 0.01 NM. that's about the best you can do since that's about the accuracy of the fix information (0.01 min of lat or lon ~ 0.01 NM). You need to use the best accuracy you have as you build the flight plan to minimize the additive effect of the error over the entire flight -- error/leg x #legs = total error. $\endgroup$
    – Gerry
    Mar 29, 2017 at 12:09
  • $\begingroup$ For a display, it's about providing data useful to the pilot. Flying a 300 mile leg at 450 kts is a 40 minute leg. That's one mile every 8 sec, so displaying XXX NM is reasonable while cruising. When you get near a fix the pilot needs to anticipate the upcoming turn, so the distance-to-go display shifts to XX.X NM within 10 NM of the fix. At cruise, those tenths tick off at less than a sec. A hundredth of a NM display would be unreadable. On approach, it's still less than 3 sec per .1 NM. And these are minimums. Some FMS display XX.X below 100 NM and X.XX below 10 NM. $\endgroup$
    – Gerry
    Mar 29, 2017 at 12:17

The requirement on Performance Based Navigation (PBN) equipment is that: (1) The Total System Error (TSE) must be equal to or less than the Required Navigation Performance (RNP) value for the procedure for 95% of the flight time; and (2) The probability that the TSE of the aircraft exceeds the specified TSE limit (equal to two times the RNP value) without annunciation (notification of the pilot) is less than 10^(−5).

My experience (doing planning work and analysis) is that the distance between two airports is calculated 'along the ground' and does not account for altitude.

The distance between two points (e.g., airports) computed using a great circle approximation is less that 0.5% of the distance for ellipsoidal earth (and usually less that 0.3%). Vincenty's algorithm is one good method for calculating distance along the ellipsoid surface. Matlab implementations are available on the web.

The problem with using Cartesian (ECEF) coordinates is you almost have to then calculate a straight-line distance for a path that pierces the earth, rather that a distance along the earth's surface (which is what you want).

  • $\begingroup$ I couldn't understand the problem with using ECEF. You have to use ellipsoidal height (Hmsl+Hgeoid) along with other parameters to calculate the 3D position of an aircraft. You're right that straight-line distances will be calculated but in the air not on ground or on a reference surface (sphere or ellipsoid). The distances are calculated on a reference surface not on the ground in fact. Let's assume the difference between ellipsoidal and spherical calculation method is about several nautical miles. Will you ignore this difference? $\endgroup$
    – K. Johnson
    Mar 28, 2017 at 23:38
  • $\begingroup$ The problem with ECEF is that it would require two coordinate translations. All your location data is in lat/lon/elevation. You'd have to translate that to ECEF and then do the reverse after your computation. That's all just added work. As for the difference between geodesics and great circles, the difference may be a few miles for most routes but it does impact ATC when they expect everyone to following the same route. And technically, the great circle would require a translation from/to WGS84. $\endgroup$
    – Gerry
    Mar 29, 2017 at 16:09
  • $\begingroup$ Re the problem with using ECEF coordinates. The comment by Gerry is correct, but not complete. If you know the Lat/Lon coordinates of two airports that are far apart, then the great circle and ellipsoidal earth distances are readily calculated. You could convert both Lat/Lon pairs to ECEF, are you then going to use the Pythagorean theorem to find the distance between the airports? If you do, you're calculating the straight-line (Euclidean) distance between the airports. $\endgroup$
    – user14736
    Mar 31, 2017 at 17:40
  • $\begingroup$ In other words, total sum of the lengths of 3D line segments gives the length of fligth path (3D polyline). $\endgroup$
    – K. Johnson
    Apr 1, 2017 at 20:38

The WGS84 standard defines the Earth as an ellipsoid with a semi-axis a = 6378137 m at the equator and a semi-axis b = 6356752.3 m at the pole.

The circumference of that ellipse can be approximated by:

C(app)= π (3(a + b) – ((3a+b)(a+3b))^0,5)

C(app) = 40007862.2 m

Now, if we take an sphere with r = (a+b)/2 = 6367444.65

C = 2πr = 40007834 m

The difference is 28.21 meters for the full circumference. Hence, for the usual distances in aviation, the difference will be of the order of a few meters, and that in the worst case…


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .