The aviation industry is in the process of moving to Performance Based Navigation (PBN) within Required Navigation Performance (RNP) airspace. Using this structure the primary performance is measured by the Total System Error (TSE) which must be less than 2 times the RNP value 95% of the time. Example: In RNP 1 airspace the aircraft must maintain positional accuracy within 2.0 NM of its desired path 95% of the time. The standard for navigational equipment (FMS) is defined in RTCA document DO-283B, Minimum Operational Performance Standards for Required Navigation Performance for Area Navigation.
The TSE for an aircraft is broken down into three components; path definition error (PDE), position estimation error (PEE), and path steering error (PSE). Your question directly relates to PDE. The accuracy of PDE depends on several components, the primary one being the accuracy of the positional (fix) information published by the regulatory agency (FAA in the US) contained within the navigational database. For enroute and terminal airspace, the accuracy of fix information is less than 0.01 minutes of latitude and longitude or about 17 meters.
Using the data from the database and the WGS84 earth model, the FMS will compute lateral path data. Lateral paths are computed as a geodesic on the surface of the WGS84 ellipsoid.
DO-283 specifies the following data (among other) resolutions for display and entry:
Distance: 0.1 NM for values <10 NM, 1 NM for values >=10 NM
Fix latitude/longitude: 0.01 min
RNP: 0.01 NM for values <1.0 NM, 0.1 NM for values >=1.0, and <10 NM, 1 NM for values >=10 NM
EPU: 0.01 NM for values <1.0 NM, 0.1 NM for values >=1.0, and <10 NM, 1 NM for values >=10 NM
Present Position latitude/longitude: 0.1 min
DO-283 also states rounding of displayed data is desired. For that reason internal FMS computations normally are performed to 1 decimal place more than the displayed resolution and then rounded.
From the above you can see that the displayed distance of a leg should be within 0.1 NM for distances less than 10 NM and 1 NM for longer distances.
As for your 3-D question, lateral (LNAV) and vertical (VNAV) navigation are computed separately. VNAV is still predominately barometric based except for precision (GBAS) and near precision (SBAS) approaches which use WGS84 geometric height.
For positional data, along with the present position, the system must calculate an estimated positional uncertainty (EPU) as shown in the table. The EPU calculation is quite complex and varies based on the source of the positional data (GPS, IRS, etc.) This value when added to the PSE (cross-track deviation + errors) should be held within the RNP value. As you can see form the structure of the EPU requirement, it is the most significant component of the system error.
Summary:
Calculating a geodesic path is required by the MOPS. Great circle is not acceptable.
Paths are calculated using lat/lon and baro altitude coordinates. Distances are output in nautical miles; not feet, meters, or km. Internally, the systems can use what ever they want. Altitude is in feet or meters, depending on airspace rules. The 3-D path is used to predict time of arrival at each fix along the route.
Positional accuracy (EPU) is calculated based on the selected nav sensor(s) using a number of statistical methods including Kalmann filters.
