Now I know the definition of transition altitude - it's the altitude where pilots change their barometric altimeter datum from regional pressure setting to 1013.2hPa. In USA the TA is 18,000ft. But I just don't understand the whole concept. Why can't they just set the datum as 1013.2hPa at ground before takeoff. Why does it have to be 18,000ft. Also isn't the atmospheric pressure at 18,000ft much lower than 1013.2hPa, so isn't it wrong to put this as the datum?
The atmospheric pressure is dropping quite signigicantly with altitude - and this is the very point of the concept of the altimeter - messure the pressure and show this as an altitude.
The basic concept of a flight level is that all high flying planes share a common reference - and do not neet to consider local weather, which can change quite dramatically during a flight. The main topic here is traffic seperation - so all should use the same reference and thus can quite compare their altitudes.
But flying low there is a nice problem called "ground" - planes should mostly avoid it. On charts the minimum altitude is noted, as well as special places like airports. Here a more precise view on the altitude is needed - so pilots need to set their QNH. Before takeoff, before landing - or when flying low for a longer time quite frequently.
But where to change from one reference to another? That much depends on the location - in the US it is mostly 18000, but in other regions this can differ substantially.
The datum setting is a sea level equivalent reference, it is not the actual pressure at the airport. The altimeter setting moves the whole scale up and down to provide an accurate altitude indication at the elevation for the reporting station. Usually this station is at an airport and if the runway is 3550 feet elevation the reported altimeter setting will result in an accurate altimeter reading when at 3550 feet. This is the only location and altitude where barometric altitude accuracy is certain to match true altitude. This is the best method to avoid contact with the ground and towers and good enough for separating local aircraft.
At altitudes above TA, all flights set the same standard setting for consistent aircraft to aircraft separation at high altitudes and over long distances. This is to reduce setting errors and because ground measurements are less accurate with increasing distance/altitude, and to reduce mistakes because of frequent adjustments in high speed aircraft.
Below TA speeds are generally much lower and flights are shorter so changing settings is less frequent and the flight is physically closer to the reporting station reducing atmospheric errors.
TA changes with country because they have different geography and different aircraft requirements, in the USA there are areas with 14000ft mountains, so it is not practical to have a TA below 18000ft. 18000ft is also the base of class A airspace in the USA and aircraft in class A are required to meet instrument flight rules, below is class E which can have VFR aircraft. VFR aircraft are allowed to use a standard altimeter, IFR requires a sensitive altimeter.
Historically aircraft did not have altimeters that could be adjusted in flight, they were all calibrated to 29.92 [1013.25] and then airport altitude adjustment was by hand calculation, with most low altitude flight segments accomplished visually. Many planes from that time did not have radio or had very poor radio, and prevented reliable reception of updated pressure settings. The calculations were both extra work for the pilot and not sufficiently accurate for high altitudes because of variation in lapse rates and temperatures and lack of weather reports. So everyone used non-adjusted indicated altitude at high altitudes.