I understand during the early days of flight that compasses were regarded as extremely unreliable and unfit for even basic navigation, but I'm unsure as to what caused this unreliability and what the first countermeasures against it were.
The unreliability is caused by the nearness of several big bits of magnetic material, chiefly an engine, and some electrical equipment. The countermeasures are something called "swinging the compass". The same fix has been used historically on boats since they started being made of metal.
In essence it involves adjusting "compensators" which are pieces of magnetic material placed near the compass and which cause it to swing by a few degrees. You point the aircraft in all the different compass directions and make the adjustments in order to minimize the deviation of the compass from an accurate reading throughout all directions. This is done in a "standard configuration" of different electrical equipment being on or off.
Once you have minimized the difference between actual directions and what the compass reads you then make up a "deviation card" which is placed near the compass and which the pilot can use to get the correct magnetic heading from the compass reading.
There is an additional unreliability caused by the aircraft's motion. As an aircraft pitches and rolls that movement causes the compass to deviate from an accurate reading. This would happen more in early planes than in more modern ones. This problem was fixed by the invention of the gyro compass or "Direction indicator", which maintains a constant direction and is not subject to the aircraft pitch and roll. However the DI does drift, and so needs to be regularly checked to ensure it aligns with the compass, something you do during level flight to minimize the compass errors.
Both of these mitigations are in use today.
One common method was to move the guts of the compass out to a wingtip, far away from the engine block and the electrical system, and then electrically link it to a remote reading dial in the cockpit. This was called a flux gate compass, which does not use a swinging needle to detect the direction of the earth's magnetic field and hence does not dance around in rough air.