I have heard of gyroscopes and AHRS being used for heading indication, but I haven't heard of a flux gate compass system until recently.
What are the principles of its operation and what types of aircraft or systems are currently using it?
Fluxgates are a way to determine a magnetic field, in this case for navigational purposes. The basic principal is based on the lines of flux that the magnetic field of the planet gives off, a nice overview can be found here:
The flux gate compass that drives slaved gyros uses the characteristic of current induction. The flux valve is a small, segmented ring, made of soft iron that readily accepts lines of magnetic flux. An electrical coil is wound around each of the three legs to accept the current induced in this ring by the Earth’s magnetic field. A coil wound around the iron spacer in the center of the frame has 400 Hz alternating current (AC) flowing through it. During the times when this current reaches its peak, twice during each cycle, there is so much magnetism produced by this coil that the frame cannot accept the lines of flux from the Earth’s field.
There is a good video here as well.
Older, pre-glass HSI's are often slaved directly to them, like this one. On newer systems the flux-gate is a component of the AHARS.
Lots of aircraft use them, they can be installed on everything from a small GA plane all the up though an A380. These days they are fairly small units.