APUs are like small jet engines, small enough to be started electrically. So you would use the power of the battery or an electric ground power cart to spin up the APU by its electric starter motor, and then introduce fuel into it once its spinning fast enough, just like a jet engine is started. Depending on the design, the electric motor will then either act as a generator to supply electrical power back to the aircraft or decouple when a separate generator is used.
The APU will spin up, and besides spinning the electric generator, some pressurized air will be 'stolen' from its compressor to supply "bleed air" for engine starting, the air conditioning packs, etc...
Big APUs like on the Boeing 777 have actually two methods of starting: They have both an air-driven and an electric driven starter motor. The method of operation is as follows: When the APU is started with the engines off, electrical power from battery or ground connection is used. After the flight, the APU will be restarted before the main engines are turned off. In that case, the APU will be started by the air-driven motor rather than the electric motor, with pressurized air supplied by the still-running main engines.
EDIT to answer question in comment:
Depending on size, the APU will take something between 30seconds and 2 minutes to start. The big APU of the 777 will deliver electrical power after 2 minutes or so, small units you are likely to find on regional jets can do that after perhaps 45 seconds. Then, it takes an additional 2-4 minutes until you can extract bleed air from it. That is because taking air from the compressor increases the exhaust gas temperature of the APU, and you want the APU to warm up evenly. The same is true for the shutdown process: You close the bleed-air valve and then let the APU continue to run for another two minutes or so to cool down, then shut it off completely. This lowers the thermal stress on the unit. On modern planes like the 777 this process is automated: You turn the APU selector switch to OFF and that will close the bleed air valve, initiate a 2 minute cooldown period, and eventually turn off the APU completely. The only way to shut it down immediately is in fact the APU FIRE handle.
So, you will turn the APU on typically while you are taxiing in after landing. Because of course you want it to be up and running and ready to take over the air conditioning by the time you are at the gate.
Some APUs like those found on the MD-80s have two settings for bleed air extraction: NORM and ECON. ECON takes less bleed air, thus not increasing the exhaust gas temperature of the APU too much. On particularly hot days, using the APU bleed in ECON might help not overheating the APU if it is to be used for an extended period of time.