When is the GPU disconnected?
Airports rules are applicable, for example airports may prevent the use of APU when the aircraft is at its stand (to reduce noise and pollutants release). The airline operating procedures also provide directives regarding when to start APU and engines.
The general principle is that GPU energy is cheaper than APU energy, hence unless an engine is running, GPU is used until pushback starts.
Disconnection
When the crew is ready for the pushback they make sure they have the APU running and the GPU isolated from the electrical bus.

B737 overhead electrical panel, source
The ground operator is in contact with the crew through the flight interphone system. The crew asks for the pushback procedure. The ground operator now has to disconnect conditioned air and ground electrical supply.
On the ground power receptacle panel there are usually lights to indicate whether the GPU is connected and in use.

B737 external electrical panel, source
The operator checks the white light is on, which indicates the GPU is not in use and using a switch on the plug electrically disconnects the ground supply. The "CONNCT" light goes off and the cable is physically disconnected. At this time the GPU may be shut down, if applicable.
Chocks are removed, brakes are released and pushback can start.
More on ground handling procedures:
Currents required for 90 kVA under 115 V are about 1 kA. If the connection is broken when such current flows, the components just disconnected can melt down, and liquid metal can be projected around (it's similar to arc welding).
GPU are fitted with some protections to prevent hot disconnection. Let's look at this aspect.
Cable layout
A GPU delivers three-phase 115 V / 400 Hz, the GPU cable/plug contains therefore 4 main conductors and pins: one neutral used as common ground (N or G) and three phases (A, B, C). When the system is balanced, there is no current in the neutral, electrons move in phases only due to the voltage 120° phase angle difference.
N, A, B, C wires are connected to corresponding pins of the plug (E/F pins are explained below):

Plug (source)
At 400 Hz the reactance of the GPU cable is 8 times the reactance at 50 Hz, and is able to alter the phase angles at the aircraft interface, leading to significant power losses and under-voltage. Inductive reactance is due to the existence of a strong magnetic field near the conductors. This field can be cancelled by distributing the phase conductors around the neutral.
As the phase conductors have a large diameter, doing so leaves some empty space between the conductors. This means conductors may move within the cable when the cable is used. One solution to minimize this space is to split each phase conductor into two separate smaller conductors connected to the same power connector pin:

Source
Other solutions exist like twisted individual cables.
In addition an adequate mechanism can be used to tweak the reactance and/or the voltage when the cable is moved and its curvature is altered and/or when the electrical demand varies. This can be done by injecting some capacitive reactance into the circuit and/or adjusting the transformation ratio.
E/F control pins
Two smaller pins (E and F) of the cable plug are used for status exchange between the aircraft electrical system and the GPU. F carries 28 VDC from the aircraft. E and F are usually shunted in the GPU plug.

Standard wiring for civil aviation (source)
This way, when the plug is disconnected, the aircraft electrical system can inhibit its ground power input and the GPU can shutdown its power generators.
E/F are connected to corresponding smaller control wires visible between the phase conductors (when E and F are shunted, a single wire is needed).
Additional features
Control wires can be used to connect optional systems to the GPU:
Red/green (ON/OFF) push-buttons for remote control of the GPU.
E/F pins fitted with a small switch to know when the cable connector is 90% inserted within the aircraft panel socket and the connection can be seen as established.
When the cable comes from cable from the loading bridge, additional push-buttons for reel control: UP/DOWN, IN/OUT.
Sensors to detect reactance variation, hot cable/plug or broken neutral.
Conversely, LEDs on the plug can be switched on or off by the GPU using the control wires.
More information: