Latest FCOM, after revision 24 of section 3.01.21, contain the explanation:
AIR CONDITIONING WITH LP GROUND UNIT
- Do not use conditioned air simultaneously from packs and LP ground unit (to avoid chattering of the non return valves).
The non-return valves prevent air in the mixer unit to return into any source in the case this source delivers air at a lower pressure, or doesn't deliver air. Valves are the one-way separation between the air generation section and the air distribution section:
Source
If the LP ground connection is used while the packs are ON, pressure will be comparable on both sides of the valves. Valves can start chattering instead of being stable in the open position. While pressures compete, air from the mixer unit can return to the packs.
Damage can occur, as described in this 1991 incident (air conditioning duct and non-return valve damaged).