Moving the flight controls above a certain threshold will disconnect the autopilot.
From the A320 FCOM (1.22.30 Auto Flight - Flight Guidance, emphasis mine):
AP DISENGAGEMENT
AP1 or 2 disengages when:
- The pilot presses the takeover pushbutton on the sidestick.
- The pilot presses the corresponding AP pushbutton on the FCU.
- The pilot pushes on the sidestick harder than a certain threshold or moves on the rudder pedals above a threshold.
- The pilot moves the pitch trim wheel beyond a certain threshold.
- The other AP is engaged, except when localizer/glideslope modes are armed or engaged, or when the rollout or go-around mode is engaged.
- Both thrust levers are set above the MCT detent and the aircraft is on the ground.
- The aircraft reaches the MDA-50 feet (MDH-50 feet), or 400 feet AGL if not MDA/MDH, with APPR mode engaged and a non-ILS approach selected.
- One of the engagement conditions is lost.
Furthermore, in normal law with all protections available, the AP will disconnect if:
- High speed protection is active;
- Angle-of-attack protection is active (α prot +1° is reached);
- Pitch attitude exceeds 25° up, or 13° down, or bank angle exceeds 45°;
- A rudder pedal deflection is more than 10° out of trim.
The different control laws act in the F/CTL COMPUTER in the following diagram:
(source: FCOM 1.27.10 Flight Controls - Description)
They are independent of the autopilot and the protections are available regardless of whether or not the autopilot is engaged.