In aircraft with mechanical controls, the position of the control column corresponds to position of the control surfaces and the force on the control column corresponds to the force on the control surfaces. At higher speeds, the same deflection generates larger force and correspondingly stronger response from the aircraft.
However in Airbus the forces are not carried and the side-stick is spring-loaded. In normal law the side-stick deflection commands specific wing loading (which mostly corresponds to pitch rate) in pitch and roll rate in roll. So you get the same rate or pitch/roll with the same deflection independent of speed.
But in direct law, the position of the side-stick corresponds to the position of the control surfaces. Because the same deflection causes larger aerodynamic forces, it leads to higher pitch/roll rate at higher speed. Significantly higher, because the control forces grow with square of speed.