According to this PPRuNe thread, the problem is not the retraction of the slats/flaps but instead turning the ADIRS (air data inertial reference units) off. The RTLU is designed to operate in a feedback loop, where its position is controlled based on input from the FACS (Flight Augmentation Computers, which take input from the ADIRS). If the ADIRS go offline while the slats are extended, the RTLU goes into a "emergency return to low speed" mode. This allows maximum rudder travel by driving the RTLU to mechanical stops. If this happens regularly it can cause damage to the stops.
This appears to be related to the Indonesia AirAsia Flight 8501 crash. Although this issue was not one of the main contrbuting factors, the report lists this as one of the Technical Followups (TFUs) that Airbus issued as a result of the investigation. It improves the reliability of the RLTU, the failure of which was cited by the investigation.
See this related question for more on the function of the RTLU in the flight control system: What are the “channels” on the A320 Rudder Travel Limit Unit?