In the context of aircraft autopilot design, I do not understand what is washout filter is [Heading Hold - Lateral Dynamics] and what is it used for. May anyone explain to me?
A washout filter is used in a yaw damper autopilot to remove the steady state component from the yaw rate sensor. Feedback from the rate sensor is used to damp dutch roll mode, but during turns coupling between yaw and roll results in the aforementioned undesirable steady state yaw rate component. Detail is on pg103 of: this document
A washout filter is a building block of a control system. As such, it's hard to say exactly where it'd be used, but I'll make some guesses about where it could go. In general it's used for removing the mostly constant part of a signal or approximating a derivative.
(If you already know the basics of control systems, skip this paragraph) Control systems in general are used to connect what you can sense (gryoscope forces, IRS readings, airspeed indicators) to what the state is (attitude, position, velocity, etc.) and connect that to the desired control surface response to smoothly reach the desired attitude and velocity. Since some of these relationships require filters like complementary filters or others are approximated by second-order differential equations, the filters used in these controls can be equally complex, incorporating elements like integrals and derivatives in their equations or even using a carefully matched second-order differential equation as the response. Imagine a designing a equation that told you what angle to hold the steering wheel based on the car's distance from the center of the lane. That's an example of the math that would be involved here.
A washout filter simply removes any slowly-changing component of the input and preserves the fast-changing component, sometimes called "high-pass" behavior because high frequencies pass through. This is similar to (and can even be an approximation for) taking a derivative of the input: sudden changes produce a large output, slowly changing or constant values produce almost no output. Washout filters have a time-constant or cutoff frequency that determine where it draws the line between "fast" changes and "slow" changes. Most washout filters are very simple (generally first order) and, unlike many filters in signal processing, don't go to great lengths to get a sharp transition between signals that are passed and rejected.
This washout behavior has several uses you could be referring to. Your desired system might respond to changes in a sensor value it but ignores long-term position of that sensor. Or perhaps you're building a PID controller and need a derivative term. The most likely case in my experience would be that you're building a complex control law, and at one point in building up that control law your math dictates that you need a washout filter. The math for deriving that control law would be too complicated to explain here, especially in a broad case like an whole mode's lateral control laws.
In my experience, most work with a washout filter in an existing design is to adjust the gains going into it to make it more sensitive or less sensitive (and therefore the system will be respectively less stable or more stable). You could also adjust the time-constant of the washout filter, but to do so you'd have to have equations based on real-life data or system response models to find the new values for the time constants.
I've seen washout filters on vertical control laws, but I've never seen one used alone for lateral modes. But I'm not some PhD student who's looked at over a hundred control laws, and it also could still be used for any of the applications listed above. Brief googling says it might be used to filter out the constant yaw attitude in a yaw dampener, which would allow you to damp out yawing motions without forcing the aircraft to have no sideslip. See this Aircraft Lateral Autopilots lecture slides from MIT for an example of that application.
An interesting reference can be found in this MATLAB guide. Here, it is described as a method to prevent over-damping of the spiral mode of an aircraft. It is implemented by a zero at the origin (a pure integrator) and a single pole to limit its effect to low frequencies.
A good yaw damper will stabilise the aircraft around the Dutch roll mode (by reducing the bank/roll coupling). However, an overzealous yaw damper will prevent normal banking behaviour from an aileron input. The desired behaviour from an aileron impulse input is that the aircraft flies with a certain bank angle. This will result in a sideslip (damping needs some yaw rate to work, so sideslip/yaw is never fully prevented). The yaw damper will then also damp any tendency of the aircraft to return to zero sideslip (remember, a pure damper only looks at the yaw rate, and has no reason to have proper tracking behaviour to zero yaw). The aircraft's natural stability (for example, due to the dihedral) will then right the aircraft again.
A washout filter will integrate the yaw, and thus force the sideslip back to zero. This way, a coordinated turn is entered (assuming proper attitude control).