I recently got introduced to the boundary layer theory of an airfoil, and I couldn't stop thinking of how it would be affected by rotation as well as the varying speeds of rotation.
You are right, the rotation does affect the boundary layer.
Normally, as a wing approaches its stall angle of attack, the boundary layer becomes thicker and the flow starts to separate near the trailing edge. On a rotating rotor or propeller blade, the slowed-down boundary layer will experience a centrifugal acceleration, so it does not come to a standstill eventually (as it does in case of separation), but merely starts to flow tipwards. Since the speed of the rotor blade increases as the flow moves towards the tip, the boundary layer experiences an additional Coriolis acceleration. Therefore, flow separation is delayed on rotors and propellers compared to the two-dimensional case (for example, when the rotor airfoil is tested in a windtunnel).
The helicopter rotor experiments of Dwyer & McCroskey (1971) also suggest favorable effects on the spanwise development of the boundary layer, which tend to delay the onset of flow separation to a higher blade section AoA and thus serve to increase the maximum thrust of the rotor system.
The stability is affected by other effects; here the peculiarities of a boundary layer on a rotating wing make little difference beyond the higher stall angle of attack.
Rotation of the blade has a whole host of effects on the boundary layer of the blade: because of the velocity distribution, almost all aerodynamic effects are found at different radii.
If we look at the velocity distribution at forward speed, we can immediately see that the forward moving blade has a high tip speed, and the rearward moving blade has a region of reverse flow. So we can observe:
- A variation in Reynolds number from very low to about 10$^6$ near the tip. Below Re = 10$^5$ the boundary layer effects are mainly laminar, above 10$^5$ mainly turbulent. So we can find separation bubbles at different chord lengths as a function of blade section radius, and of position of the blade in the rotation cycle.
A variation in Mach number from zero to critical. Shock waves may appear at the advancing blade, and disappear again later in the rotation circle. From Leishman:
If at any point during this process the shock wave becomes sufficiently strong, then the high adverse pressure gradient will cause the boundary layer to separate causing a loss of lift and an increase in drag known as shock induced stall.
The region of reverse flow is outside of centre - it still produces lift despite the reverse condition. The high tip mach may cause lift loss or rapid drag increase, resulting in roll or yaw moment.
Pitching moment of the blade also changes continuously over the rotation path. All in all the trim of the helicopter changes a lot with airspeed, and needs continuous adjustment, especially in the higher sped region. These effects are partly compensated for by blade flapping: when left free to flap like the wing of a bird, the forward blade has a tendency to rise, reducing the angle of attack, and the retreating blade falls, increasing AoA. So this flapping takes care of roll moments, but causes sine wave variations of the free stream hitting the blade. And there are many of these Unsteady Airflow effects:
Stability responses of the helicopter on these effects vary. The rotation of the blades causes a sort of averaging out of effects, and for instance angle of attack oscillations do not cause unsteady stall behaviour noticeable by the pilots: the rotor is connected to the fuselage by a hinge after all. Fuselage will follow rotor, but with a time delay that further evens out responses.
More info in Leishman, chapters 7 and 8.