On a windy day there is a typically a wind gradient, meaning that the wind speed increases with altitude. On normal into-the-wind landings, the wind gradient tends to exacerbate any accidental "ballooning" (meaning that if aircraft accidentally rises during the landing flare, it tends to keep rising, till it eventually ends up in a very low-airspeed condition.) Especially in gusty conditions. Flaps tend to aggravate this. When landing downwind, the wind gradient tends to reduce any accidental "ballooning" tendencies during the landing flare. On the other hand, downwind landings involve high groundspeeds which can tend to exacerbate steering problems, especially in tailwheel aircraft. Flaps reduce the airspeed, and therefore the groundspeed, at touchdown.
As a practical matter, when playing around with multiple upwind and downwind touch-and-goes back-to-back in windy conditions with a radio-controlled model airplane (most pilots would not be interested in doing such a test with a "real" plane!), in a situation where runway length was not a limiting factor and the criteria of interest were a smooth gentle contact with the ground followed by a highly controllable run on the wheels before liftoff, I've found that the best results were generally achieved by using flaps when landing downwind and not using flaps when landing upwind.