My answer comes from the point of view of a cyclist (also being subject to the effects of headwinds and tailwinds).
I would suggest that any form of consistent winds on a round trip can only act to make the net effort required greater than in calm wind conditions.
Consider the effect of two special cases:
Consider a strong cross-wind, blowing at 90 degrees to the path being travelled (say, a westerly wind). In both directions, (north and south), you will face increased head winds. In an aircraft, you would need to aim somewhat west of your intended direction in order to stay on course. Both legs of the journey would be more difficult.
Consider a strong head-wind, blowing parallel to your course (head-wind in one direction and tail-wind on the way home). For the sake easy maths, let's assume that the wind-speed is half that of the cruising airspeed of your aircraft. Going into the wind, your ground speed will be only half that of still conditions and so the journey will take twice at long. A one-hour flight (in still conditions) would take two hours. To make up all that lost time on the way home you would need to be moving very fast. However the tail-wind would only make your ground speed 1.5 times your normal speed and you would return along the 1-hour path in only 40 minutes to give a round trip time of 2hrs 40mins.
In both cases, and therefore all variations of wind directions and for all wind strengths above 'calm', your total round-trip time will always be longer.
If you are only concerned with differences in airspeed (and not how long it will take to cover a certain distance), then a direct tail wind will help in one direction and a direct head wind will hurt in the other direct with commensurate changes to net ground speed. However, as soon as there is any cross-wind component, that will hurt in both directions, hence the 4kt discrepancy mentioned in the question.