The short answer is no, age or airframe hours alone are not something to be concerned about on these light Cessnas.
A 1975 C172M I flew for work had about 20,000 hours, or more, on the airframe (can't remember exactly), largely in low level flight at or below 500 ft AGL, often in high winds, moderate turbulence, low level wind shear, etc., and flight including plenty of relatively high G maneuvering, and lots of crosswind landings far exceeding the demonstrated crosswind component. Years ago, before my employer bought it, someone managed to flip it over on it's back while taxiing. The thing has also had several bird strikes.
As an A&P mechanic, I have thoroughly inspected this airframe, to the level of an annual/100 hr inspection. Keeping in mind the airframe's age and history, my inspections included paying special attention to items of structural interest such as spar and strut attach points, control surface attach points, firewall, spars, etc. I have no concerns about the structure of this particular airframe, and I've trusted my life to it over 840 flight hours.
I have quite a bit of experience maintaining a number of single engine Cessna models, including quite a number of older, well worn individuals. I cannot recall encountering any structural issues that were due to metal fatigue alone. Most of the structural instances I can think of would be due to one time events such as a hard landing causing a firewall to wrinkle or crack. Some, such as cracked gear strut castings—in a U206G for example—are relatively common on such airframes that are used for rough, backcountry flying. I suppose that could be chalked up to a type of metal fatigue, but I don't think it is what you are concerned about with this question. Such items are a known issue and won't cause you to fall out of the sky.
The real age-related concerns regarding structural issues will be corrosion. Obviously, corrosion can occur over a short time frame, but give an airframe an extended stay in a harsh, corrosion conducive environment and it may fare more poorly than in a "good" environment. That being said, there are plenty of great airframes from the '50-'70s that have been hangered or sitting in barns in humid climates that are in excellent shape. The same will probably not be true of the engines if they sat unused, but the airframes may be in great shape.
Every airframe is different, and a thorough pre-buy inspection will be necessary to ensure that yours does not suffer from some major structural issue. But age and hours alone should not be of real concern.
I would not hesitate to buy a high-time, vintage Cessna airframe merely on account of those factors.