I noticed that many ultra-lights use fabric skin but are not seen much in GA aircraft, especially compared to carbon fiber? Many skin makers advertise that it can handle that sort of weight. Is this due to difficulty manufacturing, weight, smoothness of surface, strength of material or other factors?
This answer assumes that "GA" means "little piston planes."
Aluminum (or non-fabric) aircraft tend to "hold up" better against the elements. I would not store a Stinson 108 outdoors but would not think twice about storing a 172 (or Cirrus) on an open air tiedown. The fabric covering requires a great deal of care compared to the metal (or composite) skin of most GA planes.
Fabric planes must be re-covered every few decades, more frequently if neglected. This is a non-trivial operation with a price tag to match. This typically involves reconditioning the now-exposed structure which adds significant cost. Re-covering and painting a Cub (small plane), for example, can cost an owner tens of thousands of dollars. Stripping, treating, and painting a "metalized" Cessna 120 (similar to a Cub) is a much easier, faster, and cheaper task.
Another consideration is the availability of aluminum. Aluminum was declared a strategic material in WW2 and, as a result, a great deal of energy was allocated to increase production. With the war over and the infrastructure in place to produce a lot of aluminum cheaply, why adhere to an antiquated manufacturing process requiring a painstaking level of detail when lower skilled labor can use cheap aluminum to mass produce high quality planes?
In short, fabric planes typically require more care and attention than their stressed skin cousins. They also require somewhat specialized and skilled labor to build and maintain. This generally translates into higher costs for everyone involved.
I'm sure there are technical reasons for the shift away from fabric but I'll let Peter (or another technical expert) comment on those.
This should be understood as a supplement to @acpilot's excellent answer. All his points are valid and correct, and I just want to add a few more.
Fabric covering is easy to build and hard to maintain. In the early days, when engines were heavy and less powerful, a wooden airframe with fabric covering was the fastest, lightest and easiest way to get airborne with the skill set of a carpenter. Later, welded steel tubing became popular especially for fuselages because it makes it easy to introduce big loads locally. Again, fabric covering was the easiest to give the truss an aerodynamic shape. Maintenance was hardly considered, given that the lifetime of early aircraft was a few years at best and a more likely few minutes only.
With better engines, wing loading and speed increased. Now the weight disadvantage of a plywood or even metal skin diminished, and with the increased experience of both designers and operators, aircraft enjoyed a much longer life. Only now the disadvantage of fabric covering showed up, but it was still kept even in WW II airplanes (the fuselages of the Hawker Hurricane and the Vickers Wellington were fabric-covered), especially in the control surfaces so their balancing mass could be kept low. With the danger of flutter at high speed, fabric covering became unacceptable for the main structural members of fast aircraft: Only the torsional stiffness imparted by a plywood or metal skin made fast flight possible.
Low wing loading designs like gliders and motor gliders kept their fabric covering even longer, and some modern ultralight / microlight designs still use it today. Glider designers would probably still use it were it not for the much better surface quality possible with composites. Being fair-weather aircraft, gliders are less affected by the obvious disadvantages of fabric covering.