AGL is "Above Ground Level". Ground is the reference. Trees, buildings, towers, and such are obstacles. Pilots need to be aware that obstacles exist. Most of the smaller ones can be addressed by just flying at a safe altitude. This is covered (in the US) by the following Part 91 rule:
§91.119 Minimum safe altitudes: General.
Except when necessary for takeoff or landing, no person may operate an
aircraft below the following altitudes:
(a) Anywhere. An altitude allowing, if a power unit fails, an
emergency landing without undue hazard to persons or property on the
surface.
(b) Over congested areas. Over any congested area of a city, town, or
settlement, or over any open air assembly of persons, an altitude of
1,000 feet above the highest obstacle within a horizontal radius of
2,000 feet of the aircraft.
(c) Over other than congested areas. An altitude of 500 feet above the
surface, except over open water or sparsely populated areas. In those
cases, the aircraft may not be operated closer than 500 feet to any
person, vessel, vehicle, or structure.
(d) Helicopters, powered parachutes, and weight-shift-control
aircraft. If the operation is conducted without hazard to persons or
property on the surface—
(1) A helicopter may be operated at less than the minimums prescribed
in paragraph (b) or (c) of this section, provided each person
operating the helicopter complies with any routes or altitudes
specifically prescribed for helicopters by the FAA; and
(2) A powered parachute or weight-shift-control aircraft may be
operated at less than the minimums prescribed in paragraph (c) of this
section.
Following (a) and (b) is the safest course. Para (c) provides a great deal of latitude to fly low but has significant risks. Agricultural pilots fly very low all the time. But ask any one of them and they'll be glad to discuss all the stuff they've hit.
These rules are primarily directed at VFR flight. Under IFR, ATC clearances are designed to ensure operating at safe altitudes, clear of both terrain and obstacles.
For larger obstacles such as towers and bridges, they are charted. As you can see on the clip below:

There are a number of towers in a relatively small area. In this case there is the underscored BITHLO TOWERS label to bring attention to them. Adjacent to the tower icons are two numbers. The number not in parentheses is the height in MSL. In parentheses is the height AGL.