The FAA expressly forbids flying within 500' of people, structures, etc...:
§ 91.119 Minimum safe altitudes: General. Except when necessary for takeoff or landing, no person may operate an aircraft below the following altitudes:
.
.
.
(c) Over other than congested areas. An altitude of 500 feet above the surface, except over open water or sparsely populated areas. In those cases, the aircraft may not be operated closer than 500 feet to any person, vessel, vehicle, or structure.
It's hard to find much wiggle room in the plain meaning of this statement. However I feel like there would be an exception when either the structures in question are your own (or otherwise authorized by the owner) or the people on the ground are somehow participating in the flight (such as judges for a flour drop).
Nonetheless, no exception exists in 91.119(c). Is there an exception anywhere else, perhaps in a LOI or AC?