Because the planes might hit them and it's not great for your vision at night. Runway edge lights are not intended to illuminate the runway they are intended to outline it. Proper runway lighting very clearly outlines a runway at night
Generally speaking there is nothing saying that a planes wingspan needs to be narrower than the runway and at some smaller airports the wings may very well be wider than the runway. This means you would need to put any high mast lights far enough away to allow aircraft clearance. In turn they would need to be quite bright due to the distance.
Another key reason and perhaps more important is to preserve the pilots "night vision". Generally when flying at night its advised to not expose your eyes to any bright lights prior to flying and to ensure that no flash lights or other bright lights go off in the cockpit while flying. This can cause your pupils to constrict making it harder to see in a dark environment. Entering a very bright runway environment all of a sudden would be strenuous on your eyes, in the event of a go around you would then have night vision issues due to the exposure in the brightly lit runway environment. As per the FAA's suggestion:
Before and during night flight, the adaptation process of the eyes must be considered. First, adapt to the low level of light and then stay adapted. After the eyes are adapted to the darkness, avoid exposing them for more than one second to any bright white light as that causes temporary blindness. If exposed to a bright light source, such as search lights and landing lights, remember that each eye adapts to the dark independently. By closing or covering one eye when exposed to light, some night vision acuity is retained in the closed eye.
I am not a videography expert, but I highly doubt the output of a GoPro at night is representative of the human eye. So I cherry-picked the above photographs.
To complement @Dave's answer with regard to wings and poles and outlines, much like how many highways rely on cat's eyes and the vehicles' own headlights, so do the runways in a sense.
The bottom-right image above shows how strong the taxi (not even landing) lights are for example. And with the cockpit lighting brought down for night operations, the pilots' eyes adapt and see much more than a GoPro.
- How can the pilots spot any physical obstacles on the runway?
It's the airport's job to make sure the runways are clear of FOD. The pilots are not expected to spot rocks and small broken pieces of pavement 2 km down the runway before starting the takeoff roll, day or night, for example. If they spot such things at slow speed, they should report it to be removed.
What I mean is, while FOD is a major concern, flooding the runway in light at night won't give the pilots eagle eyes.
The chief reason runways aren’t illuminated using elevated lights is that it creates a collision hazard. A airplane could potentially clip a wingtip during takeoff or landing, especially in gusty crooswind conditions, and the wingspan of large airplanes like the 747, etc. are so great that their wingtips overhang the runway edges, even with the aircraft on the runway centerline. Very few runways in the US or around the world exceed 150ft in width. These could strike elevated edge lighting. Even standard runway lighting can pose a problem with twins, creating a risk of a propstrike with the flight fixtures should the pilot allow the aircraft to get too close to the edge of a runway or taxiway.
Another goal for airport lighting on taxiways and runways versus street lighting is to allow the pilot to identify the outlines of them, both on the ground and in the air in order to differentiate these areas from the surrounding clutter. Further illumination of the immediate area in front of the aircraft can be provided. Conversely streetlights are used primarily for pedestrians to easily identify features or things of interest at night and public safety. Overhead, down pouring street lighting is not needed where this is not readily happening, which is why very few highways and interstates use overhead lighting on them, preferring a cheaper alternative of cats-eye reflectors and reflective road paint which work with automobile headlights to identify the outline of roads for drivers.
People have brought up the effect of lighting on a pilot's night vision, but this is easily addressed by controlling both the intensity and the color light the light source emits. You will notice most commercial ramps, parking aprons, etc are pretty well lit with overhead lighting and this makes sense as it is primarily to aid ground personnel in locating things and working, not aircrews.
As to collision avoidance or runway incursions, yes it can happen and the risk increases at night, just as it does with driving a car. It’s also a risk during daylight operations in marginal weather. Aircrews bear that in mind and reduce taxi speeds, make use of current airport diagrams to make sure they know exactly there they are at all times, use moving map displays with correct layout diagrams that can locate their position using GPS, and employ crew resource managemnt should they get lost and become unsure of the correct taxi route, etc. Modern enhanced vision system, some even found on small GA aircraft like the Cirrus SR-2X and SF-50 series, employ night vision and infrared cameras, which can can create composite images of the ground and terrain in front of the aircraft and give this information to the pilot on cockpit displays or project it directly onto a head up display. This gives a pilot a means to see a collision hazard and stop, make a go-around on approach, etc.
Airports are well maintained, secure and sterile environments so a risk of something or someone being out there who shouldn’t be is minimal. Animals are another issue but can be controlled to a certain degree with a secured airport perimeter. When I was doing flight training I nearly hit a coyote in a C-172 during a dual night training flight at KRVS which darted across the runway during a touch and go. I’ve always worried about hitting a deer or something when landing at a remote country airport at night. And it can make for some interesting conversations, kinda like one I once heard again at KRVS.
Pilot: Riverside tower, three one Mike, I’m gonna have to delay my takeoff roll. There appears to be.......a.......a skunk out on the active.
Tower:.........well if I had a .22, I’d grease ‘em for ya but hold till the traffic clears.