It's apparently legal for pilots to fly over the top of clouds and fly VFR. However, I don't understand how it's possible to do so, especially since there is no visual reference to rely on to ensure that you are heading in the right direction. So, how exactly does this work, are there any limitations on this and is it possible to be done safely?
Perhaps the most important rule for a VFR pilot is "see and avoid" -- be able to see any immediate dangers (other aircraft, buildings, ground) and avoid them. When flying above the clouds, you can certainly see other aircraft and avoid them (as long as you're maintaining required cloud clearances), and the same for buildings. You cannot see the ground but on the presumption that your altitude is well above the ground and not significantly changing, this is not a factor.
As long as you're able to see the horizon, you're unlikely to become disoriented to the point of not actually knowing that the aircraft is or isn't flying straight and level. (Contrast that with flying in the clouds, where it's easy to become spatially disoriented and not be aware that you're starting to dive.) Limitations are basic: it's the PIC's responsibility to follow the regulations.
As to heading in the right direction and navigation in general, VFR flight does not mean one cannot rely on radio aids to navigation. VOR and GPS certainly make navigation very possible while on top, but to get to your question of "can it be done safely?" -- I suppose this comes down to a question of how reliable are your electronics and where are the weak spots; is the pilot under flight following or otherwise in contact with ATC, etc. If one were on top and were to lose the electrical system, there's certainly an emergency to be declared. If already in contact with ATC, it's an emergency that can likely be managed with the utmost of safety.
I've flown VFR-on-top plenty of times. Navigation is simple: even the most basic airplane is required to have at the very least a compass, a clock and an airspeed indicator, which can be used to navigate much more accurately than you'd expect, given a reasonably good "winds aloft" weather report. Most airplanes also have some kind of radio navigation equipment as well. Furthermore, just because you are flying on top of a cloud layer, that doesn't necessarily mean you have no visual references for navigation. For example, mountains may poke through the cloud ceilings, or there may be an area of fog covering a portion of your route, with clear skies at your departure point and your destination. On one flight, I flew "VFR-on-top", but could see through the clouds to the streetlights below (it was dusk) even though my home airport was reporting IFR conditions (1/2 mile visibility, ceilings at ground level -- but it was a very thin layer of clouds).
Whether or not it can be done safely depends upon the circumstances. Getting lost typically isn't the issue, but rather whether you'll be able to descend in VFR conditions to land at your destination. Check the weather at your destination, paying attention to temperature and dewpoint spreads, and make sure you've got other options within the range of your aircraft. Also keep in mind that while you're on top, if you have an engine failure, you'll be making an engine-out IFR descent into unknown terrain below, unless you are within gliding distance of a suitable landing spot in clear weather nearby!