In addition to the other answers, let´s look at this L/D(=E) diagram of the enticing DG-1000 from DG Flugzeugbau (but fear not, 'tis true for all gliders) :
The best L/D ratio is equal for different wing loadings, but is occuring at different speeds - the higher the load, the higher speed. You can also see that the minimum/stall speed is also higher for higher loads.
The next diagram shows the polar curve:
You can see that the minimum sink rate occurs at lightest load. The heavier the load, the longer you will have to circle in the same thermal for a given height gain.
The loading is a tradeoff between higher average speed and less efficient climbing. In case of strong thermals and/or long glide intervals, the optimum moves toward more, in weak conditions towards less or no ballast. The good thing is that you can dump water rather quickly (also partially), so that in a competition you usually tend to fill up (and dump in case) rather than start light (the Quintus e.g can take up to 250 liters!)
Aft ballast in the vertical tailplane is sometimes used to balance a forward CG caused by water in the wings - depending on your ship, partial dumping can be problematic.
Of course there are many philosophies and tactical debates concerning the "water or no water" dispute, but once you´ve overtaken an identical, lighter ship with full wings and no height loss, you get to see how much fun ballast can be (until the next thermal, that is).