Restricted Areas (...) If the restricted area is active and has not been released to the controlling agency (FAA), the ATC facility will issue a clearance which will ensure the aircraft avoids the restricted airspace unless it is on an approved altitude reservation mission or has obtained its own permission to operate in the airspace and so informs the controlling facility. (AIM)
How close you can fly, per the AIM's wording above, is not a distance, rather up to the ATC's instructions, unless the mentioned waivers apply. Checking 14 CFR:
(...) Upon the request of the FAA, the using agency shall execute a letter establishing procedures for joint use of a restricted area by the using agency and the controlling agency (...) 14 CFR § 73.15
Which means the established procedures, like the regular one you encounter, are coordinated between the using agency and the FAA.
Regarding VFR flights, restricted areas are not wholly prohibited:
(...) while not wholly prohibited (...) Penetration of restricted areas without authorization from the using or controlling agency may be extremely hazardous to the aircraft and its occupants. (AIM)
- How close is managed by the ATC (see @randomhead 's answer)
- ATC's actions are already coordinated with the using agency
- While not wholly prohibited, restricted areas may be extremely hazardous.