When am I allowed to change my transponder code to 1200?
Is it after clearing the boundary of regulatory airspace (ie. airspace requiring radio contact and an operating transponder)?
ATC will tell you something like, "N1234R radar service terminated. Squawk VFR. Frequency change approved; good day" which you will repeat back as an instruction.
You do not change from the assigned transponder code until instructed.
If you're in a hurry to get away from ATC, you may request to terminate radar service as soon as you are out of their regulatory airspace.
Normally, as a courtesy, you terminate radar service by notifying ATC in a handshake as in Mortensen's answer. In most cases this never happens, because as soon as you leave their zone ATC will terminate service unilaterally--you don't have to ask for it.
If you are flying under IFR rules in any airspace except G, you must have a clearance and be using the code currently assigned to you under the clearance.
If you are flying VFR, you normally use code 1200. You can change to 1200 whenever you want, but if you previously had a clearance, you will immediately lose your clearance when you switch to 1200.
Therefore, the general principle is that you can switch to 1200 whenever you are flying VFR and do not require a clearance to be where you are. In class B, C, and D you are required to be in "voice contact" with ATC, which could be construed to require a handoff. In class E you are only required to be in voice contact if you are in the vicinity of a towered airport, in which case you have to be in voice contact with the tower (not ATC). So, basically what this means is that if you are in G or E (not near an airport) you can go to 1200 whenever you want. In B, C, D you should always do signoffs ("Cessna 5678 Golf terminating radar service"). If you are in an area that requires a clearance, you must get permission to terminate radar service.
So, your own statement is basically correct: once you leave a radio-contact required area, you are free to go VFR. Note that if you abandon a flight plan and go VFR without notifying ATC, even if you are in a radio-free zone, they can get miffed about that because it messes up their rack, so it is always a good practice to explicitly terminate your service/flight plan before switching to VFR and going numberless.
Example: You are flying VFR through a restricted military operating area and have a clearance to do so. You must stay on code as long as you are in the restricted area, because you require a clearance to be there.