Even if the pilot is instrument rated, what stops them from doing it anyway? I’d imagine in certain situations where danger is approaching, it may be best to switch to IFR, it can’t be that hard to rely on your instruments to get through a cloud? Would controllers know if you switched from VFR to IFR and this could get you in trouble or something?
You are confusing VFR, VMC, IFR and IMC. Let’s briefly define the terms before I attempt to answer your question:
VFR – Visual Flight Rules. The pilot navigates at will, but must comply with cloud clearance and airspace requirements.
VMC – Visual Meteorological Conditions. The weather is good enough to operate under VFR.
IFR – Instrument Flight Rules. The pilot must file a flight plan with ATC and be cleared for the altitude and route flown. This defined and regulated process is required under IMC, but can also be followed when the weather is clear.
IMC – Instrument Meteorological Conditions. Weather conditions are poor enough that operation under IFR is required.
So to answer your question, if an instrument rated pilot is operating under VFR and finds him or herself unable to maintain VFR cloud clearance requirements, (i.e. the weather is no longer VMC) then they should contact ATC immediately and request an IFR clearance. By definition it is not possible to fly under IFR without the knowledge and consent of ATC.
And yes, it is possible to fly into IMC without ATC's knowledge or a clearance, but it is neither safe nor legal. It is unsafe for the non-instrument rated pilot because of the extreme likelihood of spacial disorientation, and for both because of the risk of loss of separation from other traffic. An instrument rated pilot would of course be able to rely on their skills to penetrate a cloud safely, but it is not something that can be done at pilot's discretion as you suggest.
I am guessing that the origin of your question is what we call “inadvertent IMC”. In this case, instrument rated or not, the pilot should fly by reference to their instruments and execute a level 180 degree turn to exit the cloud. This is an emergency situation.
Without going into the definitions of IFR, VFR, IMC, VMC, which has been so well covered...
Pilots CAN and DO fly VFR into IMC all the time. The NTSB actually has statistics on this occurrence. They usually are alerted to it by the big crashy noise the plane makes.
The reason responsible pilots can and don’t fly VFR into IMC is that it is neither safe nor legal. Just like your car CAN probably go over 120 miles an hour (200kph), a responsible person doesn’t. Even if you can do it without crashing into someone or something, it is still not legal (in most places). You may get away with it because the police can not cover the entire road system with radar. But, it is still not safe.
Flying VFR, you have to stay a certain distance from clouds depending on where you are. You also have to have a certain amount of visibility. ATC can not see the clouds nor visibility on radar. They can only see precipitation. They do know what the ground based weather stations say the sky coverage and visibility is in that area. At a certain level, they will call that entire area IMC.
You could try flying in clouds in an area designated VMC. If you pop out of the clouds and nearly hit someone, that other pilot has to report the incident. Or, ATC will also file a report of a near miss recorded on their radar regardless of clouds and visibility. The two reports will eventually meet each other.
Even if you don’t have a near mid-air collision, other pilots can report you based on the fact that they see you popping out of the clouds. But, they don’t hear you in communication with ATC on the radio.
In any of the above mentioned incidents you could face legal repercussions. ATC radar coverage is way more complete than police radar coverage. Even if you are squawking VFR, or not squawking at all, they can track you down by reviewing the radar records of where and when you took off and landed.
The IFR/VFR rules are written in blood for a reason. Yesterday it was too foggy to fly. A pilot decided to taxi their plane back to the hangar. Since they were not on the movement area (taxiway nor runway), I doubt they got ATC Ground clearance. I could hear their engine from the apron/ramp about 1000 feet away. But, I could not see them. Now, imagine if I had been in my cockpit with my headphones on and my just as loud engine running. How about if we had both been moving. Imagine the catastrophe that would have happened at 10-20 mph. In the air, we would have slim chance of survival.
Visual Flight Rules (VFR) and Instrument Flight Rules (IFR) are, as the names imply, two sets of rules that govern how a pilot flies. Assuming you are in controlled airspace, ATC will know which rules you are using, and if you want/need to switch, you must coordinate with them because it affects the services they provide (or don't provide) to you and possibly other aircraft around you.
One of the key differences is that VFR aircraft are expected to be able to "see and avoid" other aircraft, terrain, obstacles, etc., and the rules require a certain minimum visibility, distance from clouds, etc. so that they can reliably do so. In most airspace, VFR aircraft aren't required to be talking to ATC, and even if they are, ATC will still assume they are following the rules and thus don't need help not hitting things.
What this means is that if a VFR aircraft flies into a cloud, that breaks the most fundamental assumption of VFR flight, which can put the aircraft in grave danger from collision with other aircraft, terrain, obstacles, etc. That's why it's illegal.
Spatial disorientation is just icing on the cake.