I am studying for my PPL, and after going through aerofoils, lift, drag, optimal L/D ratio, propeller blades and the like, the need for a variable pitch propeller is absolutely clear to me.
It is also clear to me how a variable pitch propeller works, both from an operational and a mechanical point of view.
What I really don't understand is why. What is the basic reasoning behind its implementation?
Why does a propeller have to be decoupled from the crankshaft in the first place? How did we switch from talking about a blade's angle, to caring about its RPMs? If the problem was the blade's efficiency because of its AoA, why are we even talking about RPMs?
Why can't I just have my big simple knob by which I could directly control the blade pitch angle?
Controlling a blade's pitch by its RPMs and manifold pressure, to my very uneducated mind, sounds indeed like a very counterintuitive way of solving a problem.
After stumbling on the concept of two-pitch propellers, I managed to find this excellent page which clearly describes the evolution of propeller technology, from the "Ground Adjustable Pitch", to the "Two-Position", to the "Controllable Pitch" (which is basically what I described with my knob example), up to the "Constant Speed" which is an evolution of the latter, for reasons already exhaustively described in some of the answers.