Pitch angle only corresponds to an angle or rate of climb at a particular excess thrust.
Consider the typical stall exercise: you pull the engine to idle, then apply more and more elevator to try to maintain altitude, thereby pitching up and simultaneously reducing airspeed. At one point, airspeed drops to the point where you lose lift, and you fall out of the sky, despite the fact that you are nose high.
Once you apply power and push the nose down a little, it again becomes easy to establish a climb to regain any altitude lost.
Heck, consider how you slow an airplane down in the air, by raising the nose without having excess thrust.
If rate of climb (expressed as either best angle or best rate) related only to pitch angle or angle of attack, then instead of slowing down, you'd start to climb and maintain that climb as long as the pitch angle is unchanged, no matter the power setting. That's clearly not happening, at least not for long, so there must be something else to determine whether the airplane actually climbs or descends. That factor, as already mentioned, is (excess) thrust.
Therefore, expressing Vx or Vy in terms of pitch angle would require taking into account the amount of excess thrust available. Accurately estimating that directly is non-trivial, at least without specific instrumentation, and would also likely require the pilot to memorize a whole table, rather than a single (or two) indicated airspeed(s). During a high-workload phase of flight, including take-off and initial climb-out, every little bit that adds to the mental workload of the pilot risks increasing the workload beyond the pilot's capabilities.
If instead you aim to maintain a given indicated airspeed, you will be at a point of the power curve where all the factors line up for that airspeed, regardless of the external conditions (pressure altitude, wind, etc.). If airspeed drops for any reason, you pitch down slightly to pick up more speed and maintain the climb; if airspeed rises, you pull the nose up instead to slow down (trading airspeed for climb angle). You're using the pitch angle to maintain an airspeed, and the only instrument you need to pay very close attention to is the airspeed indicator.