The control problems are caused by a shift of the aerodynamic forces. In subsonic flight, they are mostly created in the forward part of the wing (the lift increase with angle of attack works at the quarter chord point of a wing, regardless of sweep), while in subsonic flow they work equally over the full chord. The center of lift in pure supersonic flow is at mid chord, and in reality the aerodynamic center shifts slowly back when the aircraft accelerates through Mach 1 and more of the surface is exposed to supersonic flow.
Since the c.g. position is mostly fixed (Concorde pumped fuel between tanks to shift the c.g.), this shift must be compensated with elevator deflection. The elevator must move trailing edge up, which creates a break in the airfoil contour, which in turn creates a heavy shockwave at high subsonic and supersonic flight speed. This shock causes flow separation and can lead to elevator reversal. Only when the fixed part of the horizontal tail can be moved, too, or you have a full-flying tail (the whole stabilizer is moved, not only the rear part of it) to create the needed lift change without a contour break, the aircraft can be trimmed for both subsonic and supersonic speed.
The benefit of the X-1 was a stabilizer with variable incidence, which the P-51 lacked. Sweep would have helped, too, but was not really necessary.