Others can probably give you more precise info since I'm not a P-51 pilot, but I am studying aircraft dynamics in school right now, so maybe this will help.
Many fighter aircraft are designed such that they are very agile. This is especially true with roll dynamics, where modern fighters are even designed to be inherently unstable. They can then roll faster since the dynamics promote the roll. For roll stability, the wing design is often the main concern. Now, to spins.
I believe spins are mainly a yaw-plane event (though I'm sure there is some coupling of the equations). this means its a rapid rotation about the z-axis, the axis normally considered down. For stability, or desirable spin recoverability characteristics, the tail design is key. I don't know all the details or considerations behind the P-51's tail design, but any number of things could cause slow spin recovery, meaning you need to fall for a while (sorry, pilot!)
For example consider the distance between the tail's centroid and the overall plane's center of gravity. Imagine we added a lot more fuselage, basically extending the tail farther out. Now, with our extended tail, we have a different system. a really long tail might be too resistant to any yaw - basically it wants to got a certain direction, like an arrow.
Now, shorten the body instead, making like a short little squished P-51 with the tail right behind the pilot(shorten the arrow). while flying, the tail now might not help the plane fly very straight, at least not easily.
Finally, put the tail in front of the pilot and try to find anyone brave enough to fly it! No one will want to, because they won't make it home. its like shooting an arrow backwards, or throwing a paper airplane backwards- the dynamics of the vehicle just aren't favorable - any sort of deviation and the instability induces more deviation.
Thats only one part of tail design. You also need to consider size, number of tails, control surface size, and this doesn't even include the yaw dynamics that come from the rest of the plane in a spin.
I apologize if you already understood all that, but maybe it would help with my key idea. That is, that the reason it takes so long to recover from a spin is due mainly to its tail design. I don't know specifically how they designed it, or what design goals they were looking for, but because of what they chose, maybe considering manufacturing ease, structural considerations, aesthetics, drag, turn performance, weight, heck, maybe they just sketched it up and tried it, they ended up with a plane with this yaw plane recovery performance. It might be bad, but maybe their other design goals were met and they chose to live with it.
Other aircraft with better spin recovery performance will likely have different tail features like larger control surfaces, different tail volume... Overall, their tail design will give it better yaw stability and authority (control power available) to recover sooner.
Its late, and I'm sure I've missed things. I hope others can chime in and fix anything I may have mistaken. Let me know what I can add to improve my answer