I am working on an enclosure that is to be ejected from a rocket at apogee (10,000') and fall to a target altitude at a descent rate below 20ft/s before releasing its payload. The idea is to have coaxial rotors with collective pitch control deploy and autorotate to slow the descent of the enclosure and payload.
My question is how I can start autorotation with blades that have essential zero angular momentum, and are there special considerations for the use of coaxial rotors?
Go back to Estes rocket days. There is a whole class of competition rockets known as "heliroc" or helicopter recovery. These ascend like a conventional rocket, in a vertical(ish) path and fin stabilized; at (or near) apogee, they release a rotor by some means and the rotor spontaneously begins to autorotate, often giving a lower descent rate than a parachute.
Go even further back: look at maple seeds (aka maple keys). They autorotate so efficiently that they'll enter stable rotation and constant (low) sink rate in as little as one blade length of fall (they do have very low wing loading, however).
So, yes, it's possible for a rotor disk to spontaneously begin autorotation from a dead stopped starting condition. Mother Nature does it billions of times every autumn, and school children do it thousands of times every year.
Yes, this will work with a coaxial rotor, if you can keep the rotors from interfering with one another during deployment (for instance, by braking them until fully deployed). However, since there's no reaction torque from freewheeling, autorotating rotors, there's no reason to have coaxial rotors. It's far simpler to use an ordinary rudder for yaw control than differential collective, and you'll have half as many parts for pitch and roll.
...how I can start autorotation with blades that have essential zero angular momentum...
By using the collective pitch control mentioned in OP. At release, the blades should be aligned with the free stream like a feathering propeller, then gradually be set with decreasing blade AoA as the rotor speeds up.
...are there special considerations for the use of coaxial rotors?
Yes, coaxial rotors complicate all design efforts tremendously. Both rotors would have to have blade pitch control, one blade shaft must be mounted within the other which counter-rotates, etc. Much simpler to use only one, larger rotor, since yaw compensation is not required during autorotation.
Note that at vertical autorotation, a rotor has a slightly lower drag coefficient than a flat plate of equal area, which has a lower drag coefficient than a parachute. As mentioned in this answer.
