Short answer: The constraints are that you maintain sight, and procedures are to train and practice regularly, and thoroughly brief all contingencies.
Visibility can vary significantly in different cloud types, and even within a layer. Cumulus are generally the most dense and therefore have the poorest visibility. They are generally the most turbulent as well, which makes compensating by flying closer more of a challenge.
That said, there is no real lower limit or minimum visibility for scheduling operations as it would be impossible for a ground station to measure or forecast.
The reality is that the lower limit is established by the comfort level of the wing pilot. You tuck it in as close as you are able to when the clouds get thick, and for several minutes your entire world may shrink down to the immediate and life-affirming task of holding a small airfoil shaped wingtip or nav light in a constant relative position on your canopy. You might be doing this while getting bounced around and having lost your other positional reference points because the clouds are so thick you can no longer even see the fuselage of the lead aircraft.
This is where a having an NFO, (Naval Flight Officer) with a soothing voice in the right seat is worth its weight in gold. While you fight to keep from either losing sight or hitting the other plane, your situational awareness is enhanced by little sugar calls on the intercom like "in an easy right turn to base leg now, we should be rolling out in about 10 more degrees... passing 4000 for 3000 at about 800 FPM down..."
Sometimes that wingtip will even flash in and out of sight briefly as you transit the thickest sections, anything more than about a second or so should trigger execution of lost sight procedures, which takes me to...
Lost sight under IFR is an emergency because you suddenly have an aircraft, (the wingman) operating in IMC, without an IFR clearance, WELL inside of ATC separation requirements. So, you can’t count on ATC for the immediate separation needs, you need to take action.
Lost Sight procedures are briefed on every formation flight, and they need to be deeply ingrained and automatic so when that small piece of metal 10-15 feet away, (that's attached to a much larger piece of metal 30-40 feet way) suddenly disappears from view there isn't any confusion about what to do.
The important thing is for the lead to remain predictable and smooth, continue to comply with the IFR clearance, and for the wingman to take action.
- Climb or descent - Lead continues to assigned altitude, wingman adds power (if in a descent) or pulls power (if in a climb) to level off.
- Turn away - Lead continues the turn, wingman levels off to achieve a heading split.
- Turn into - Especially important for the wingman to call "lost sight" on the radio, and over bank to get away. Lead should level wings until a 30 degree heading split is established, then resume turning to assigned heading if being vectored.
There are additional techniques that may be covered in more detail depending on the circumstances, and/or experience levels of the crews or number of times they have flown together.
For example, if I was the formation lead pilot and was briefing with a new pilot/NFO crew I hadn't flown with before I might add some mission specific details like "there's a pretty thick layer on the way to the MOA, if you lose sight in the climb call it out on the inflight freq and I will keep climbing, calling every thousand feet as I pass. Get on the radio with ATC and pick up your own squawk, and ask to resume climbing once we have at least a thousand foot split. I will let you know as soon as I find VMC, and audible a rendezvous plan on the inflight freq."
Those who fly together regularly start to think alike, and you would likely spend 3-5 seconds reading the bulleted points above straight off the mandatory items on the briefing card, and then move on with the rest of the brief.