This is not a question that can be answered with a general rule.
What must be examined, when taking off in a condition of lower atmospheric density, is the altitude limit for that weight.
The issue is: can the engine/prop generate enough thrust to fly?
The best hope is at the speed where prop efficiency is the highest.
is this Vmin power?
Well, no, because at full RPM the prop spins faster, so it needs a faster forward speed to keep relative wind at the best angle of attack.
what is that angle of attack?
Well, why not put a Clark Y on your wing and your propeller?
The magnificent Clark Y maintains optimum Lift/Drag efficiency from about 3 to 7 degrees, making it ideal for fixed pitch prop applications and wings for GA aircraft.
One can then evaluate the effects of lowering flaps even 10 degrees on lift and drag: it shortens gliding distance. In other words, it's L/D ratio is less.
At high altitudes, where thrust is limited, it is best to fly in the "sweet spot" where Lift/Drag ratios are optimum for prop and wing.
This is where climbing performance, so important at higher altitudes is best.
Flaps may get one off the ground faster, only to provide a nasty surprise, once trying to climb out of ground effect, if power is insufficient.