With more flaps, you will lift off at lower speed, but will have worse climb performance due to the increased drag.
On short field you are limited by the field length, so you need to lift off early and therefore need to use more flaps.
But in the mountains you are limited by the climb performance. Therefore you need to use less flaps.
Of course, this is just general rule. You should look up performance for given density altitude and compare it with available runway length and required climb gradient.
Update: Extending flaps never improves rate of climb. Up to some points it does improve angle of climb though, because it reduces Vx and the engine can produce more thrust at lower speed. The setting for best angle of climb is higher than for normal take-off and is used on obstructed field.
However as excess power reduces (with density altitude), Vx increases towards Vy (at absolute ceiling, Vx = Vy and rate of climb is 0) and the flap setting for best angle of climb decreases. So even for best angle of climb you shouldn't use as much flaps at high density altitude as at low.
And most mountain airports are not obstructed. In most cases the valley ahead is large enough, but you want to get above the tricky winds fast and you want high rate of climb in case you encounter a downdraft. And that means no or little flaps.