My answer is (as in most cases): It depends. Mostly on the type of flap used and the position defined as zero.
Camber flaps do not necessarily increase drag, at least at moderate deflection angles. Maybe it helps to think of an unflapped wing as of one where the flaps have been permanently fixed in one setting. And this setting does not necessarily need to be the one which offers minimum drag. Changing the setting of chamber flaps shifts the optimum angle of attack range up or down without changing drag much - until you reach extreme flap settings where the contour break is too big to be left unnoticed by the airflow.
The plot (made with XFOIL 5.4; own work) shows how the laminar bucket is shifted up and down the c$_L$ range without affecting drag much except for the -20° setting - here, the suction peak on the lower side in combination with the strong flap camber is too much for the boundary layer to remain attached. But this setting is really useable for inverted flight only.
Setting flaps to a slightly positive angle does indeed increase performance:
- The maximum lift coefficient is higher, allowing lower lift-off speed and a shorter ground run.
- The maximum climb angle is increased greatly especially for propeller aircraft as the aircraft can climb at a lower speed where thrust is higher for the same engine performance. Note that the polar point for maximum climb angle sits at a high lift coefficient, especially for a wing with high aspect ratio. Therefore, positive flaps might be required to operate the aircraft at its best climb angle speed safely.
- Maximum sustained turn rate is also helped greatly with moderate flaps. The plot below shows calculated turn radii for a large unmanned reconnaissance aircraft at several rates of climb (or descent). The line for 0 m/s is the stationary turn rate.
Turn rate diagram for two flap settings: 0° (blue lines) and +10° (red lines). Own work.
Several fighter aircraft used flaps effectively to maximize their turn rate in dogfights (google for "combat flaps"). Also, the living wing concept of the F-18 adjusts flaps to maximize wing performance in all situations. This would not help if flaps only reduce take-off runs.
These flaps increase wing area, so here any actuation will also increase the friction drag coefficient, since that is referred to the wing area with retracted flaps. However, since the increased wing area allows to fly more slowly, the total drag might be lower, depending where on the polar the aircraft flew before extending flaps. A moderate take-off setting will, therefore, also result in reduced drag if the aircraft is able to fly more slowly and induced drag is less than half of total drag.
At full deflection, fowler flap drag grows out of proportion to lift, but that is one of the design goals. The higher settings are meant for landing only.