In general aviation aircraft such as Diamond star DA-40, flaps to take-off are used until 500ft AGL and speed before retracting flaps is Vy (best ROC speed)-67kts and after retracting flaps is 80kts.
Why can't we use take-off flaps until cruise altitude such as 6000ft MSL and maintain 67kts and climb sooner?
The image below from this answer shows characteristics of airfoils with flaps.
As you rightfully concluded, lift ($C_{L_{max}}$) goes up with the deployment of flaps, but the drag also goes up and even quicker than the lift. Increasing lift is good, but if it comes at the cost of more drag, it will require more thrust (therefore fuel) to maintain this higher lift.
Thus, the value we should maximize is the ratio of lift and drag. The ratio of the two $\frac{L}{D}_{@C_{L_{max}}}$ is also shown below, and it shows that the basic airfoil performs better than those with flaps.
This is a good question, and I don't feel the other answers get at the essential part which is:
Is it optimal to climb with the flaps deployed?
As with any optimal question, the answer relies on what it is we wish to optimize. It's worth examining two goal states:
In general, getting off the ground and clearing any obstacles while remaining close to the airport is generally considered best practice, even if it means burning a little more gas. Depending on a number of design choices, the climb angle with flaps can be much improved, which means that obstacle clearance is better and in the event of a takeoff emergency the plane has a lot more runway in front of it, or it's not so far from the airport.
The advantages of taking off with flaps down:
So what are the disadvantages?
As @ROIMaison shows in this answer, for a Clark Y airfoil the L/D ratio with flaps deployed isn't even remotely close to the normal airfoil. At higher airspeeds this loss of efficiency is acutely felt.
Of course, the DA40 has a much more advanced airfoil and so the spread might be much closer together. Diamond's airfoils come from gliders, and gliders use flaps in low-speed flight in order to turn more quickly. As you might imagine, gliders are optimized for efficiency, so it's fair to reason that the flapped L/D ratio for DA40's airfoil is potentially much better than the venerable Clark Y's.
In case the link between drag and climb rate is not immediately obvious, the more drag there is the less surplus energy is available to increase the plane's potential energy, i.e. to climb.
The engine cowling is designed to provide appropriate cooling at relatively high airspeeds. There's a certain thermal inertia which protects the engine for a minute or so, but after that temperatures start reaching critical points. It's important to nose over and pick up airspeed in order to improve cooling.
For a fixed-pitch cruise prop, efficiency suffers quite significantly at lower airspeeds. Speeding up will gain some extra propeller and engine performance.
[*] Note that this doesn't apply to constant-speed props.
With the above in mind, we can see that the best practices of getting up off the ground quickly, with plenty of runway to spare, and with enhanced obstacle clearance is a Good Thing (TM). These goals are largely met by 500', so this is an opportunity to reevaluate our optimal process. Do we still want to climb, or do we want to go somewhere as well?
Unfortunately, I don't have any basic sense of whether it's generally optimal to continue to climb with the flaps deployed. It might even depend on the particular plane model whether absolute climb rate is better with flaps up or down. If the climb rate is worse with the flaps down, then the answer is clearly to get them up as soon as practical.
Supposing climb rate is better with flaps deployed, the situation doesn't really become clear. If the winds aloft are favorable, then getting to altitude quickly is valuable. But more valuable than proper engine cooling? Hmmm...
Because it would not be efficient. Flaps increase drag (and lift), so you would burn more fuel climbing to cruise altitude with flaps extended compared to if if you retract them.
Most people fly in order to get somewhere, so cruise climb is used as it covers more ground while getting to altitude.
Here is the DA40 airspeed table that would have been nice to include in your question. Depending on weight, cruise climb speed is 6-9kt higher than takeoff climb speed. If your objective was just to gain altitude, as for local observation, you could use takeoff flaps if you wanted. In some aircraft you have to keep an eye on engine temps if you fly slowly at high power.
A better illustration is in transport aircraft where the speed difference is much higher. Here is the BaE146 takeoff sequence. There are speed limitations for deployed flaps, so retraction is structurally required as speed increases. Staying under a 135kt limit for takeoff wastes a lot of time when 220kt cruise climb is available and 250kt is the target.
Flaps give you more lift, but also more drag. The drag means that your horizontal speed is lower.
The usual phases of flight where low speed is an advantage are:
When taking off from a runway. You don't want to run out of runway and you want to end wheel drag as soon as possible.
When trying to clear obstacles. You want as much time to gain altitude before you get to them.
When at low altitudes. You don't want to get too far away from the airport when your gliding range is low.
When near landing. You need to drop altitude but also keep your speed down.
When landing. You want to get as slow as possible before you touch the ground.
Flaps make the lift/drag trade-off worse for you. That's only sensible when the drag is a good thing or the extra lift is absolutely necessary.
Once the drag becomes a loss rather than a benefit and you have enough lift without flaps, the logic of flaps stops applying. You want the most efficient wing configuration with as little drag as possible for the lift generated. That's the configuration your wing was built for.
Flaps provides extra lift at lower airspeeds but for the cost of huge drag.
At some point the drag stops the plane acceleration and it cannot climb any faster.
Note that the airlines want as fast transport from Gate A(XYZ) to Gate B(HKL). Airports want as short runways as possible. Government wants as small noise-loaded areas as possible.
Short runway limits the take-off/landing speeds significantly so the wing setup must provide lift at low airspeeds even at the cost of drag.
The government demands as fast climbing as possible and from some altitudes the noise is spread so wide the ultimate climb rate is not mandatory and airlines are free to optimize for the most crucial parameter - cost.
Here the drag is the adversary so flaps are retracted. Plane can fly faster for smaller thrust needed and the thrust reserves can be used for nose-up climb as well.
