At what rpm does the tip of a 54 inches propeller start to incur mach losses? The max rpm of my engine is 3600.
A 54 inch propeller would start to incur efficiency loss at approximately 4300 rpm.
At 4300 rpm a 54 inch propeller would have a tip speed of 691 mph which, relative to the 767 mph speed of sound, is about the maximum which still allows propeller efficiency.
You can calculate the variables using this tool: WarpDriveProps.com Propeller Tip Speed Calculator
Assuming you have properly sized the prop for mission and power, peak performance is 0.8-0.9 mach as shown in comments and other answers. Aim low if your prop has fat or blunt tips. You would normally also consider advance ratio (helix) but in a slower aircraft this won't have much effect.
If you consider all losses (not just mach) you might ask what prop speed gives peak efficiency, and it is slower. I usually consider 0.65-0.75 mach a better target because efficiency gives me range. It is also quieter. Every prop is different but here are a couple of examples. They are RC props because data was available, and the curves give indications. I wouldn't believe any exact numbers.
.....at the first sign of motion? Seriously, the instant a propeller blade - like any object moving through a fluid - starts moving, it’s creating drag as it takes energy to displace the fluid in its path of motion. That applies to the entire surface of the prop, not just the tip.