For a turboshaft engine, does increasing its shp also increase its payload capacity? What are the other factors determining the payload capacity of a helicopter?
This is a complicated question, here is a simplified answer.
Generating lift requires setting in motion a large quantity of air, as downwash. More lift requires more downwash, and that requires setting in motion a larger quantity of air- which requires more horsepower applied to the main rotor shaft.
Note however that the rotor blades are designed to generate their optimum lift at a certain design speed and angle of attack, which means that if you add horsepower you would also have to increase the rotor blade span and/or add more blades to absorb and make best use of the additional power- all while maintaining the blade tips subsonic, and not overstressing the blade attach points or the main transmission gears and shafting.
In addition, more power to the main rotor disc means more torque reaction, and you'd need to redesign the tail rotor to counteract that.
All this means that you cannot increase payload by simply increasing the shp of the engine.