What is the name of the maximum altitude at which a helicopter can fly?
I recall it as "vertical limit", but I can't find a Wikipedia page for it other than a movie of the same title, which seems to concern mountain climbing altitudes.
Like with fixed wing aircraft, it is called the ceiling. Usually two ceilings are distinguished;
service ceiling, where the aircraft can still achieve a positive rate of climb of either 100 fpm (propeller) or 500 fpm (jet).
absolute ceiling, the theoretical altitude at which no positive rate of climb can be achieved. In other words: the maximum altitude at which the aircraft theoretically can fly under maximum engine power.
Helicopters also have a stat you might have seen called In Ground Effect (IGE) and Out of Ground Effect (OGE) Hover altitudes. This is the maximum altitude (MSL) the helicopter can hover at when it is in or out of ground effect. Essentially, when IGE, a helicopter can hover at higher Mean Sea Level (MSL) altitudes because they get a little help from the proximity of the ground, which reduces the rotor tip vortices (reduces the drag on the rotor). This site has some good info on ground effect as it relates to helicopters.
Under 14 CFR Part 29.1527, the Maximum operating altitude for a transport-category helicopter is:
§ 29.1527 Maximum operating altitude. The maximum altitude up to which operation is allowed, as limited by flight, structural, powerplant, functional, or equipment characteristics, must be established.
So its not just a matter of power or thrust, or even just retreating blade stall, but rather the entire design envelope for the helicopter must be taken into consideration for maximum altitude.
For example, the Robinson R-66 has a number of operating limitations that pertain to factors other than maximum thrust:
And there is a Vne chart which limits forward speed to prevent retreating blade stall at high density altitude: