Unfortunately, this is an opinion based question. My opinion is based on the fact that my first few hours of helicopter instruction were harder than my first few hours of fixed wing instruction.
Takeoff in a helicopter was fairly easy after hover taxiing to the takeoff pad and hover turning 360° with yaw control to visually clear the area. Landing in a helicopter was fairly easy up to the point of bringing the aircraft to a hover. In both cases, the hard part came in the form of any of the low speed maneuvering. That includes just getting the skids off the ground and putting them back down under control.
Takeoff in a fixed wing aircraft was easy except for directional control to remain on centerline before rotation and the extended centerline after rotation. Landings in a fixed wing aircraft are in another realm of difficulty until you get the hang of it. From stabilizing the approach to slowing down to a speed appropriate to turn off the active runway and everything in between are hair-raising and nerve-wracking your first few hours of instruction.
Cruise flight in both is fairly simple and enjoyable once the aircraft is trimmed out and the friction locks are set properly. Although, changing flight parameters in a fixed wing can be done with as little as input on one control. Say for instance, changing the throttle input in a trimmed out airplane will make you change altitude. Changing flight parameters in a helicopter will involve input into more than one if not all of the controls. Changing the collective will require a change in throttle input, which will require input into the anti-torque pedals, in order to change altitude in a helicopter without a governor at low speeds.
The biggest difference is that a good instructor can verbally walk you through flying an airplane up until short final on your first flight. Although, they will always be ready and prepared to take the controls at any time. In a helicopter, the instructor will be on the controls the entire time your first flight. They will give you command of each control, one by one, as your confidence and competence grows.
Bottom line is that hovering is the deciding factor on whether it is harder to fly either a fixed wing or a rotary wing aircraft. Although, it is easier in a fixed wing. In a fixed wing aircraft, all you need is plenty of altitude, and a headwind that matches your airspeed (or an aerobatic plane with lots of thrust). In a helicopter, you need to be able to juggle while riding a unicycle.
