How do they cancel the rotor torque (often canceled by the tail rotor)?
This is a basic physics question, involving Newton's third law of motion (For every action, there is an equal and opposite reaction.) When a centrally mounted engine applies force to turn the rotor, the equal and opposite reaction creates torque on the fuselage. With a tip jet, the force is applied by the jet shooting its exhaust perpendicular to the blade, so the reaction is that the blade moves in the opposite direction.
So the answer is that tip jets don't CANCEL the torque, they simply don't create the torque in the first place.
Think of the aircraft engine as one isolated system and the rotor as another isolated system. In its simplest terms, torque is the force required to move mass in a circular motion. Torque is caused by the engine providing power to the rotor shaft which moves the mass of the blades. The torque is the interaction of the stationary engine trying to move the stationary rotor blades in a circular motion. If you disconnect the engine from the rotor (for instance via a clutch), you eliminate the torque. The rotor is allowed to spin freely.
Tip jets remove the torque being supplied by the engine via the shaft because the engine is no longer providing power to the shaft. The reactive force of the jets is moving the mass of the isolated system of the rotor. The rotors mass is being moved without the need to receive power input from another outside system moving it in a circular motion.
Think of 2 scenarios: in one you are holding a rocket shaped projectile that is inert and you throw this projectile using your arm. In the second scenario you hold a real rocket, which is then launched.
In the first scenario you throw the projectile, and you have to counter the twisting motion of your arm because your arm is imparting a force. In the second scenario you don't have to take any action because the rocket does the work of moving the projectile instead of your arm.
A regular helicopter is like the first scenario, the engine pushes on the rotor, so the rotors push back. A tip jet is like the second scenario as the tips push themselves.
Ejection of the tipjet exhaust streams in different directions creates a reactive torque on the rotor. This causes the rotor to spin up until aerodynamic drag exactly opposes the jet reaction. So in a way you can say that tipjet helicopters cancel the rotor torque through aerodynamic drag.
But no torque is applied to the rotor through the rotor hub. Its bearings may create a tiny amount through friction, but that is negligible.