In case of F-35, the yaw control during hover (and transition to forward flight) is provided by the Three-bearing swivel module (3BSM), which forma a part of the Rolls Royce Lift System. As Lokheed Martin notes:
Two roll ducts supplied by engine fan air provide roll control. Yaw control is through swivel nozzle yaw. Pitch control is effected via Lift Fan/engine thrust split.
The three-bearing swivel nozzle, 3BSN, design uses three segments of the engine exhaust duct cut on an angle and joined by two airtight circular bearings. ... The forward and aft segments always stay aligned with one another in the rotational axis. The center segment rotates through 180 degrees relative to them. The third bearing is on the back of the engine aft of the turbine stage and provides the ability to swivel the nozzle thrust axis in yaw at any pitch angle.
Consider the nozzle in VTOL position, like below:
By Tosaka - Made by uploader (ref:見森昭編 『タービン・エンジン』 社団法人日本航空技術協会、2008年3月1日第1版第1刷発行、ISBN 9784902151329, 1), CC BY 3.0, Link
If we hold the aft two segments and rotate only the first one, it is obvious the rear segments will rotate along the engine axis, producing yaw by virtue of its position at the rear. There is a slight associated loss in lift produced, but the Digital Flight Control System can easily compensate for it.
Of course, this method of controlling yaw in hover is specific to F-35. In case of Harrier, the reaction control system had 'puffers' at the tail to provide yaw control.
Reaction control system, showing pipes and nozzles for engine HP bleed air.; image from harrier.org.uk
In addition to the vectoring engine nozzles, the Harrier also requires a method of controlling its attitude during jet-borne flight, when the normal aerodynamic surfaces are ineffective. To this end, a system of reaction control nozzles in the nose (blowing down), wingtips (blowing up and down) and tail (down and lateral blowing) are fitted to the aircraft.