Did any aircraft ever use stick twist for rudder control like some joysticks for video game flight simulators?
Yes, the RAH-66 Comanche FBW system used a twist control for the yaw channel.
The RAH-66 Comanche used a twist in the pilot's stick to control the rudder/yaw inputs. It had no rudder pedals. (Yes, a helicopter is an aircraft).
Comanche unfortunately was canceled near the end of LRIP (Low Rate Initial Production) in 2004 as the program was moving into EMD (Engineering and Manufacturing Development). There were at least 2 prototypes flying at the time. Test and development flights took place from 1996 to 2004.
"... the RAH-66 Comanche design incorporated a 3-axis, limited displacement, uniquetrim sidearm controller [replacing cyclic and yaw pedals] for control of the longitudinal, lateral, and yaw axes. A proportional collective with approximately 6 inches of displacement was used. An enhancement tailored to the scout mission was the incorporation of limited control in the vertical (fourth) axis of the sidearm controller. This allowed the pilot to command stabilized climbs and descents with the altitude hold system engaged. This was used primarily for vertical unmask and remask maneuvers, which enabled the pilot to fly through the autopilot without even temporary disengagement."
— "Impossible To Resist" - The Development Of Rotorcraft Fly-By-Wire Technology (PDF—Paywall)
"The RAH-66 Comanche program has proposed 4-axis controller (Harvey, 1992), but this approach is now questionable."
Note that either the '3 + 1' or '4 + 1' controller configuration would likely have been the ones decided on for the Comanche, on the basis that all the cockpit pictures that have been seen feature a sidestick and a collective stick, but no pedals.
As noted in the first text block (Impossible to Resist) the 3+1 (collective) was the final form, although the cyclic had some control authority in the vertical axis (as described). I added the emphasis in bold, and the note on "side arm controller" vs "cyclic" in brackets.
The closest thing to a twisting motion was the pre-1919 'rudder bar' shown above.
[The] pre-1919 era rudder control was most often operated with by a center-pivoted, solid "rudder bar" which usually had pedal and/or stirrup-like hardware on its ends to allow the pilot's feet to stay close to the ends of the bar's rear surface.
A stick twist does not provide a mechanical advantage to control the rudder in a real aircraft.
And in case of fly-by-wire aircraft, it doesn't provide the needed input accuracy. (See KorvinStarmast's answer.)
Some Russian fighters and trainers however have a 'trigger' brakes lever on the stick, with differential braking being provided by the rudder pedals (that do not have a toe brake motion).
PC pilots are heroes for trimming an aircraft without an elevator feedback, and for controlling the rudder with a twisting joystick.
Trivial information: the ERCO Ercoupe doesn't have a separate rudder control. That's the equivalent of a flight simulator's auto-rudder feature.
Lacking rudder pedals, the Ercoupe was flown using only the control wheel. A two-control system linked the rudder and aileron systems, which controlled yaw and roll, with the steerable nosewheel.
It's difficult to apply enough twisting torque with your wrist. Using your legs provides a lot more power. Aerodynamical forces on the rudder (or whatever control surface) are quite strong at high speed. You need some large lever arm moment in order to have authority on it. However, there could be some design solutions to allow a twisting stick yaw control. For instance adopting large aerodynamic compensators.
Hydraulic or electric commands:
Twisting stick yaw control is possible, maybe for a pilot having lost his legs. Otherwise I don't see any advantage adopting this configuration, since 3 axes are mixed, you may lose accuracy.