AOPA has an article under Pilot Resources that specifically covers this situation called Logging Pilot-in-Command (PIC) Time (emphasis added by me):
Unlike driving cars, the PIC may allow anyone, including a non-pilot,
a pilot who may not legally act as pilot in command, or another fully
qualified pilot fly the airplane, or be "sole manipulator of the
controls" during the flight. The PIC is not required to sit in the
left pilot seat. Regardless of where the PIC is sitting in the
airplane or who is manipulating the controls, the PIC is ultimately
responsible and accountable for the safety and operation of the
flight.
If you are the PIC letting someone else fly, remember (like they say above) that you are still responsible for everything that they do, so make sure that you never put yourself in a situation where you are not able to maintain (or regain, if necessary) control of the aircraft.
I would recommend that you carefully brief the passenger about not fighting you on the controls and to immediately let go if you tell them to. Do this before you let them fly and be clear that it is about flight safety. Most of them will be listening carefully once you say that. :-)
I have done this plenty of times, and another thing you can do (especially when at lower altitudes) is to keep your hands and feet on the controls at all times so that you can stop them from doing anything too out there.
Now that being said, air carrier regulations specifically prohibit pilots from letting passengers fly so the above would only apply to private flights:
§135.115 Manipulation of controls.
No pilot in command may allow any person to manipulate the flight
controls of an aircraft during flight conducted under this part, nor
may any person manipulate the controls during such flight unless that
person is—
(a) A pilot employed by the certificate holder and qualified in the
aircraft; or
(b) An authorized safety representative of the Administrator who has
the permission of the pilot in command, is qualified in the aircraft,
and is checking flight operations.
§121.545 Manipulation of controls.
No pilot in command may allow any person to manipulate the controls of
an aircraft during flight nor may any person manipulate the controls
during flight unless that person is—
(a) A qualified pilot of the certificate holder operating that
aircraft.
(b) An authorized pilot safety representative of the Administrator or
of the National Transportation Safety Board who has the permission of
the pilot in command, is qualified in the aircraft, and is checking
flight operations; or
(c) A pilot of another certificate holder who has the permission of
the pilot in command, is qualified in the aircraft, and is authorized
by the certificate holder operating the aircraft.