The aileron going up does produce drag as well - once it is producing a sizeable amount of downforce. At rest, it is aligned with the back end of the wing, and in a wing with no camber at zero Angle of Attack the aileron produces symmetrical drag. But wings usually do have camber and/or AoA ≄ 0, so at rest the aileron points trailing edge downwards and starts with positive lift plus the associated drag. When deflected upwards, lift and drag first reduce until the aileron is flush with the airstream, while the other aileron produces more lift and drag.
Adverse yaw has two causes:
- The "hooking" effect described above, and depicted in the figure from my old uni book. It shows the other way to explain the adverse yaw drag: the wing producing more lift also produces more induced drag.
- The roll rate resulting from aileron deflection, which increases Angle of Attack of the downwards travelling wing. This tilts the local lift vector forwardsvectors of rising vs. falling wing, and the horizontal component produces an additional adverse yaw effect. As depicted below: $\bar{\Delta c}_rn$ points upwards at one wing, and downwards at the opposite wing, at different tilt..