Can planes bank without turning
Yes, a plane may fly in a banked attitude without turning.
But when the plane is tilted to the left or right relative to Earth's
surface and gravity, shouldn't it be "pushed" to the left or right?
Yes, whenever a plane is banked and the wing is generating lift, the lift vector has a horizontal component that tends to "push" the plane to the side, i.e. tends to make the flight path curve1. If the plane is banked and still is travelling in a straight line, some other force must be opposing the horizontal component of the lift vector. That other force is typically the "sideways lift" force generated by making the fuselage fly sideways through the air, so that it acts like an airfoil-- like a "mini wing".
Here's how it works in actual practice-- use the ailerons to roll left. As the bank angle increases, the plane will tend to turn left, but you can prevent this by applying right rudder as needed to hold the heading constant. The wind (airflow) is now hitting the left side of the fuselage and creating a force toward the right that opposes the force from the banked wing. Once you reach your desired bank angle, you can relax some of your aileron input, but you'll typically2 need to maintain some amount of left aileron deflection, or else the bank angle will start to decrease, due to the aerodynamic effects of the sideways airflow created by the right rudder input. So you are holding left aileron and right rudder, and you are flying sideways through the air at a constant bank angle without actually turning. This is called a "cross-controlled sideslip".
Footnotes--
1-- unless the wing is "unloaded" to the zero-lift angle-of-attack, as would be the case in a sustained 90-degree "knife-edge" maneuver.
2-- an exception would be an aircraft with no coupling between slip and roll, such as an aircraft optimized for aerobatics with a mid-mounted wing with zero sweep or dihedral. In such an aircraft, the aileron input could be relaxed all the way to zero once the desired bank angle is reached, and the aircraft would remain in a banked attitude. In such an aircraft, the sideslip could be maintained with rudder deflection alone. There are even some aircraft that actually require the ailerons to be deflected in the same direction as the rudder, rather than the opposite direction, to maintain a constant bank angle during a sideslip, but these are unusual cases. In most aircraft, the rudder and aileron inputs must be "crossed" to maintain a steady-state sideslip with constant heading and bank angle.