Assuming a conventional, stable aircraft, most likely yes (unless the changed thrust moment around the centre of gravity balances the aircraft in trim in the new aerodynamic state).
„Normal“ aircraft are statically stable in pitch, meaning that a disturbance in either speed or angle of attack will lead to a change in aerodynamic forces towards restoring the old state. In your case, adding thrust will increase speed and hence lift, so the aircraft will naturally pitch up to restore its old speed and angle of attack (now in a climb due to added thrust). To prevent this and keep level, angle of attack must be reduced to keep lift equal weight at the higher speed.
Note the stability in angle of attack and speed is a simplistic view - real life might be more complicated, as the aircraft will most certainly enter some form of oscillation before (again, on an aircraft of conventional design) settling into a new state.
Some electronic flight control systems are designed to pronounce this self-restoring tendency to the trimmed speed (e.g. that’s Boeing’s Fly-by-wire philosophy), some other flight control systems are designed to suppress it (e.g. on a Fly-by-wire Airbus).
Hence (and slightly on top of what was actually asked), on Fly-by-wire Boeings and conventional aircraft, the throttles are more or less „up/down“ controls and the yokes are more or less „fast/slow“ controls, while on an Airbus it’s vice versa (and sticks not yokes).