I might be mistaken, but it appears to me that you're missing an important piece on how trimming works. It's not a button you press to magically remove the control forces, but rather, you need to adjust the trim tabs (on most smaller aircraft at least, some aircraft, especially new large ones, have other mechanisms, but that's outside the 'basics' scope) to achieve the desired result.
I'll use the elevator as an example, but it applies to ailerons and rudder as well. This is a common image used when showing trim tab movement:

Basically, the trim tab extends into the airflow, pushing the elevator in the other direction. This is key though, as it's using airflow, its effectiveness diminishes when airspeed is reduced and it becomes more effective when airspeed is increased, which is exactly how the elevator itself is affected. This means that, should the airspeed change, both elevator and trim-tab will be affected the same way and the forces, and as a consequence the elevator deflection, will remain in an equilibrium. Maintaining a specific elevator deflection will cause the (aerodynamically stable) aircraft to remain at a selected angle of attack and therefore airspeed. In order to change airspeed, you need to change angle of attack, and so you need select a different elevator deflection, prompting the need to 're-trim' fairly often during a flight, especially during departure and approach. This applies to elevator trim, I've never flown with (in-flight adjustable) aileron or rudder trim, but I assume it is primarily used to counter effects of engine torque and P-factor, so it would be adjusted whenever you're changing the power setting.
You could, theoretically, skip the trimming step and hold it with arm strength, but your arms would eventually go numb and it would be virtually impossible to keep it on airspeed while accomplishing the other tasks in the cockpit during high workload periods. The March/April 2014 issue of FAA Safety Briefing talks a bit about trimming, and mentions that the Weber-Fechner law, which discusses the concept of Just-noticeable difference, states that you need about 14% of difference to the pressure you're manually holding to notice a difference, so trusting the feel of your hand pulling the yoke is not a particularly good idea. I do believe some aircraft, in particular larger ones, would require more force than humanly possible for the needed elevator deflection in certain situations, suggesting the need to use trim, but I might be mistaken on that point.
The movement of the trim tab is sometimes driven by an electric motor (or otherwise powered, hydraulics for example) in which case you have up and down buttons, and sometimes it's manual, in which case you usually have a wheel which can be rotated to pull the tabs up or down.
Up and down, which is often referred to as 'nose up' and 'nose down', signify what kind of control forces you want to remove. If you find yourself constantly pulling the nose up to maintain the desired airspeed and attitude, you'll need to apply nose up (or 'up') trim while simultaneously releasing the back pressure and vice versa. This sounds harder than it is, it just means you hold the yoke in place, and you will notice the amount of pressure needed to hold it there drops until it's completely gone. This is one of those things which is actually harder to do in a home simulator than in real life, as you'll have to move the control while applying trim. Note that the 'up' and 'down' labels do not mean the tab will move up or down, remember the tab travels in the opposite direction of the elevator, and to pull the nose up, the elevator needs to move up into the airflow, meaning the tab will actually move down, as can be seen in the picture.
As a bonus; in aircraft with no autopilot and autothrottles, if you do not apply any control forces the aircraft (aerodynamically stable) will maintain a particular airspeed. This means that for a specific configuration (flaps, landing gear, weight, power, etc) it's possible to trim the aircraft for a specific speed, this is very handy when it's crucial to maintain a specific airspeed such as during departure or approach.
This also means, which might not be obvious for someone not familiar with aircraft, that it's not possible to trim an aircraft to maintain altitude. Sure, there's usually a specific airspeed which will keep the aircraft in level flight, but that's only until you enter the next up- or downdraft, or until you've burned off a bit of fuel and aircraft has become lighter. So you will either need constant speed or power adjustments to keep it on altitude, most aircraft without autothrottles use the former.