For example, ailerons allow air to flow past both the top and bottom surfaces, which makes it more aerodynamic than speedbrakes. Also, plane rudders have air flowing on both sides, which is apparently mote efficient than a spoiler. Why does this happen?
For low drag, you want to minimize the surface changes presented to the air flow. Large changes in the surface will create turbulence and pressure changes that contribute to drag. Small changes will minimize these sources of drag.
When you move a control surface like an aileron or elevator, air is flowing past both sides of the airfoil. Although this changes the direction of the flow, which gives the control force as well as some drag, the air can still flow past both sides.
In the case of a spoiler, the air cannot flow past both sides. This increases the frontal area presented to the flow, and creates a low pressure area behind the deployed surface. Creating a low-pressure area behind an object is a big contributor to drag, which is why aerodynamic bodies tend to have a teardrop shape.
Spoilers on the wing also have the effect of reducing lift. By disrupting the smooth low-pressure flow over the upper surface of the wing, they reduce the effectiveness of the wing, effectively stalling the affected portion.
For an illustration of both principles in action, see this video of a speed brake on a Mooney. This shows how even a relatively small object, when disrupting the air flow, can add a significant amount of drag.
The reason is that both sides of a control surface experience pressure changes, both ahead and aft of the hinge line. In case of spoilers and speed brakes, only one side is affected, so the effectiveness is only half as big.
Please see the pressure coefficient plots of an airfoil for three different flap deflections below. Upper and lower surface pressure are shown by color-coded lines, and lower lines belong to the lower surface. Dashed lines show the inviscid pressure, and solid lines the pressure distribution with friction effects added. The wider two lines of the same color are apart, the more lift is created. Note the contour plot below, which follows the color scheme of the pressure plots.
While control surfaces normally are deflected by less than ±20°, regular spoilers are extended by 60° or more to create as much drag as possible. But even if they are extended by similar angles to those of a control surface, the air will have no chance to flow orderly along the leeward side. Instead, this side will only have separated, chaotic airflow, while the windward side will show a flow pattern very similar to the windward side of a control surface. The pressure in separated flow will not change with deflection angle, so this side will not contribute any lift effect.
Note in the picture above that the pressure level over the whole chord is pushed up or down by the flap, because the flap is located in the aft portion of the airfoil. In contrast to that, spoilers are mounted mid-chord, where their influence over the pressure level is smaller. This reduces the lift effects of spoilers even more.
The main answer is drag. A spoiler will create more drag than a surface that allows flow over both sides. In the case of speed brakes this is what you want to happen as you want to slow the plane down. With control surfaces you want to alter airflow with out creating excess drag.
I think you question comes to the difference between aerodynamic body and blunt body. It is no so much linked to exposing one or two surfaces to the external flow, but more linked to the shape exposed to the external flow.
An aerodynamic body has a shape which produces low drag, aerodynamic bodies have a very small size normal to the direction of flight. For example, the wing is really thin in the vertical direction, but huge is the direction the plane is expected to flight.
A blunt body is the oppossite way around, exposes a big surface in front of the moving direction. The spoilers are a very good example.
I can give you a practical example, you hand. When you are in your car (in movement) and you extend your hand outside the car, having the hand parallel to the ground, you don't notice significant "pressure" or "force" over your hand (please use some nice speed...) however is you set your palm looking to the direction the car is moving, the "pressure" over you hand is much bigger as you are exposing a bigger surface.
Example the same phenomena with your hand is happening with ailerons, rudders and airbrake.
We use an airbrake to stop the airplane, that is why we use like your palm in the car. Stop the airplane.
Ailerons and rudders are intended to not stop the airplane, but control it. That's why the intention is to have low drag.