The small wings make it fly like a brick. Without the wings it would fly like a stone (e.g. not at all).
Seriously, you are taking the expression too literally. The Space Shuttle is landing like a glider plane with a (not so good) glide ratio of about 4.5:1 (see What was the Space Shuttle's glide ratio?). No brick would be able to achieve that.
Designing the Space Shuttle as a lifting body ( ≠ brick) was actually considered. It appears that a lifting body design was not able to comply with the flight envelope requirements.
