The main problem with the B-2 Spirit (or any flying wing aircraft, for that matter) is that they are statically unstable.

Source
If the lift vector of the wing acts before the center of gravity (as is the usual case when only wings are considered), any perturbation which results in an increase in angle of attack results in a positive moment about the center of gravity, which further increases the angle of attack an so on.
The only way to control such an aircraft is through computers, which continuously detect any changes in the aircraft attitude and apply corrections as required, as it is near impossible (or incredibly taxing) for human pilots to fly the aircraft.
This is the method used in order to 'fly' the B-2 Spirit in the absence of a horizontal stabilizer. If there is any problem with the flight computer (which is remote) or with the sensors supplying data to it, the aircraft will become unflyable and crash.
Another control issue with the B-2 is that the aircraft has little directional stability with no vertical tail. The swept wing offers some stability, though not in the same level as a vertical tail.

Source
Actually, only 21 of these were built, for the following reasons:
The B-2 is costly; by costly I mean probably the most expensive aircraft ever built. With a total program cost of over 45b USD, each aircraft costs a whopping 2.1b USD (which was caused due to reduction in numbers from the initial 132). Also, the (hourly) operating costs of B-2 are twice that of other bombers in the USAF fleet.
With the dissolution of the Soviet Union, the B-2 had effectively become an aircraft without a mission, as it was primarily designed to penetrate Soviet air defence systems.