I read this in the normal procedures section of the A320 FCOM. Why is it so?
The use of speed brakes is not recommended. When the speed brakes are deployed, extending the flaps beyond FLAPS 1 may induce a slight roll movement, and in calm conditions a small lateral control asymmetry may remain until disturbed by a control input or by an atmospheric disturbance.