The table in the report omits a condition. Here is the ECAM message in full:

The limitation applies when the aircraft is in emergency electrical configuration (run by the ram air turbine), which is one way to have the flight laws degraded.
The manual does not give a direct reason (not unheard of), but in fact there's more than one reason.

► As a quick summary, you lose 2/3 of the spoilers and the outer ailerons, and if more unlucky, you lose 5/6 of the spoilers.
If you happen to use the now-not-very-helpful spoilers, and you lose more hydraulic pressure, they'll get stuck in the up position (adding drag and degrading roll). So, DO NOT USE makes sense.
The how:
In emer elec config, the plane loses 3 of the 5 flight control computers; what remains are P1 and S1 (refer to diagram above).
The outer ailerons go into an unpowered state to conserve hydraulic fluid/pressure – the right aileron upfloats* like on the A320 (no P3 and S2).
If the planes loses G or Y hydraulic pressure, the plane is left with one panel on each wing, which is likely to be needed for roll control at slower speeds (the outer ailerons are already out).
The references (click to view):

All images are from the A340 FCOM, and color highlights are mine. Do not use for flight.
A330 vs A340
Thanks to @JZYL, note that there are some differences between the A330 and A340, while the A340 would announce MOST SPOILERS being INOP, the A330 would announce HALF SPOILERS (an extra computer remains powered). The same rationale given should apply though.
* This is also confirmed from this FCTM (smartcockpit.com) of an airline with both the A330 and A340 in its fleet (see PDF page 207):
(...) The AP will disengage when the failure occurs but AP 1 can be reengaged on the A333 if the EMER GEN is powered by an EDP. As there is no AP available in this configuration in the A340, the Captain must take control and fly the aircraft manually. The A340 aircraft is out of trim in roll due to right outboard aileron upfloat, caused by no power being available to Prim 3 or Sec 2, and neither pitch trim or rudder trim is available. (...)
Note: I haven't found a reason for the different architecture, but ETOPS is a possibility.