The comments above by John K. and Maury Markowitz are on the right track. The serrated trailing edge of the spoilers has an effect on [vortex shedding][1]. On modern jet engine nacelles the serrated edges at the rear reduce noise. However, on the B-52 the most likely reason for this feature of the spoilers is to reduce the possibility of vortex shedding inducing vibrations that could cause the wing structure to resonate due to the harmonic oscillation of the vortex created by the spoiler. This is also known as flutter. My guess is that at some point in the life of the aircraft design it was determined that this could be an issue, and this was the solution.

If you think about the last time you flew on an airliner where the pilots deployed the spoilers while still in flight, you may remember the vibration generated by them. That is the consequence of vortex shedding, and that is what the serrated edges are designed to minimize on the B-52.


  [1]: https://en.wikipedia.org/wiki/Vortex_shedding