Why does everyone hold in position when Air Force One is just taxiing? This can cause a delay that could cascade to other airports.
No doubt security has a lot to do with it. With only one vehicle supposed to move on the airport premises, everything else that moves can be easily spotted and classified as a potential threat, and resources dispatched to handle it.
If there are dozens of other aircraft and hundreds of cars and trucks moving around, this gets a lot harder.