Today's highly automated airplanes should be not too hard to control remotely - after all, all the cockpit control inputs become electrical signals, and at that point a multichannel remote could be plugged in. The Russians even flew their Buran version of the Space Shuttle autonomously, using a neural network to command the craft on its single flight in 1988.
However, a test pilot flies an aircraft with all his senses. Without the acceleration, a full 3D-view and the force feedback from the controls, he doesn't get the full picture when sitting in a control room and looking at screens full of data. That is done by test engineers anyway, and the pilot's job is to complete the observation of a new airplane's behavior by exposing himself to it.
You might argue that the 3D-view is just a matter of several cameras and displays, but the needed telemetry bandwidth for this would be immense. In a normal test flight, already hundreds of parameters are measured many times a second and transmitted to a ground station, where specialists for each system watch for anomalies in the data. It is much better to spend the telemetry budget for this than to send pictures down. The lack of acceleration and vibration alone would deprive the test pilot of an essential part of the picture, because it simply cannot be realistically recreated in a ground station.
Also, the inevitable delays incurred by radio transmission and processing for display will quickly put a pilot out of a loop which he could easily control if he were sitting in the airplane. The range of controllable eigenfrequencies would be reduced to phenomena happening at maybe 0.2 Hertz or less, whereas a pilot in the cockpit can comfortably stay ahead of oscillations with 0.5 Hertz or less.
However, it must be said that sometimes the test pilot does turn out to be a burden for the progress of the flight. On the second flight of the British TSR-2 prototype, one fuel pump began to develop an imbalance, oscillating just at the eigenfrequency of the pilot's eyeballs. He had to throttle back one of the engines to regain his vision.