The answer is both modelling and/or empirical methods are used.
For aircraft with a closed loop Flight Control System then the clearance requirements before flight requires that sufficient gain and phase margins exist for the system. For example with Military Aircraft this was defined in MIL-F-9490D. This would involve building mathematical models of the aircraft and control laws and calculating margins from the frequency response of the model. This allows the engineers to analyse the robustness of the system under various tolerance and failure states before flight.
It is possible to analyse stability margins from flight data, but note even the RTSM software referenced in the comments talks about comparing the flight data calculated margin against an a priori model.
Also note that Handling Qualities requirements, i.e. saying it is PIO prone, are additional requirements over and above the basic stability requirements. This introduces another feedback path with a pilot in the loop. For this there are additional frequency response parameters to meet, for example average phase rate. Again examples in MIL-Spec requirements MIL-F-8785C, these will all be assessed using mathematical models.