There are many factors going into the sizing of elevators, and they are all interconnected with other considerations such as CG envelope, static margin, flight controls, etc.
Here are some factors that will influence up-sizing the elevators. Usually, the minimum elevator sizing is determined by low speed requirements:
Ability to perform early rotation in both all engine operating and one-engine-inoperative conditions (14 CFR 25.107(e)).
Ability to perform go-around in the highest flap extended position (14 CFR 25.119).
Ability to stall the aircraft to demonstrate the intended stall speed (14 CFR 25.201). This may not be obvious, but if you lack the elevator authority to demonstrate the intended stall speed, then you can only declare what you have demonstrated, which negatively impacts all the performance speeds for takeoff and landing.
If the elevator is power actuated (e.g. hydraulically powered), then you also need to consider regular takeoff, landing, go-around, and maneuvering at low speed, with failures (14 CFR 25.671(c)).
Here are some factors that will influence down-sizing the elevators:
Demonstration of static longitudinal stability at Vmo/Mmo. For column based inceptors, the average stick force to speed change must be no shallower than 1 lb per 6kt (14 CFR 25.173(c)). For FBW aircraft, too large an elevator can also lead to residual oscillations.
General over-sensitivity to control inputs at high airspeed.
Of course, most of these aforementioned points are heavily influenced by flight control design, such as anti-servo tab, bob weights, artificial feel force, etc.