Aircraft POH and AFM utilize engineering data obtained from test flights, conducted by highly experienced test pilots who have a goal of presenting the best possible data for an aircraft. Performance data is used to sell airplanes, long before the owner is working the numbers for his or her operations. The subject aircraft have been primped, and the engines are working and in ideal condition. So, if one flies with the same skill, and a finely tuned aircraft, with a ideally running engine, and can repeat a short field take off again and again, until their technique is highly polished, then perhaps one might consider using the book figure.
That doesn't often happen. For most of us, the aircraft we fly have bug splat on the leading edge, tires that are not perfectly balanced, and other imperfections. Furthermore, for the weekend flyer, and even for a research test pilot, all operations are not conducted with the degree of refinement which is done during the gathering of data for performance publication.
So given all of that, what are the margins?
I have not seen large scale studies, but I have had several hundred students, and several score of flight instructors who I have taught and prepared. To my knowledge none of them have had any serious accidents, and I have never scratched paint on an aircraft. So here are my guidelines:
Range in VMC conditions, non-sparse areas 30%
Range in IMC, or night, or over sparsely populated areas 50%
Reduce margins by 10% for every 500 hours of operation in those conditions, down to 10% and 20% respectively.
For operations like take off from short fields, add 50%. Reduce the margin by 10% for every 20 similar operation over a one month period, which you can verify book performance has been achieved. (Note this seldom happens, because we are seldom watching closely.) Maintain a minimum margin of 10%
Why 10% margins? Well, the windsock may change during your takeoff/landing, the tires might be a little under inflated. There might be bug splat on your leading edge and the airfilter on the engine may have a little dirt on it. There is always something which is not perfect. So keep a margin.
Determine go/no-go points for TO and LNG. Do the same for fuel burns on XC operations. If you are going to eat into your predetermined margin, divert. Chances are the flight that is causing you to eat into your margins may be at night, with rime and clear ice, winds higher than forecast, and more turbulence than you are comfortable with. So with things less than perfect you are absolutely justified in being conservative.
If you are getting book figures, usually there is something not exact as the book. For example, you may be under gross weight measuring climb. So don't fool yourself.
One thing you should be able to duplicate is cruise performance. Dip your tanks after every flight and compare the findings with your calculated figures. If you can't duplicate the book figures, without margin, then something is wrong. Climb performance is a bit harder to achieve the precise figures in the book. Even if you can duplicate fuel burn rates, keep a margin.
Making copies of the performance data, and marking the copies with your achieved results will help you recognize where you and your plane are performing. Having friends spot you on takeoff and landings, perhaps when you are practicing specialty landings, and recording that data is also helpful to assess how you are doing. Don't forget to record and factor in things like changes in gross weight, winds and even how wet and long the grass is. They all impact performance.
In my own plane, I have dip sticks which I have calibrated for fuel level, I have a light weight digital bathroom scale, etc. as tools for conducting a flight, with verifiable performance.
In the professional world, I have had to optimize payloads with aircraft to get acceptable performance for low level survey operations. Repeating tests, with supplemental metrics (eg radar altimeter data) and then developing SOPs is something I have spent allot of time doing. All of us can work to refine our procedures and practices to enhance the safety of our operations, without imposing extreme and unjustified margins.
Over the years I have lost too many colleagues and friends to aviation accidents. Retrospectively, every one of them has made one or more judgment errors. Why cut it close, when a little more margin means you may fly again another day?