According to FAA JO 7110.65 5-12-8 'Validation of MODE C READOUT', Mode C is valid if there is less than 300 feet deviation from the pilot reported altitude.
If ATC instruct 'A' aircraft to maintain 5,000ft, and its Mode C indicate 4,900ft on radar, and ATC instruct 'B' aircraft to maintain 6,000ft and its Mode C indicate 6,000ft, are A and B vertically separated?
By looking radar, it is 900 feet separation, but pilot reported altitude is 1,000 feet separated.
Yes, they are separated. Both according to mode C (in your example, there is 1,100 ft between them - not 900) and in general. It is normal for there to be small deviations in mode C in the range of 100 to 200 ft. If I have two aircraft crossing each other, supposedly with 1,000 ft in between, but the indicated (mode C) separation is slightly less, I will certainly ask the pilots if they are indeed at their assigned altitudes. If they confirm, then all is well. A pilot report generally overrules mode C (to the extend where there is even a procedure for asking the pilot to turn off mode C reporting if there is a discrepancy between pilot report and mode C).
The IFR-IFR separation standard is 1000ft precisely to accommodate altimeter error.
For instance, if the lower aircraft is actually 300ft higher than assigned (5300ft) and the higher aircraft is actually 300ft lower than assigned (5700ft), then they are 400ft apart, which is still safe though obviously not ideal.
The standard for IFR-VFR or VFR-VFR separation can be reduced to only 500ft (less than the combined margin of error) because the aircraft should be able to see and avoid each other in the event of worst-case altimeter errors.
The Mode C Altitude being displayed on ATC's radar can be used for separation purposes if it is within 300 feet of the pilot's reported altitude as the OPs question points out.
Important, however, is the understanding that the actual maximum altimeter error allowed during FAR Part 43 testing and checking (with respect to the Mode C readout versus the actual altitude the altimeter is displaying) is 125 feet.
(REF: See the excerpt from FAR Part 43, Appendix E, (C) below)
(c) Automatic Pressure Altitude Reporting Equipment and ATC Transponder System Integration Test. The test must be conducted by an appropriately rated person under the conditions specified in paragraph (a). Measure the automatic pressure altitude at the output of the installed ATC transponder when interrogated on Mode C at a sufficient number of test points to ensure that the altitude reporting equipment, altimeters, and ATC transponders perform their intended functions as installed in the aircraft. The difference between the automatic reporting output and the altitude displayed at the altimeter shall not exceed 125 feet.
Also, as noted in the Aeronautical Information Manual (among other guidance), when the pilot sets his altimeter to the airfield elevation (at the correct location on an airport), if the altitude (displayed on the altimeter) is +/- 75 feet or greater than the actual airfield elevation, the error should be referred to a repair station.
(REF: AIM, para. 7-2-3)
If the difference from the known field elevation and the altitude read from the altimeter is plus or minus 75 feet or greater, the accuracy of the altimeter is questionable and the problem should be referred to an appropriately rated repair station for evaluation and possible correction.
So, the FAR Part 43 periodic inspection/test is successful only if the Mode C reported altitude (during the bench test) is +/- 125 feet. And, most importantly, if the crew is following the published guidance, the maximum actual error between what the altimeter is displaying (to the crew) and the actual altitude of the aircraft will be +/- 75 feet.
As a result, not considering any other potential errors (e.g., cold temperature, instrument failure, etc.), two aircraft that are supposed to be separated by 1000 feet would likely never be actually separated by less than 850 feet, assuming the aircraft are at their assigned altitude and using the same altimeter setting, (if both aircraft altimeter's were off by 75 feet). The fact that on ATC radar the Mode C readout being displayed might show that as little as 400 feet of vertical separation (technically possible, but extraordinarily implausible) is really irrelevant in light of the fact that the altimeter the pilot is relying on is displaying an altitude that has a max instrument error of only +/- 75 feet.
