TCAS air-to-air messages use the same format and frequencies as "Mode S" surveillance radar. They're defined (in great and overwhelming standardesque detail) by a document called "ICAO Annex 10".

ICAO wants serious money for a copy of it -- and of course if you're producing actual avionics nothing less than the official source will do -- but for satisfying idle curiosity it's often possible to find a draft or bootleg by persistent googling.

The Mode S message format is also used for ADS-B broadcasts, so equipment to receive and decode them is relatively easily available to hobbyists. The lower layers of open-source ADS-B software will also be generally relevant.