I have read that the oceans are big enough that ground radar stations only reach so far out. I assume that means after a certain distance from the coast they leave ATC services. Do airplanes employ any techniques to avoid traffic separation outside of ATC services aside from following their flight plan and the rules of the airway they are flying, and for an emergency backup the TCAS system?
They don't leave ATC services when they enter oceanic airspace, but the service provided is no longer based on radar. Instead separation is maintained by following procedures. For example, that means that only certain routes (tracks) are flown, with 15 (or 10) minutes separation between aircraft on those tracks. The aircraft strictly adhere to speed instructions and report to ATC at specific points.
Another form of separation is vertically, just like in a radar environment, by assigning different altitudes (flight levels) to aircraft.
The voice communication is over HF and is supplemented by data link over CPDLC / ADS-C via satellite link.
To expand on DeltaLima's answer:
They use routes called the NATs (North Atlantic Tracks), which can differ each day depending on weather and wind speeds. Each track is a set of coordinates. A pilot will do a position report at each coordinate or every 10 degrees of longitude. The aircraft has to get special clearance to fly oceanic about 30 minutes before crossing the entry waypoint for the NAT. Each NAT is 60 NM apart.
To clarify a bit more, HF radio is used to communicate, since it has a longer range than VHF, but is much noisier. This is where systems like SELCAL come in. SELCAL stands for SELective CALling, which allows the pilots to lower their radio volumes. Each aircraft has a unique SELCAL code which consists of four characters that can be made up of A, B, C, D, E, F, G, H, J, K, L, M, P, Q, R, and S. When a controller wants to talk to a pilot, they send a "ping" to the aircraft, which plays a sound in the cockpit to tell the pilots to get on the radio. Position reports can be done via HF radio, or increasingly via text (CPDLC/ACARS), taking advantage of ADS-B/C. These communication systems are pretty complicated, so I recommend you search for them on the web or look at the sites I linked below.
Aircraft are required to be 1,000 feet apart vertically. Eastward flying aircraft have odd altitudes such as 37,000, 39,000, 41,000, etc, while westerly flights will be the opposite. They are also required to be about 10 minutes apart from each other.
Gander Radio (notice not "Radar"!) controls much of the North Atlantic airspace from Gander, Newfoundland, Canada. Shanwick is another major Radio station. New York controls further south, along with Santa Maria controlling relatively small airspace.
There is a similar system over the Pacific ocean, with stations in Anchorage, Alaska, and Oakland.
You should at least watch this video, it's only 3 minutes, and it's summarized well:
More about ADS-B
More about SELCAL
More about ACARS: