I have read that the oceans are big enough that ground radar stations only reach so far out. I assume that means after a certain distance from the coast they leave ATC services. Do airplanes employ any techniques to avoid traffic separation outside of ATC services aside from following their flight plan and the rules of the airway they are flying, and for an emergency backup the TCAS system?
They don't leave ATC services when they enter oceanic airspace, but the service provided is no longer based on radar. Instead separation is maintained by following procedures. For example, that means that only certain routes (tracks) are flown, with 15 (or 10) minutes separation between aircraft on those tracks. The aircraft strictly adhere to speed instructions and report to ATC at specific points.
Another form of separation is vertically, just like in a radar environment, by assigning different altitudes (flight levels) to aircraft.
The voice communication is over HF and is supplemented by data link over CPDLC / ADS-C via satellite link.
To expand on DeltaLima's answer:
They use routes called the NATs (North Atlantic Tracks), which can be different each day depending on things like weather and wind speeds. Each track is a set of coordinates. A pilot will do a position report at each coordinate or every 10 degrees of longitude. The aircraft has to get special clearance to fly oceanic about 30 minutes prior to crossing the entry waypoint for the NAT. Each NAT is 60 NM apart.
To clarify a bit more, HF radio is used to communicate, since it has a longer range than VHF, but is much noisier. This is where systems like SELCAL come in, but I won't talk about that here. They can also do position reports over the radio or over text (CPDLC/ACARS), taking advantage of ADS B/C, which I also won't talk about here. These communication systems are pretty complicated, so I recommend you search it on the web or look at the sites I linked below.
Aircraft are required to be 1,000 feet apart vertically. Eastward flying aircraft have odd altitudes such as 37,000, 39,000, 41,000, etc, while westerly flights will be the opposite. They are also required to be about 10 minutes apart from each other.
Gander Radio (notice not "radar"!) controls much of the North Atlantic airspace from Gander, Newfoundland, Canada. Shanwick is another major Radio station. New York controls further south, along with Santa Maria having relatively small airspace.
There is a similar system over the Pacific ocean, with stations in Anchorage, Alaska, and Oakland.
You should at least watch this video, it's only 3 minutes, and it's summarized:
More about ADS-B
More about SELCAL
If you want to learn more about ACARS: