Visual separation is addressed specifically in the AIM (see below). It is an ATC instruction sometimes given when the pilot reports another aircraft in sight. ATC is then able to instruct the pilot to "maintain visual separation" from the other aircraft, and it doesn't matter if they are IFR or VFR.
Then it becomes quite simple:
- Don't hit the other airplane!
- Do whatever you have to do in order to avoid them. (Usually this is nothing since you just get closer than ATC is allowed to let you otherwise.)
- If you are on a specific ATC clearance, then notify ATC immediately if you have to do something contrary to that clearance.
- Keep the other aircraft in sight.
- Immediately report to ATC if you lose sight of them.
- Avoid the wake turbulence of the other aircraft.
Standard radar separation no longer applies since the pilot has accepted separation responsibility from the one specific aircraft that the clearance applies to. This allows ATC to let aircraft fly more efficient routes (not having to turn to avoid the other aircraft) and also lets more aircraft land in a given amount of time because they can be closer together on final.
Pilots have to use good judgement as to how close they get to the other aircraft, but it is done visually. They can use any tools at their disposal to help (TCAS, asking ATC how far away they are, etc.) but it all comes down to "that looks good enough for me".
Section 5. Pilot/Controller Roles and Responsibilities
5-5-12. Visual Separation
a. Pilot.
1. Acceptance of instructions to follow another aircraft or to provide visual separation from it is an acknowledgment that the pilot
will maneuver the aircraft as necessary to avoid the other aircraft or
to maintain in‐trail separation. Pilots are responsible to maintain
visual separation until flight paths (altitudes and/or courses)
diverge.
2. If instructed by ATC to follow another aircraft or to provide visual separation from it, promptly notify the controller if you lose
sight of that aircraft, are unable to maintain continued visual
contact with it, or cannot accept the responsibility for your own
separation for any reason.
3. The pilot also accepts responsibility for wake turbulence separation under these conditions.
b. Controller. Applies visual separation only:
1. Within the terminal area when a controller has both aircraft in sight or by instructing a pilot who sees the other aircraft to
maintain visual separation from it.
2. Pilots are responsible to maintain visual separation until flight paths (altitudes and/or courses) diverge.
3. Within en route airspace when aircraft are on opposite courses and one pilot reports having seen the other aircraft and that the
aircraft have passed each other.
More info is here:
4-4-14. Visual Separation
a. Visual separation is a means employed by ATC to separate aircraft in terminal areas and en route airspace in the NAS. There are
two methods employed to effect this separation:
1. The tower controller sees the aircraft involved and issues instructions, as necessary, to ensure that the aircraft avoid each
other.
2. A pilot sees the other aircraft involved and upon instructions from the controller provides separation by maneuvering the aircraft to
avoid it. When pilots accept responsibility to maintain visual
separation, they must maintain constant visual surveillance and not
pass the other aircraft until it is no longer a factor.
NOTE- Traffic is no longer a factor when during approach phase the other aircraft is in the landing phase of flight or executes a
missed approach; and during departure or en route, when the other
aircraft turns away or is on a diverging course.
b. A pilot's acceptance of instructions to follow another aircraft or provide visual separation from it is an acknowledgment that the
pilot will maneuver the aircraft as necessary to avoid the other
aircraft or to maintain in-trail separation. In operations conducted
behind heavy jet aircraft, it is also an acknowledgment that the pilot
accepts the responsibility for wake turbulence separation.
NOTE- When a pilot has been told to follow another aircraft or to provide visual separation from it, the pilot should promptly notify
the controller if visual contact with the other aircraft is lost or
cannot be maintained or if the pilot cannot accept the responsibility
for the separation for any reason.
c. Scanning the sky for other aircraft is a key factor in collision avoidance. Pilots and copilots (or the right seat passenger)
should continuously scan to cover all areas of the sky visible from
the cockpit. Pilots must develop an effective scanning technique which
maximizes one's visual capabilities. Spotting a potential collision
threat increases directly as more time is spent looking outside the
aircraft. One must use timesharing techniques to effectively scan the
surrounding airspace while monitoring instruments as well.
d. Since the eye can focus only on a narrow viewing area, effective scanning is accomplished with a series of short, regularly
spaced eye movements that bring successive areas of the sky into the
central visual field. Each movement should not exceed ten degrees, and
each area should be observed for at least one second to enable
collision detection. Although many pilots seem to prefer the method of
horizontal back-and-forth scanning every pilot should develop a
scanning pattern that is not only comfortable but assures optimum
effectiveness. Pilots should remember, however, that they have a
regulatory responsibility (14 CFR Section 91.113(a)) to see and avoid
other aircraft when weather conditions permit.
For even more information, there is an entire section of the Air Traffic Control job aid which the controllers use about visual separation.
