The captain is the captain, so when he says "go around" then the PF needs to initiate the go-around. If the FO doesn't initiate the go-around as directed, then the captain can (at that time as at essentially any other time he sees it necessary) take over control of the aircraft. That's the captain's authority and it's pretty clear.
The other case is slightly less clear, but only slightly.
Most operations probably have a directive along the lines that the PF is required to go around when the PM so directs. So if the captain is flying and the FO directs a go-around, what the captain/PF should do is perfectly clear. If he doesn't do so, the next step is probably spelled out in the operations manual; probably something along the lines of, repeat the directive once, then assume incapacitation and take over the controls to execute the go-around.
The really bad case would be, the captain is intent on landing regardless and the FO believes that they're about to crash. At that point, the question (really for both pilots) becomes, are you sufficiently sure of your position that fighting for the controls close to the ground is less dangerous than doing what the other pilot wants? If we're somehow down to our last minutes of fuel & the copilot wants to go around because a dog ran onto the runway, a reasonable captain would probably overrule him and land anyway. If the captain is so intent on landing that doesn't see the aircraft on the runway that he's about to land on top of, a reasonable copilot might push the throttles forward like his life depends on it.
If you've reached the point where "go around" "go around" "I have the aircraft, go around thrust" hasn't worked, you've left the realm of most manuals. The implicit assumption is that an incapacitated PF isn't going to have the capacity to fight you for the controls; I doubt very many operations manuals devote much space to the nightmare case of wrestling for the controls against a maniac determined to do something awful. Thankfully, such scenarios are almost entirely the stuff of imagination -- commercial flight is essentially the safest form of transportation ever known to mankind. That remains true even though the one in (very literally) millions of flights where something does go horribly wrong (Germanwings crash into the Alps, etc) gets extraordinary publicity.