Q: Can ATC issue instructions to exclude certain aircraft over international waters?
No. Live as of writing this, below shows a Russia-operated plane snaking its way through the Baltic Sea.

— Flightradar24
Civilian FIRs that extend beyond the territorial waters (12 nautical miles) are for traffic coordination, but when a country is banned from overflying the actual borders, international waters and EEZs are fair game, which are always inside somebody's FIR.
Exclusive Economic Zones (EEZs) only have material effect on maritime resources, including the sea bed, and do not have any legal consequences for aviation (UNCLOS Articles 57 and 58).
— International Airspace and Civil/Military Cooperation. ICAO. (PDF) [emphasis added]
6-Mar update: With Canada, USA, Denmark (inc. Greenland), and Iceland now not permitting Russian planes, below is an example of an Aeroflot flight to Dominican Republic where the Boeing 777 threaded its way between Greenland and Iceland (through one or both FIRs), and flew through the Western North Atlantic that is managed by Canada (Gander FIR) and USA (New York Oceanic FIR). (An earlier flight overflew Greenland, Canada, and USA.)

— Flightradar24
Another case is the 2017–2021 airspace closures to Qatari planes:

— washingtonpost.com
Above shows how Bahrain FIR – which is disproportionately big – extends to share borders with UAE and Iran FIRs; by staying away from territorial waters, Qatari planes could just fly through.
Notes:
The Qatari case went to the International Court of Justice, which judged ICAO should rule in the matter, but then the airspaces were reopened before it got that far (un.org; July 2020).
Military planes also do it all the time, e.g. like how Russian military planes have been going to the edge of UK's borders for many years now (here's one from 2015), cutting through many civilian FIRs while doing so.
Could be of interest: passing by Denmark into the Baltic Sea is open for all by the Copenhagen Convention of 1857.