Is it possible for ATC to apply 3 miles separation between an IFR aircraft in class A and a VFR aircraft operating in class E, just below class A?
Keep in mind that ATC only separates IFR traffic from other IFR traffic. So they would not provide separation from any VFR traffic no matter what the airspace (with the exception of Class B).
AIM 4−4−11. IFR Separation Standards
b. Separation will be provided between all aircraft operating on IFR flight plans except during that part of the flight (outside Class B airspace or a TRSA) being conducted on a VFR−on−top/VFR conditions clearance. Under these conditions, ATC may issue traffic advisories, but it is the sole responsibility of the pilot to be vigilant so as to see and avoid other aircraft.
It would be possible for U.S. ATC to apply 3 miles lateral separation between aircraft in Class A airspace (for example at FL180 or higher depending on local altimeter setting) and a VFR aircraft operating in class E airspace at any altitude below FL180 (by FAR no higher than 17,500 ft. msl).
But, ATC in the U.S. does not apply/provide required vertical or lateral separation between aircraft in Class A airspace (all of them operating under IFR) and VFR aircraft operating in Class E airspace.