The ATC Orders 7-5-1(a)(4) say:
a. SVFR operations in weather conditions less than basic VFR minima
- Only within the lateral boundaries of Class B, Class C, Class D, or Class E surface areas, below 10,000 feet MSL.
- Only when requested by the pilot.
- On the basis of weather conditions reported at the airport of intended landing/departure.
That means, ATC will issue SVFR clearances - or not - in the class E surface area according to the official airport weather.
The 'twist' here is that ATC can't issue an SVFR clearance anyway unless the pilot requests it. The AIM 4-4-6 simply says that an SVFR clearance is required "when the weather is less than that required for VFR flight". So as far as I can see, if you're in VMC in a class E surface area and you expect to remain in VMC then you aren't required to request SVFR, whatever the reported airport conditions are.
I'm actually not sure what your exact scenario is here (in contact with ATC or not, within 4NM or not, etc.) but in reality I suspect that ATC would say "N12345, the field is IFR, say intentions". If you tell them you're in VMC then they have no way to challenge that but they might hint strongly that you should request SVFR - "N12345, is there anything special you'd like to request?" - because it would clarify the situation for everyone and help keep you safe and legal.