In poor weather there are apt to be less VFR aircraft and more IFR. IFR are required to participate with ATC and automatically receive separation by time, altitude, and radar (when available). VFR do not require any separation and are not required to participate with ATC. There are also more flights in VFR weather than there are with IFR.
I might get flamed for this, but in general a IFR pilot is more proficient than a VFR pilot. The medical (commercial opps), testing, and currency requirements for a IFR license are usually considerably more complex than VFR - meaning (admittedly there are exceptions i.e. aerobatic pilots) IFR pilots are generally more proficient.
The following added in response to comments below
According to the FAA, "... the majority happen within five miles of an airport" https://www.faasafety.gov/gslac/alc/libview_normal.aspx?id=6851. So, cruise environment is not a factor, and IFR have ATC separation at all times, making this predominantly a VFR landing/takeoff problem. Most IFR flights are commercial requiring a 6mo medical where as private VFR pilots require 2yrs or LSA and Ultralight do not require a aviation medical. (The private medical requirements just changed, but this discussion is based on prior years data.)
In addition to excersize IFR privileges, a pilot must remain current every 6mo where as a VFR pilot does not (except for night landings).
IFR flights will have a higher percentage of two man crews compared to most VFR flights with a single pilot. The extra set of eyes is extremely significant particularly during the landing/takeoff phase of flight.