Stability is more a feature of the whole aircraft, not the wing. But I guess you are asking about wings that create a destabilizing pitch or yaw moment.
For yaw, the dominant factor is going to be sweep: forward sweep is destabilizing and aft sweep is stabilizing.
For pitch, any cambered airfoil creates a nose-down pitching moment which causes the Center of Pressure to travel aft as angle-of-attack is decreased, and causes the Center of Pressure to travel forward as angle-of-attack is increased. This is destabilizing. For aft-swept wings, washout creates a stabilizing effect in pitch, and negative washout (never actually used) would create a destabilizing effect in pitch. For forward-swept wings, washout would create a destabilizing effect in pitch.
We shouldn't forget about roll stability. In the roll axis, dihedral and sweep are stabilizing (tend to make the aircraft roll toward wings-level), and anhedral and forward sweep are destabilizing (tend to make the aircraft roll toward a steeper bank angle). Anhedral is very often used to offset the dihedral-like effects created by other aspects of an aircraft's geometry (e.g. a tall vertical fin that creates a lot of surface area above the CG), but in and of itself it could be called intrinsically destabilizing.1
Since we are talking about several different kinds of stability here, and since in any given aspect, the stabilizing or destabilizing effects of the wing geometry may be offset by other aspects of the aircraft's geometry, it's kind of hard to evaluate any given wing shape to say whether it is "as unstable as forward swept wings".
If there were a specific reason to use a forward-swept wing on an airplane, the resulting destabilizing yaw and roll characteristics could be offset by other aspects of the aircraft's design, such as a large, tall vertical fin, high wing placement, dihedral, etc. Arguably more troublesome are the structural characteristics of a forward-swept wing-- the wingtips tend to twist toward a higher angle-of-attack under heavy lift load, which further increases the lift load and twisting moment, and the bending moment on the wing spar(s). A forward-swept wing must be engineered to have a great deal of torsional stiffness to avoid problems in this area.
So in summary:
I know that forward swept wings are unstable and all, and that's one
of the reasons they aren't vastly used in planes in general.
It may be "one of the reasons", but it's actually a rather minor one. The main issue is structural-- the torsion issue.
However, I would like to know more about other types of wing
configurations that were attempted or suggested in the past that would
be as unstable as forward swept wings
A wing with significant anhedral and little or no sweep would seem to fit your description, at least if roll stability is a parameter of interest, and we are looking at the wing alone, in isolation from the rest of the aircraft. One example of such a configuration would be the wing used on the Lockheed F-104 Starfighter. Another good candidate would be the wing on the McDonnell Douglas AV-8 Harrier II-- it has some sweep, but appears to have slightly more anhedral than the wing on the F-104, and certainly has more span, giving the anhedral geometry more leverage with which to generate a destabilizing roll torque.
Footnotes:
- In the interest of brevity, we'll avoid opening the whole "can of worms" relating to complex forms of dynamic instability that can actually be exacerbated by too much roll or yaw (directional) stability, such as "Dutch Roll" oscillations (exacerbated by too much "dihedral effect", whether from sweep, actual dihedral, high wing placement, or other aspects of an aircraft's geometry), and spiral instability (exacerbated by too much yaw (directional) stability.)