You are right, it helps to de-sweep the outer wing. The main advantages are (read this Flight article from 1965 here for a much more detailed explanation):
- An identical critical Mach number over the whole span as sweep is reduced together with relative thickness.
- A smooth cross-section distribution in the direction of flow which helps to reduce transsonic drag.
- A tendency to pitch up in ground effect such that the wing will essentially land itself, especially when combined with a T-tail.
However, as long as wings are made of aluminum, the manufacturing effort is significantly higher.
The basic problem is the needed two-dimensional bending of the skin panels. Using a straight spar line and a straight leading edge requires bending only in one dimension, but when sweep changes, the skin panels would need to stretch in some corners while being bent. This requires much more expensive tooling, or a break in the structure which increases structural mass.
Even one composite wing I know of has a straight leading edge, even though the aircraft would have benefited from de-sweeping. Both the TKF-90 and the X-31, which were designed by the same people before, had reduced sweep on the outer wing, and for good reason. When the multinational project started and they talked to the BAe engineers, the British side insisted on a straight leading edge for easier manufacture, ignorant of the fact that using composites made their point moot. Yes, that was practically the same company that had built the Victor before.
Another disadvantage is the fact that any sweep change will convert bending moment into torsion. Normally, this reduces the incidence of the outer wing when a swept-back wing is bending under lift loads. De-sweeping the outer wing will introduce an uptwisting torsion which can cancel the torsion from sweep-back, and Handley Page claimed that the Victor wing was close to aero-isoclinic. However, this reduces flutter damping, resulting in a lower wing flutter speed (PDF!). If the wing's aspect ratio is low, this is manageable, but with higher aspect ratios the wing needs to be stiffened.
To answer your question directly: In the majority of cases it is aerodynamically advantageous to vary sweep with relative thickness, but it is not structurally and economically efficient. In the delta wings of combat aircraft which need to operate over a wide range of angles of attack, de-sweeping the outer wing produces indeed the best wing planform, even when economics is included.