There are basically two categories of aircraft with canards:
- modern European fighter jets (Grippen, Eurofighter, Rafale), and
- many of Burt Rutan's designs
The fighter jets have delta wings because they are most appropriate for fast flight in the high transsonic and slow supersonic region (as these jets have supercruise capability). And the reason they have canards is that they are more efficient for pitch-unstable designs (aerodynamically unstable with computer stabilization provides faster control response), and double as vortex generators at high angle of attack, improving slow speed or high-G handling.
For Burt Rutan, well, he always liked doing thing differently than everybody else. Even if it usually was not the most efficient way, but he was good enough to make it work. But not all of his canard designs actually have swept wings.
The best known of his designs, the VariViggen/VariEze/Long-EZ line, uses swept wings. Here the reason is that he moved the vertical stabilizers to the wing-tips to make room for the pusher prop and avoid the weight of additional booms. And because it was inspired by the Viggen.
But he also made other canard designs like the model 77 Solitaire that has straight wings. And if you go through the list of his designs, you'll see most are straight wing, often either with canard or tandem wing (the surfaces are of comparable size and contribute similar amount of lift).
Regarding the stability: swept wings increase lateral stability (yaw-to-roll coupling), and the thing with that is that you don't actually want as much as you can get in a design, you want just the right amount. Otherwise you'll get strong tendency for Dutch roll. The stability can be adjusted most easily by changing the dihedral, so simply if the aircraft has swept wings, it ends up with less dihedral or even anhedral (as in most high-swept-wing designs as high-wing configuration also increases the coupling).
There is no reason to use swept wings because of canards, they don't affect lateral stability much. It is just that for the fighter jets it happens to be a good design point and for the VariEze it was part aesthetics and part other reasons.