There is more than just the moment of inertia involved in the roll response, and the response itself is more than just one figure.
Let's consider something like a swing-wing airplane, where nothing changes except for the sweep angle. Furthermore, we'll assume standard ailerons control, which rarely happens on the actual swing-wing aircraft.
The difference in inertia will be noticeable but not that great: most of the mass is still in the fuselage. (Especially this applies to real swing-wings, where it is difficult to carry the fuel and ordinance under the wing). That said, the contribution of the wingtips will reduce as a square of the reduction of the wingspan.
But there are also forces involved. The most significant ones will be the ailerons (control) moment and the roll damping.
If we have ailerons near the wing tips, the moment from them will reduce linearly with the wingspan. At the same time, roll damping depends on the square of wingspan and so will reduce more. Damping reduction is the most drastic effect: the wing is almost exclusively responsible for it. This fundamentally differs airplanes with the proverbial ice-skaters and ballerinas.
Overall, in a typical case, with a longer wingspan you may get faster initial response (ignoring aeroelastic effects, which may reverse the trend even here), but significantly slower sustained roll rate.
Aircraft with short stubby wings may get in trouble due to their fast roll rate, even if (and partly because) their ailerons do not seem that efficient: see inertia coupling.