Short answer:
- This is very normal because the outer wing is thinner.
- The warm fuel (which is warmed by the IDG) is not always returned to outer/inner tanks.
Some explanation is needed wrt the circulation function on the A320. Excerpts below will be denoted (M) and (F), for maintenance and flight manuals, respectively.
The FADEC controls the fuel return valve.
If the outer tank is already full, the fuel overflows to the inner tank through a spill pipe. On ground, the fuel recirculation is not inhibited if there is an overflow in the surge tanks. (F)
The main takeaway from this is the first line. The fuel isn't always returned. The return is managed by the engine's FADEC, not the fuel system. The second thing is that the circulation also works on ground.
The fuel system:
- Stores fuel in the tanks.
- Supplies fuel, in the correct quantities, to the fuel tanks during refueling.
- Supplies fuel to the engines and the Auxiliary Power Unit (APU).
- Circulates fuel to cool the Integrated Drive Generator (IDG).
- Keeps fuel in the outer tanks for wing bending and flutter relief. (F)
The function of the circulation is cooling the IDG, not warming the fuel. But the fuel is indeed warmed due to the process, but the engine fuel system mixes that warm fuel (inside the fuel IDG cooling system) with fresh supply of cold fuel, to be used again by the engine (combustion) and for the IDG cooling.
- In other words, the engine burns the same fuel that was used for cooling the IDG (not necessary to return it to the tank).
The FADEC only returns the fuel to the outer tank in 1 condition, and it does that by opening the FRV (fuel return valve). That condition:
The return valve opens at low engine fuel flow if the engine oil temperature is above 93°C (199°F). (M)
Low engine FF (e.g., idle) will not be encountered in cruise. Because of the low FF, there is no high demand for the warm fuel (inside the fuel IDG cooling system), i.e., the engine is not using it quickly for combustion. And it will only keep getting warmer. So the FADEC mixes it with fresh fuel, and reintroduces it to the outer tank (schematic below). The mixing ensures it will be reintroduced at less than 100°C.
- In other words, when the engine is not burning enough fuel, the fuel inside the IDG cooling system is returned to the tank, because it will keep getting hotter since it is not being consumed at a fast enough rate.
The return valve mixes the hot fuel with cold fuel from the LP pump to maintain the temperature of the returned fuel below 100°C (212°F). (M)
This means, for example, if the plane is sitting on the ground with the engines running for too long, the outer tanks will become warmer than the inner tanks.
In cruise, because the outer wing is thinner, the surface area to volume ratio is higher than the inner wing, and thus the fuel will lose temperature faster than the inner tanks. This is highlighted in Airbus' FAST magazine issue 36:

As you can see, the outer tanks cool down at a faster rate. Note: the magazine discusses the long-range operation of the A340, which unlike the A320, does have a fuel transfer procedure when a LO TEMP
message is shown.
And below is the A320 IDG cooling schematic. Close the FRV, and you'll see how the fuel is mixed and reused by the engine:

The above explains why when the crew gets a confirmed FUEL L (R) OUTER (INNER) TK HI TEMP
warning, the actions could be:
- Turning off the galley (reduces load on IDG).
- Delaying takeoff and shuting down affected engine (side).
- Disabling auto thrust and increasing FF of affected engine (side).
- Using APU IDG instead of engine IDG (taking it offline).
(Source: A320 QRH)
All this would reduce the hot fuel being fed back into the affected tank.
A slightly more detailed operation of the FRV is found in the engine technical training manual (CFM56):
The purpose of the Fuel Return Valve (FRV) is to bypass warm fuel from the Hydromechanical Unit (HMU) to maintain the efficiency of the Integrated Drive Generator (IDG) Oil Cooler and engine oil/fuel heat exchanger.
Consequently this prevents engine and IDG oil over temperature.
FRV logic: the FRV is fuel pressure operated, and electrically controlled by the Electronic Control Unit (ECU).
The ECU control logic of the FRV is mainly based on the engine oil temperature.
Above a certain engine oil temperature, the ECU orders a low fuel flow return to the A/C fuel tanks.
When the engine oil temperature increases, the ECU orders a high fuel flow return to the A/C fuel tanks.
The two return fuel flow levels are 500 kg/h and 1000 kg/h, or 1100 lb/h and 2200 lb/h. The hot fuel is mixed with the cold fuel to limit its temperature, before it is returned to the A/C fuel tanks.