You're actually closer than you think: it's not where the temperature is adjusted, it's where the pressure is adjusted.
The reason is that colder air above a ground pressure has a steeper pressure loss gradient with altitude than warmer air. Because of its greater density near the ground, it needs less volume to make the same weight (pounds per square inch =pressure).
The column of air above a warmer place (at the same ground pressure) will be taller and have higher pressure at altitude.
Look at weather. Cold fronts roll forward on the ground due to greater density, warm fronts ride up over cold air at altitude due to greater temperature.
To further understand this phenomenon one can compare pressure change differences to temperature change differences in passing fronts. Pressure change differences occur gradually over many miles, whereas temperature change differences occur in a frontal passage over only a few miles.
Using the Gas Law P is proportional to Density and Temperature (Kelvin scale), one can see, with a passing front dropping temperature 30 degrees$^1$, we have (273+30)/273 = an 11% difference in temperature at a similar local pressure$^2$.
So, local pressure gradient "stacking" with altitude, as confusing as it is, must be accounted for, as the altimeter only reads pressure.
$^1$ Celsius and Kelvin units are the same, 273 Kelvin = 0 Celsius, 303 Kelvin = 30 Celsius
$^2$ Even a hurricane pressure gradient can be as little as 960 mb/942mb over 100 miles, or around 1 percent!
What we need to know as pilots is the:
altimeter error relative to the ground pressure reported at a take-off/landing site based on deviation +/- from standard temperature
altimeter error relative to passing through a warm air mass into a cold air mass from temperature change and
altimeter corrections based on changes in reported ground pressure as we fly deeper into the cold air mass (they're bound to increase).
More reading here.