It may be helpful to remember the ideal gas law here: $PV=Nk_BT$ (the physics version). If we divide both sides by $N$ we have a $V/N$ on the left which is just a $1/\rho$ where $\rho$ is the number density. Thus:
$\frac{P}{\rho}=k_BT$
In an ideal gas, to which air is close enough, the speed of sound is given by:
$c_s = \sqrt{\gamma \frac{P}{\rho}}$
Throwing in our first results, we see that:
$c_s = \sqrt{\gamma \frac{P}{\rho}} = \sqrt{\gamma k_BT}$
In other words, the speed of sound reduces with either pressure or temperature because those two are in turn connected by the ideal gas law. It's basically two different ways of looking at the same question.
Side note: throwing in the effect of density is another question all together since $c_s$ increases with an increase in temperature or pressure but decreases with an increase in density. The question then becomes whether the decrease due to the drop in pressure overcomes the increase due to the drop in density at higher altitude.