It has to do with the Birnbaum distribution. The center of its area is at one quarter.
In potential flow theory, lift can be calculated as the linear superposition of a contribution from camber and one from angle of attack. While the camber-related part of lift is constant, the angle-of-attack related part varies linearly with this parameter. This means that changing the angle of attack adds or subtracts circulation, the chordwise distribution of which is described by the Birnbaum distribution. The important part is the self-similarity of the angle of attack dependent lift resulting from this addition or subtraction of the same chordwise distribution.
The center of pressure of the resulting angle-of-attack dependent part is constant and at 25% of chord for 2D flow and wings of large aspect ratio, at least as long as flow remains attached and viscous effects are negligible.
The pictures below (own work) show the circulation over chord produced by camber alone (so AoA is 0°) on the left and the Birnbaum distribution alone on the right (so using a symmetric airfoil). Total circulation is simply the sum of both. Copied over from this answer thanks to the helpful suggestion of ROIMaison.
ultimately the aerodynamic center is determined by how the distribution of lift changes with angle of attack
That is precisely right: The aerodynamic center (a better word for it is "neutral point", btw.) is the point where the additional lift force from a change in angle of attack can be summed up. Do not look at the lift distribution per se (that is what determines the center of pressure), but at its change over angle of attack, because that is what determines the neutral point and what shifts the center of pressure back and forth. The constant part caused by camber does not change with angle of attack and the part caused by angle of attack has its center always at 25% (under the conditions mentioned above) and its magnitude grows with angle of attack. This growing contribution of the angle-of-attack-dependent force is what shifts the center of pressure forward (on airfoils with positive camber) rsp. backwards (on airfoils with negative camber) towards the 25% point.
BTW, I wish we had more people like Aaron Swartz. That article is now 97 years old and these monopolists still want 42 € from me for a copy. Sorry, no details from it – I could not find any online.