There are two factors which limit the possible speed of piston engines:
- Flame speed in the fuel-air mixture, and
- relative speed of moving parts, here the pistons in their cylinder barrels.
If the engine runs too fast, the flame front originating from the spark plug will not have traveled far enough to have burnt most of the fuel by the time the piston moves down again. This puts a fundamental limit to the growth in power output over speed in piston engines.
Also, if the relative speed between moving parts is too high, the lubrication will fail and the parts will overheat quickly. In order to run at the highest possible speed, rotary engines used castor oil, which in turn motivated early aviators to wear long scarves over their mouth so their digestive system would not be compromised by sitting right in the exhaust stream of their engine. Today, lubricants have improved but still set a clear limit to the maximum speed at which a piston engine can be run. Cooling can be managed by pressurizing the cooling system and improving radiator and pump performance while inertial loads on the engine components can be dealt with by lightweight, high-strength materials.
Reducing the bore will allow a higher speed before the ignition limit bites while reducing the stroke will reduce piston speeds, again allowing a higher engine speed before lubrication fails. An optimimized engine will balance both so that the possible speed can be fully exploited. For the IO-360 and IO-540, those dimensions are 5⅛" and 4⅜", respectively. This places them below the last generation of big aviation pistons and closer to the typical interwar engines.
I could not find reliable data on the engine speed of Reno Air Race winners, which should come close to the maximum sensible engine speed at that size class. The best I could find was the 3,700 RPM of the Jumo 213 E, an engine with 5.9" bore and 6.5" stroke, so I would guess by using good fuel and lubrication, the Lycoming might end up somewhere between 4000 and 4500 RPM. I'm sceptical that much useful power can be extracted at 5000 RPM and am sure that the lifetime at this speed will be measured in minutes, not hours.