To my understanding, the input from the MAN START p/b (a ground/open discrete into the Engine Interface Unit), is only taken into account by the FADEC logic if the engine is not already running (engine running = N2 above idle). Once the engine goes into running mode, the only way to come out of it (from the FADEC's point of view) is to set the engine master lever (EML) to Off, so it would appear that even in case of flame-out, the position of the MAN START p/b is irrelevant (however, note that the FCOM procedures for engine failure often call for the EML to be set to Off).
One risk I can think of is that if you are on the ground, shut the engine down, or even power off the aircraft, then when you power everything back on and attempt to start the engine (i.e. turn the engine start selector to IGN), the autostart procedure will be inhibited by the MAN START p/b being already on, and the FADEC will go into manual start mode instead. Engine relight in flight could also be affected, I suppose, but I would have to look at it more in detail.