To my understanding, the input from the MAN START switch (a ground/open discrete into the Engine Interface Unit), is only taken into account by the FADEC logic if the engine is not already running. Once the engine control logic goes into running mode (i.e. N2 rises above idle), it will stay in that mode until the engine master lever (EML) is set to Off, regardless of whether the engine is lit or not.
In other words, once the engine has started, the position of the MAN START p/b is irrelevant, even if the engine flames out (however, note that the FCOM procedures for engine failure often call for the EML to be set to Off, at which point the position of the MAN START switch comes to play in any subsequent attempt to relight the engine).
One risk I can think of is that if you are on the ground, shut the engines down, or even power off the aircraft, then when you power everything back on and attempt to start an engine (i.e. turn the engine start selector to IGN), the autostart procedure will be inhibited by the MAN START switch being already on, and the FADEC will go into manual start mode instead. If you then immediately set the EML to On (which wouldn't be a problem during a normal autostart), fuel would start to flow right away, while the engine may still be below the required minimum N2 for fuel admission (16%), therefore the engine could fail to start or worse (an engine fire).