By regulation, a "weight & balance" must be computed before departure to ensure that the weight is within the weight limits for a number of different conditions and that the weight is distributed such that the c.g. (center of gravity) of the airplane is also within the longitudinal c.g. limits for those conditions. That c.g. is typically computed assuming the the passengers are in the seats they have been assigned. If the passengers are seated elsewhere, the computed c.g. is not accurate. In other words, the pilots wouldn't know where the c.g. is or whether it is even within limits.
The takeoff trim setting is affected by the c.g. If the pilots set the trim according to the computed value, and the c.g. is considerably aft of that, on liftoff the nose is going to want to pitch up more than the pilots expect. If the c.g. is so far aft that it is well beyond the aft limits, serious control problems can occur. If the c.g. is far forward of what the pilots are expecting, they're going to have to pull harder on the elevator control than expected to get the aircraft to rotate, and that is going to extend the takeoff roll.
Different makes of aircraft vary widely in c.g. limitations and how tolerant they are limit wise of passenger placement in aircraft that are not full. For example, when I was flying Metroliners, if you had few passengers, you had to ask them to sit toward the rear. I once got a strange look from Rosey Grier (Google him if you don't know who he is) when I had to ask him to sit in the back of the aircraft.
Once you're in the air, the movement of passengers is of lesser concern balance wise. You're at speed, the elevator is fully effective, and the pilots or the autopilot keep the airplane in trim. In other words, the trim is what it is rather than a computed value that might not reflect reality.
Also, on large aircraft, once you're in the air in level flight and if you're so inclined, you can work out your weight by using your fuel flow, pressure altitude, and airspeed. Then with the weight and your trim setting you can come up with your c.g. I would guess that most of the glass cockpit aircraft probably do that for you automatically if you request it. My experience stopped with the 747-200, which is now a near relic.
The c.g. limits are defined conservatively, and usually in terms of the percent of the mean aerodynamic chord. It's not a case of that if the aft c.g. limit is 33% mac (a common value for 747s) and you're at 33.1% you're going to crash. It's just a case that you're beginning to move into an area that gives you a lesser degree of controllability than is deemed necessary for some situations. Continued movement to an extreme value would eventually make the aircraft uncontrollable.
Insofar as the bags-left-behind part of the question, I don't think their absence would cause a problem. If the computed c.g. including the bags was right right up against the aft c.g. limit, their absence could conceivably put the c.g. beyond the aft limit. The pilots might notice it, but handling it would not be difficult. However, aircraft are typically not loaded such that their c.g. is close to a limit. A preferred aiming point for 747-100s, -200s, and -400s is 26.6% mac.
Just for the heck of it, I did a weight & balance for a 747-400, loading it so that the takeoff c.g. was 26.6% mac. I then took 5000 lbs (100 bags x 50 lbs each) out of the most forward baggage area. This changed the the c.g. to 28.0% mac, still within limits. On the 747, a given weight of baggage misplacement has less potential to affect the c.g. than the same weight of passenger misplacement because the passenger seating extends over 20 feet forward of the most forward baggage compartment and over 17 feet aft of the most aft baggage compartment.
