First, a caution about definitions. Some European countries define the word "ultralight" a little bit differently than the US. In those countries, "ultralight" means something more along the lines of what would be called a "light sport aircraft" in the US. Those countries typically use the term "microlight" for an aircraft that would be called an "ultralight" in the US. The CFRs you cited in your edit are for actual ultralights, but just to be safe, I'm going to answer for both LSAs and ultralights.
Light Sport Aircraft
In the US, an LSA is a one- or two-person aircraft that meets certain requirements. The only difference between an LSA and a normal GA aircraft is that someone who holds a Light Sport Aircraft License can only fly Light Sport Aircraft, while someone with a full PPL can fly both LSAs and not-LSAs.
Since an LSA is a regular aircraft, there are no special restrictions on where or when* it can fly. LSAs have the same radio and transponder requirements as regular aircraft for flying in controlled airspace.
* A Light Sport License prohibits you from flying at night, but that's a limitation of the license, not the aircraft. If an LSA meets the minimum requirements for night flight (lighting, electrical, etc.), a regular pilot could fly it at night.
In the US, an ultralight is a single-person aircraft that meets certain (far more restrictive) requirements. Ultralights have no licensing requirement, so you can fly one regardless of what kind of license you have, or even if you don't have any kind of license at all.
However, you're not allowed to fly an ultralight within class D or higher airspace, class E surface airspace, or over a built-up area**. You can get airspace authorization from the facility controlling the airspace, and a waiver for the built-up area restriction from your local FSDO, but you'd need to ask every single time you wanted to go flying, and it's not guaranteed. Much easier, IMHO, to cart your ultralight out to an empty field or untowered airport and launch from there.
** Read: Cities, towns, etc.