Yes, an ultralight aircraft operated under 14 CFR 103 can be operated within the "Mode C Veil" within the limitations of §103.
As you note, 14 CFR 91—including §91.215, ATC transponder and altitude reporting equipment and use—does not apply to ultralights governed by §103.
The limitations of §103 do generally prohibit: operations over congested areas; operations in certain types of airspace; operations in certain types of airspace.
§103.15 Operations over congested areas.
No person may operate an ultralight vehicle over any congested area of a city, town, or settlement, or over any open air assembly of persons.
§103.17 Operations in certain airspace.
No person may operate an ultralight vehicle within Class A, Class B, Class C, or Class D airspace or within the lateral boundaries of the surface area of Class E airspace designated for an airport unless that person has prior authorization from the ATC facility having jurisdiction over that airspace.
§103.19 Operations in prohibited or restricted areas.
No person may operate an ultralight vehicle in prohibited or restricted areas unless that person has permission from the using or controlling agency, as appropriate.
Let's look at an example, the MSP Class B area:
A, near A R S Sports Strip (7Y7), would be an area where operations under §103 would be allowable. It is not a congested area, and is not a type of airspace for which operations under §103 are prohibited.
D, however, would be areas where operations under §103 are generally prohibited. Point
B is arguably over congested areas, and by that determination would therefore be prohibited under §103.15. Points
D are in Class D and B airspace, respectively, and are therefore prohibited under §103.17. The exception would be where the operation has received prior authorization as specified in §103.17 above.
For what it is worth, I have personally witnessed ultralight type aircraft flying in the area of Point