In 14 CFR 1.1, we find two definitions:
Civil aircraft means aircraft other than public aircraft.
Public aircraft means any of the following aircraft when not being used for a commercial purpose or to carry an individual other than a crewmember or qualified non-crewmenber:
(1) An aircraft used only for the United States Government; an aircraft owned by the Government and operated by any person for purposes related to crew training, equipment development, or demonstration; an aircraft owned and operated by the government of a State, the District of Columbia, or a territory or possession of the United States or a political subdivision of one of these governments; or an aircraft exclusively leased for at least 90 continuous days by the government of a State, the District of Columbia, or a territory or possession of the United States or a political subdivision of one of these governments. …
(2) An aircraft owned or operated by the armed forces or chartered to provide transportation to the armed forces if - …
(3) An aircraft owned or operated by the National Guard of a State, the District of Columbia, or any territory or possession of the United States, and that meets the criteria of paragraph (2) of this definition, qualifies as a public aircraft only to the extent that it is operated under the direct control of the Department of Defense.
The most visible example of a “public aircraft”, beyond military and diplomatic flights, would probably be police helicopters.
As to the “exemption”, if you look through 14 CFR, you will find many sections and even some entire Parts that only apply to “civil aircraft”.
For a trivial example, 91.105(b) only requires pilots of civil aircraft to wear shoulder harnesses during takeoff and landing, ergo pilots of public aircraft do not—though they probably do so anyway unless there’s a mission-related reason not to.
That’s the rub: Public aircraft voluntarily follow most of the rules, most of the time. They just know that when they have a compelling reason to break the rules, they can’t be hassled by the FAA for it.