Short answer: the FAA's UAV regulations probably stop applying at 3 nautical miles, but this is more about maritime law than aviation regulations and you might get a better answer on law.SE.
The FAA's UAV regulations do state where they apply. 14 CFR 107.1(a) (emphasis mine):
Except as provided in paragraph (b) of this section, this part applies
to the registration, airman certification, and operation of civil
small unmanned aircraft systems within the United States
14 CFR 1.1:
United States, in a geographical sense, means (1) the States, the District of Columbia, Puerto Rico, and the possessions, including the
territorial waters, and (2) the airspace of those areas
Defining different types of marine jurisdiction is apparently so complicated that it requires a diagram (33 CFR 2.1), and the definition of "territorial sea" is in 33 CFR 2.22. Unfortunately, there are two possible definitions: 12 nautical miles or 3 nautical miles from shore. My reading of it is that for treaties, criminal law and other 'big stuff', the waters extend 12 miles. For more general purposes, it's 3 miles per 2.22(a)(2):
Unless otherwise specified in paragraph (a)(1) of this section,
territorial sea means the waters, 3 nautical miles wide, adjacent to
the coast of the United States and seaward of the territorial sea
Having said all that, I don't know for sure what the definition of "US territorial waters" really is. The 3NM distance seems very likely (to me) for part 107, and you might get a better answer on law.SE.