This isn't a complete answer, but the practical answer is that you may not need to contact anyone.
I'm aware of an airport, within the surface-level Class E airspace designated for another airport, where ultralight operations are routinely conducted-- in fact an ultralight airplane manufacturer was long based there. As far as I know, no arrangement for prior authorization has ever existed. Paramotors also routinely launch from nearby fields within the same airspace. This site is on the outskirts of a major city that has a FSDO. An FAA Examiner routinely gives checkrides at this site, and no one I've spoken with (including the Examiner) was aware that there was any requirement for prior authorization for Part 103 operations here. (However, it may help that the field is not the same one for which the surface-level Class E airspace is designated-- rather, it is several miles distant, and is much smaller, with a much lower frequency of operations.)
So yes, you definitely do need to obtain authorization if you want to fully comply with 103.17, but if things are working for you as is, I wouldn't worry about it, especially for low-level operations.
If you do want to seek permission, it seems perhaps that some sort of standing Letter of Agreement permitting low-level operations under certain specified circumstances might work better for you than trying to telephone the ATC facility on a case-by-case basis.
While your specific question makes it clear that you want to take off and land within the airspace in question, and apparently on the grounds of the actual airport for which the airspace is designated, at least one answer has suggested that calling the ATC facility on the radio in flight might be an option in some circumstances where an ultralight pilot wishes to enter Class E airspace that extends down to the surface. I'd suggest that pilots not do that unless the facility has already specifically instructed them to do so. The staff may have no idea that FAR 103.17 even exists, and might have trouble understanding exactly why they are being called. Furthermore, they might not have a clear understanding of the vertical extent of the airspace1 in which FAR 103.17 says ultralights can't operate without prior authorization. The situation seems ripe for confusion.
Footnotes:
- For more, see the related ASE question In reference to airspace, what does the FAA mean by "within the lateral boundaries of"? Is this construed to also imply a vertical limit?