Yes, you can define your own waypoints. When we use the waypoint identifiers that are published ATC knows where the waypoint is. That's because we're using the same database they are which contains the Lat/Lon location for each of the waypoints in the database.
When we create our own waypoints, ATC doesn't know anything about them. So we just have to give them the waypoint in a form they can interpret.
Almost all FMS provide the capability for pilot-defined waypoints. They can be entered in the form of Lat/Lon, Place/Bearing/Distance (PBD), or Place/Bearing-Place/Bearing (PBX). Once created, the system assigns a short ID to be used in the flight plan leg list. Usually something starting with LLxx, PBxx, PXxx where xx is a 2 digit number. Some GPS navigators just have one pilot-defined list of WPTxx.
While these identifiers work well in your FMS or GPS, they don't help ATC. You have to give ATC the actual position reference in the Route section of your flight plan. The most common formats would be:
Latitude/Longitude can be entered in either decimal deg or in deg min format. A decimal point is used for decimal degrees. An apostrophe is used for deg min. Use NSEW to identify directions. They can be either prefix or suffix. Be consistent. For example:
N28.52W83.68 or 83.68W28.52N or N28'51'W83'40'
Place Bearing Distance and Place-Bearing/Place-Bearing waypoints start with a known database waypoint - typically a VOR - but any fix in the database will work. Examples:
ATL/162/18 to identify Atlanta VOR, 162 deg, 18 NM.
ORL70/MLB348 is the intersection of the Orlando VOR radial 070 and Melbourne VOR radial 348.