First off, it is VERY important to note that you DON'T fly a Drone. RC Model Aircraft are flown by hobbiests, and Drones are flown by the government. It is enumbant on all of us R/C modelers to enforce that at all times, lest the government decide that all R/C hobbiest model aircraft be banned.
(I am making an assumption that you aren't doing this as a member of the armed forces, or working for the government, also my comments are US oriented.)
FAA Modernization and Reform Act of 2012
Senate Bill, Section 607(g) …exempts most model airplanes used for recreational or academic use from any UAS regulations established by the FAA
Conference Committee Report
Senate bill with modifications… Language including model aircraft for the purposes of sports, competitions and academic purposes is removed and replaced with "hobby". The modified section includes language requiring that the model aircraft must be operated in a manner that does not interfere with and gives way, to all manned aircraft. In addition, language that requires that model aircraft flown within five miles of an airport will give prior notification to the airport and the air traffic control (ATC), and that model aircraft that are flown consistently within five miles of the ATC will do so under standing agreements with the airports and ATC. Lastly, language is added that will ensure that nothing in this provision will interfere with the Administrator's authority to pursue enforcement action against persons operating model aircraft who endanger the safety of the national airspace system. In this section the term "nationwide community-based organization" is intended to mean a membership based association that represents the aeromodeling community within the United States; provides its members a comprehensive set of safety guidelines that underscores safe aeromodeling operations within the National Airspace System and the protection and safety of the general public on the ground; develops and maintains mutually supportive programming with educational institutions, government entities and other aviation associations; and acts as a liaison with government agencies as an advocate for its members.
SEC. 336. SPECIAL RULE FOR MODEL AIRCRAFT.
(a) In General. --Notwithstanding any other provision of law relating to the incorporation of unmanned aircraft systems into Federal Aviation Administration plans and policies, including this subtitle, the Administrator of the Federal Aviation Administration may not promulgate any rule or regulation regarding a model aircraft, or an aircraft being developed as a model aircraft, if--
(1) the aircraft is flown strictly for hobby or recreational use;
(2) the aircraft is operated in accordance with a community-based set of safety guidelines and within the programming of a nationwide community-based organization;
(3) the aircraft is limited to not more than 55 pounds unless otherwise certified through a design, construction, inspection, flight test, and operational safety program administered by a community-based organization;
(4) the aircraft is operated in a manner that does not interfere with and gives way to any manned aircraft; and
(5) when flown within 5 miles of an airport, the operator of the aircraft provides the airport operator and the airport air traffic control tower (when an air traffic facility is located at the airport) with prior notice of the operation (model aircraft operators flying from a permanent location within 5 miles of an airport should establish a mutually-agreed upon operating procedure with the airport operator and the airport air traffic control tower (when an air traffic facility is located at the airport)).
(b) Statutory Construction. --Nothing in this section shall be construed to limit the authority of the Administrator to pursue enforcement action against persons operating model aircraft who endanger the safety of the national airspace system.
(c) Model Aircraft Defined. --In this section, the term "model aircraft" means an unmanned aircraft that is--
(1) capable of sustained flight in the atmosphere;
(2) flown within visual line of sight of the person operating the aircraft; and
(3) flown for hobby or recreational purposes.
When this law is passed, you'll need to contact the airport tower/operator if you intend on flying your model aircraft within 5 miles of the airport.
If, at the time of this answer, the law has not yet passed, the rules prior were to keep your model aircraft under 400ft AGL within 3 miles of an airport. Flying your model aircraft over the boundary fence OF an airport is considered tresspassing and an incursion.