Sometimes, prevailing winds will account for the difference -- flights going west to east in the US will generally be scheduled for less time than flights going the opposite direction.
The other major factor that can play is in taxi times. If the delays waiting to get in to a gate matched the times waiting to depart at each airport, these delays would affect both legs equally, but that's not generally the case. Most of the time, at a congested airport like Atlanta, you can generally get into the gate fairly quickly, but the delays getting airborne can be far longer.
Thus, a flight scheduled from Podunk into ATL might be scheduled for 10 minutes taxi from the gate to departure at Podunk, the flight time, and another 10 or 15 minutes to taxi in, but in the other direction they might build in 30-45 minutes of taxi time departing ATL, plus the flight time, plus 5 minutes landing until blocking in at Podunk. Thus the leg into ATL is scheduled for flight time plus 25 minutes, while the leg from ATL is scheduled for flight time plus 35-50 minutes.
If it's winter time and your flight was going from BOS or PHL to MCO, another factor in addition to the potential for massive departure delays would be the need to build in time for deicing -- done before departure in PHL or BOS, but not so much leaving Orlando.
Of course, if you didn't need to deice "today," then you'd probably beat the scheduled time by a good bit. Likewise if the weather was clear and the airport had no delays. But the mega-hub airports run so close to capacity that any little hiccup in the weather or otherwise causes delays to start building, and then you're using all of your scheduled time, and sometimes more.