The simple answer that covers the majority of engines is that a fan has a shroud. The possible exception are unducted fans or Open Rotor Engines, which are a hybrid between a turboprop and a fan engine.
A more scientific answer is based on the difference in disc loading: How much power per disc area can be pumped into the fan for propulsion. For propellers, this is distinctively less than for fans. Some examples:
- Cessna 172 with O-320 engine (150 hp) and McCauley 1C160 prop (1.9 m diameter): 39.45 kW/m².
- P-51D with Packard-Merlin 1650 (1590 hp) and Hamilton-Standard prop (3.4 m diameter): 130.59 kW/m².
- Lockheed C-130H Hercules with Allison T-56 engine (4590 hp) and Aeroproducts propeller (4.1 m diameter): 259.25 kW/m².
With jets, this comparison needs power when thrust is listed. Therefore, we make the heroic assumption that the speed at the fan is Mach 0.4 in static conditions and thrust is split according to bypass ratio (which is completely imprecise but will do for the purpose here):
- BAe 146 with Lycoming ALF 502 (31 kN static thrust, 5.7:1 bypass ratio and 1.02 m fan diameter): 4254.35 kW/m².
- Boeing 747-200 with Pratt & Whitney JT9D (213 kN static thrust, 5:1 bypass ratio and 2.34 m fan diameter): 5388.74 kW/m².
- Boeing 777 with GE90 (388.8 kN thrust at take-off, 8.4:1 bypass ratio and 3.124 m fan diameter): 6077.23 kW/m².
I guess by now you see where this is heading: Turbofans are simply a different class when thrust per frontal area is concerned. And for this to be possible, three conditions must be met:
- A well-designed intake which produces uniform flow over the engine face. The shroud is only the consequence of providing the fan with an intake.
- A high solidity factor of the fan/propeller (the ratio of the total blade area of the propeller to the disk swept out when the prop turns)
- High dynamic pressure at the engine face, so the engine power can be absorbed by the fan/propeller.
Note that those conditions depend on each other: Without the solidity, much less power could be absorbed. Without the intake, the supersonic flow at the fan tips would produce horrible losses and noise.
Now for the crazy ones. The misfits. The rebels. The troublemakers. The round pegs in the square holes:
Here it is hard to make a clear call whether these are fans or propellers. Their proper name "propfans" hints at that already.
A last word on terminology
Efficiency is a measure how much effort is needed for a certain result. An efficient prop needs less power per unit of thrust produced. It has few blades, turns slowly and has a big diameter. A subsonic fan would be more efficient, but would create much less thrust for a given size.
If you want to express the capacity for thrust creation at a specific diameter, use disk loading, not efficiency.