Indoor model helicopters would not be able to fly if the answer were no.
Of course it can, but the enclosure must be big or small enough to avoid a toroidal vortex from developing around the rotor. To summarize my answer, three conditions make it possible:
- A very big enclosure which allows enough forward speed so the helicopter never flies in its own wake,
- a very low ceiling or
- a very narrow enclosure.
Case 1 is trivial, so I dwell here on case 2: If you have flown an indoor model helicopter, you probably know that it is not advisable to come too close to the ceiling: The helicopter will become unstable and will be sucked right into it. This is caused by the ground effect (or better ceiling effect in this case): The efficiency of the blades increases the closer they are to a horizontal surface and the helicopter needs less torque for the same lift. Once the rotor head is touching the ceiling, you need to cut power almost to zero to free it again. The same effect can now be used to fly in the constrained space because it will prevent the toroidal vortex of the vortex ring state from developing.
Now the explanation to case 3: If you insist on not touching any surface, the vortex will soon develop and reduce lift. How long you can hover depends on the air volume, and there is a minimum when the diameter of the enclosure is approximately twice the rotor diameter. Once the diameter becomes smaller, the vortex will again be inhibited, and when the enclosure is small enough the rotor will look like a fan in a duct. Now again less energy is needed for lift - just enough to maintain the necessary pressure difference through the rotor disc.
I should add that this solution will best work with a co-axial rotor and no tail sticking out at one end.