Pondlife pretty much hit the nail on the head.
When a business goes bankrupt or bust, depending on jurisdiction, the assets of the company may not belong to the company anymore, they may belong to creditors. Monarch may not have even owned them in the first place, since some airlines use leases from other airlines or banks to get aircraft.
Even if Monarch did own the aircraft, and they went belly-up meaning the aircraft were basically nobody's, now you have to worry about insurance and liability. The aircraft need to be insured to fly, and there is an assumption of liability if something (like a crash) were to happen. There are logistics like maintenance, repairs, etc. Somebody will have to go through all the aircraft and make sure that they are legal to fly (review logs, maintenance records, etc).
Then you have the crews: they are not employed, and the CAA isn't going to employ a bunch of pilots for a couple of weeks just to run ferry flights. The pilots may demand benefits, etc. It would be difficult or impossible to contract them as private parties (since most insurance companies would not let you do that).
So the easiest option is to let somebody else take care of all that by using charter flights. They have the infrastructure (pilots, planes, fuel contracts, insurance, etc) to get these people out. The CAA just has to call them up and say "we need X number of people picked up from ABC on some-date", and the charter company takes care of it. Of course it is a little more expensive than doing it yourself, but you aren't trying to basically build a short-term airline and then shut it down again.