Paragraph 7–1–2 of the 7110.65 permits a controller to issue a "clearance" to maintain "VFR conditions" if an IFR-flight-plan pilot requests a VFR climb or descent. The paragraph goes on to say that an "alternative clearance" must be issued if there is reason to believe the pilot cannot maintain VFR.
Remember that VFR is not the same as VMC; there are rules pertaining to cloud clearance that you must follow if told to maintain "VFR."
But of course the Note at paragraph 2–1–4 is very clear that "It is solely the pilot's prerogative to cancel an IFR flight plan." 4–2–10 prescribes the phraseology to confirm that a pilot has cancelled: "(Call sign) IFR CANCELLATION RECEIVED." If you did not hear that, you can assume your IFR flight plan is still active. Reading between the lines of the phraseology prescribed at 7–1–2, my interpretation is that an IFR pilot cleared to "maintain VFR conditions" does not receive IFR separation for a period of time1 but may subsequently be instructed to maintain a set altitude and/or heading and would from that point receive IFR separation again without needing to hear "Cleared to [clearance limit] via..." As I read it, the act of requesting a VFR climb or descent does not necessarily imply a request to cancel IFR altogether.
So to sum up: My opinion of the rule is that the controller was correct, you had been VFR since being issued the VFR descent and you were not receiving IFR separation. But you were still on an IFR flight plan, and that plan had not been canceled yet. The situation was probably muddled by your being close to your destination airport.
If I was the controller I would want to hear you say, whether before or after the VFR descent, "N12345 cancel IFR." If I switched you to CTAF before then I would issue "Report cancellation of IFR in the air this frequency or on the ground at [other frequency]."
1Supporting evidence that a pilot cleared for a VFR climb/descent does not receive IFR separation is found at paragraph 13–2–2a8(a)(9) which deals with the "Conflict Probe" tool used to ensure separation in the oceanic environment. The tool does not account for, among other things, separation provided by using the procedures of paragraph 8–8–5, VFR climb/descent. Of course, VFR pilots are separated from other pilots by the "see-and-avoid" rule, which the Conflict Probe cannot anticipate. But note also that the tool still tracks these aircraft, meaning their flight plans are not cancelled altogether.