Your description sounds like the diagram below, so I'll proceed based on that.
When a controller gives you a landing clearance while you are a ways out with no intermediate instructions, it's usually because there is nobody else around the controller needs to coordinate and you are being given discretion to proceed by the most direct logical routing to get to the threshold of Rwy 5.
Although there is a published "left" traffic pattern, and normally you would follow the usual protocols to get established for the left pattern for Rwy 5, the landing clearance at that stage was granting you discretion to use a right pattern if it was more direct and that's what the controller expected. When you crossed the field to get to the "proper" side (blue line below), this would get the controller wondering what you're up to as you appear to be just flying off to the northwest until you turn downwind, especially if you are in the habit of using wide patterns.
I would have proceeded direct to a right downwind or even a right base leg (red or black line below) if I was already line up for that, and would have mentioned that in the readback for the landing clearance, such as "Cleared to land runway 5, proceeding direct right base", just to let the controller know my intentions, and if there was a misunderstanding, I'd find out about it then.
There is a contributor here who is an air traffic controller, so hopefully he/she will chime in.