Please tell your instructor from myself, another flight instructor, that he is, to put it delicately, completely full of sh*t.
Here's the thing about flying an aircraft. It encompasses a wide range of skills and abilities which will highlight your strengths and weaknesses as a human being. A student pilot may be quite skilled with flying, say, steep turns, but then struggle with ground reference maneuvers or instrument flight. What you have to do is simply persevere; if you are weak on a certain skill, you need to practice it more until you are good at it.
And some people are naturally gifted at certain things and lousy at other ones. Itzhak Perlman is gifted at playing a violin, Tom Brady is a gifted football quarterback, and Bob Hoover and Chuck Yeager were (are!) gifted pilots. Just because you can't play a violin or chuck a pigskin or fly an airplane like those guys doesn't mean you can do any of those activities for pleasure or for profit. You might even be quite good at them, too. But some are more naturally talented than others.
Second as my scathing remark at the beginning of this post might suggest I have a major problem with a lot of people out there doing flight instruction and the general state of flight instruction today. Most are just building hours until they can get a better paying pilot job or are terrible teachers with little interest in their students. This hurts aviation tremendously.
Keep in mind as well that the fact that someone was once a military pilot does not mean a high degree of skill or proficiency. There are plenty of mediocre or unskilled military pilots out there. Even some fighter pilots are lousy aviators. Its true.
There is also a tendency among some - often ex-military aviators - to take a 'holier than thou' attitude towards civilian pilots and instruct in a manner similar to the mean neighborhood kid who wanted you to feel intimidated when trying out his new 10 speed bicycle. My general opinion of such people is with poor regard; he's abusing students to stroke his own ego. If I have a CFI working for me and find out he's doing that kind of crap with a student, he'll be fired faster than you can say 'pink slip'. Scroll back ten or so years to their training in the military, they were the same people in a T-6 or T-38 crying like a little punk while some hard nosed IP berated them for every mistake they made. They never learn that this sort of crap is a mark of a poor educator, let alone a human being.
All that aside, I'd have to see what it is that you are struggling with to learn. There are only three cases where I would terminate training a student; struggling to learn some skills isn't one of them. If you have the time and commitment to learning to fly, you can do it. It's that simple.
And like any other pilot certificate or rating you get, these are just journeyman's licenses to practice airmanship. Flying takes a lifetime to master; there are people with tens of thousands of hours of flight time who go up and learn something new on each flight.
Don't quit, don't suck, don't be an a**hole. And fly good.