You aren't crazy. People have started later than you and been successful. However, age does bring additional challenges, which one could argue is an ageist bias of sorts.
The first is age itself. Part 121 carriers have a mandatory retirement age of 65. That puts a hard limit on a potential airline career in terms of seniority and wages. There is no hard limit for other flying jobs, at least for domestic flying. Int'l rules are a bit tricky when you pass 60.
The second is medical. Even if you are in great health for your age, you likely have significant medical history that will need to be documented and reviewed, which can delay getting even the class 3 medical certificate you will need to solo in an airplane and take the PPL test. To fly commercially, you will need to pass a class 2 and, for the airlines, a class 1. And you have to keep passing them every 24, 12 or 6 months for as long as you want to keep flying. As we all know, health inevitably declines with age, so there is a soft limit on how long your career will last.
The third is cognitive decline. While it's not impossible to teach an old dog new tricks, it does take longer. Your training will probably take longer than someone half your age, though your glider experience should mitigate that somewhat.
There is no need for an aviation college. Some airlines still require a bachelor's degree, but they don't care what subject it is in or where it is from (as with most jobs). Other airlines may prefer one but don't require it, and non-airline jobs may not even ask.
It will be difficult to get the seniority you need to get the highest-paying airline jobs; you may well hit mandatory retirement before you reach what you consider a "decent" salary based on your previous career. Other jobs, such as charters, may have higher initial pay but less opportunity for advancement, which might be a better fit for your timeline. Others may have more flexibility, such as part-time instructing or being a ferry pilot. There are a wide variety of jobs to choose from to fit your lifestyle priorities.
Your apparent financial status, though, has one significant advantage: if you can buy your own plane to train in, you can get it done for much less money, and you can probably get it done much faster too, if you can find a CFI with enough free time. Many people who have done this even report selling the plane at a profit when they're done.