There are probably several major effects.
1) Piloting is a "bikini inspector" job, one that people will do because they enjoy it and the social status it may convey. If you have a passion for a profession, you will work in it for less.
After all, people pay to pilot planes as a sport, so some percentage of pilots will work for relatively little pay just so they can fly. Those people making sub-optimum strictly economic decisions, pull down salaries in the field. You see that in other fields as well, such as Chefs.
2) Deregulation and globalization: From the early 1950s through the mid-1980s, airlines were run as government monopolies with government price fixing and protection from competition. Flying was a relatively high-margin business that catered to those flying on corporate money or who were individually wealthy ("the jet set.") People who chauffeur the rich tend to be better compensated.
The deregulation and opening of air travel to international markets drove down all costa as customers sought the best value. It has happened in every deregulated industry. Back when AT&T was a national government corporate monopoly, everyone, even down to the janitors, got above-market wages, unionized or not. After the monopoly broke up, wages went down.
In the past, airlines were semi-militarized and I know for some period back in the 50s and 60s, you had to be a US citizen to pilot commercial aircraft in the US.
3) As a field matures, its wages drop. Most technologies have a sideways-s growth curb where they start slow, have two to three decades of explosive growth and then plateau off. In the growth era, wages are high and then they regress relative to the general wage. In cars, it was 1920s-1960s; in computers, it was 1980s-now; in aviation, the explosive growth was 1930s-1970s. (WWII threw things off some.)
4) People who can be pilots are not as relatively rare as they used to be. My grandfather was a mail pilot in the 1930s and WWII. Pilots back then had to be physically strong, immune to fatigue and able to do a lot of math on the fly (so to speak.) They were very extraordinary people and were often paid accordingly (e.g. clipper plane pilots made ridiculous amounts of money in the 1930s).
As planes gained larger crews and have become progressively automated, the potential talent pool in the general population has expanded. It's not just that more people can become pilots, it's that existing pilots can fly longer. In the 1950s, you didn't see 60 year old pilots. Now you do.
It's easier to become a pilot these days. When I was in my teens circa 1980, I wanted to get a civil aviation license, but couldn't because I was blind in one eye. Today, I can. An expanded talent pool decreases prices.
5) Pilots and aircrew in general are not as key as they once were. Now they are part of a larger team, many of whom stay on the ground.
I would also speculate that pilots just aren't as skilled, relative to the rest of the population, as they used to be. It's not just that the job is more automated and team driven, but that other jobs have reason to and have surpassed the technical complexity of piloting. Compared to other professions out there now, pilots are not the elites they once were. Their pay has regressed accordingly.
If there is a "pilot" shortage, its likely that 1) airlines can't find pilots of a certain rating or pilots who can fulfill some regulation of training or flight time or 2) the margins in the industry are so slim that the industry has a very narrow compensation window.
I suspect its the latter. Flying is commodity now. People fly like they used to take the bus 30+ years ago. If you're working the same economic niche as a bus driver, well, you will get paid like one.
lots of pilots + limited positions = low wages
$\endgroup$