There is no regulatory requirement for oxygen to be available for anyone on board an aircraft so long as it remains at altitudes of 12,500' MSL or less. And only the required minimum flight crew is required to have oxygen if they are flying above 12,500' up to and including 14,000 MSL' IF they will be at those altitudes for more than 30 minutes. CFR 91.211 If they were going above 15,000' MSL that would be another story.
So the short of it is, No, that pilot was not irresponsible, perhaps, maybe slightly inconsiderate, but not irresponsible.
The reason for this is the lower density of the air isn't low enough to harm the majority of passengers. Except for extreme cases, such as those with severe and uncontrolled asthma or heavy smokers, the worst thing that would happen would be the passengers would just get sleepy and fall asleep. This is natural, and because the body does not need as much oxygen when asleep. For passengers there generally won't be any harm done from a mild case altitude induced hypoxic hypoxia.
And YES! You can ask someone if they will be 'OK' at higher altitudes! Just as it is not only 'OK' but required to ask their weight for weight and balance calculations. (And quite frankly I believe most people find that more intrusive.)
In fact you should take 5 minutes to tell your passengers that the flight will be in excess of, say 11,000', that way they can be prepared for some unusual effects and not have to worry about them. To take it a step further, remember you are the Pilot in Command and you should be aware of your passenger's limitations as that may trigger any self imposed safety limitations you have for the safety of the flight.
Now, if you just want passengers to be comfortable and/or they absolutely insist on not taking a little nap, then it is entirely up to you if you want to carry a portable aviation oxygen canister with you for your passengers.
For further information on hypoxia I would recommend the FAA's page on it and the differing effects on pilots, and by extension passengers.
Hope this helps!