Your instincts are correct, or at least the NTSB agrees with you - they don't like the idea of "Lap Children" on aircraft, and they've come out and said as much in this Safety Alert.
They reiterated that position again in a 2010 letter to the FAA which includes a number of incidents from NTSB investiations, including:
… on July 19, 1989, United Airlines flight 232 crashed during an attempted emergency landing at Sioux City, Iowa, after the fragmentation and separation of an engine fan disk. Of the 296 airplane occupants, 111 were killed, 47 received serious injuries, 125 received minor injuries, and 13 were not injured. Four children, ages 11 to 26 months,17 were aboard the airplane and were being held by adults. A 23-month-old child was killed, and the other three children received minor injuries. The parents of the four lap-held children were instructed to place their children on the cabin floor and hold them in that position while the adults assumed the protective brace position. After the accident, three of the parents reported that they were unable to hold onto their children during the accident sequence.
… USAir flight 1016 in Charlotte, North Carolina. The flight crew decided to continue an approach into severe convective activity and was executing a missed approach when the airplane collided with trees and a private residence near the airport. Of the 57 airplane occupants, 37 were killed, 16 received serious injuries, and 4 received minor injuries. Among those occupants who were killed was a 9-month-old child who was held by her mother on her lap. The child’s mother survived the accident but was unable to hold onto her child during the impact sequence, and the child struck several seats. The NTSB believed that the child might not have sustained fatal injuries if she had been properly restrained in a child restraint system.
Granted both of these were pretty severe accidents, but the NTSB investigators found that the lack of restraints was potentially a major contributing factor to the injuries of children on these flights.
In that same letter the NTSB also notes that unrestrained children are not just a hazard to themselves in a crash:
Passengers are required to securely stow all carry-on baggage during takeoff and landing because of the potential risk of injury to other passengers in the event of an unexpected hazardous encounter. However, passengers are permitted to hold a child of equal size and weight in their lap. When children under 2 years of age are not required to be restrained for their own safety, the safety of their fellow passengers also becomes an issue.
See here for more NTSB pronouncements on the subject of child safety in transportation.
They even have some excellent videos and simulations showing restrained versus unrestrained children.