The Concorde mainly transported fuel across the Atlantic so that enough was available to land safely. Passengers were just an extra on top. Also, being fist-class only, it held fewer passengers than what low budget airlines have taken to cramming into their planes lately.
To look at the structure relative to passengers is a bit misleading. Let's look at the fuel load and the mass fraction of empty weight relative to MTOW instead:
747-100: 333.4 tons MTOW, 162.5 tons empty, 183,380 liters tank capacity. That is 1.1285 liters per kg of empty mass. The mass fraction of empty relative to MTOW is 48.74%.
Concorde: 186.88 tons MTOW, 78.9 tons empty, 119,800 liters tank capacity. That is 1.5146 liters per kg of empty mass. The mass fraction of empty relative to MTOW is 42.22%.
If you compare passengers plus fuel to the empty mass of each airplane, the Concorde actually comes out well ahead of the 747 and looks like a miracle of lightweight design. Having structurally efficient delta wings helped and should explain most of the difference.