If the pilot and skydivers maintain true integrity and aren't actually exchanging money or favors, then yes, it is legal.
Let's look at a couple of possible scenarios.
Scenario A: Three private pilots who have skydiving certification take turns flying the plane for each other while two of them dive. They go up acting as the pilot an equal number of times, and they all split the cost of the aircraft 3-ways. None of them were compensated or hired for their turn flying, and they all paid a pro-rata share.
This is clearly legal, and this scenario above sounds plausible and makes sense to an outside observer.
Scenario B: a pilot agrees to spend hundreds of dollars to take two other people skydiving for no compensation and even agrees to split the cost of renting the plane 3-ways. (*How generous of him...)
If this were actually what was happening, this is perfectly legal. However, it is suspicious; I think that if this got the attention of the FAA, they might start wondering how the pilot is really being compensated.
Is one of the skydivers mowing the pilot's lawn for free as "a favor"? Did one of them let you borrow his backhoe a couple of times to do some landscaping? Are they watching your kids while you're at work?
The FAA ultimately decides what compensation is, and it doesn't necessarily mean cash, so be careful with this.