From my general knowledge about TCAS (that is, not from actual simulations of such scenarios), I'd say that yes, it can be overwhelmed, but not in the sense that it will give up.
First, TCAS can demand a very limited range of resolution advisories (RAs). They are only vertical. The standard "Climb" or "Descend" RA assume the rate 1500-2000 fpm. TCAS can ask to increase it to up to 4400 fpm. (Or it can ask not to manoeuvre). In most installations, it will also not produce some RAs depending on the aircraft configuration; for example, it will not ask to climb at all if the airplane has flaps down past certain point.
Second, TCAS doesn't think too strategically. It can handle many targets ("intruders"), but in the end it will sort them according to their severity (primarily using the time-to-encounter factor) and will handle the most immediate ones in RAs.
It may well happen that after a short time, another intruder will become more severe. TCAS will then recalculate RA accordingly. (It takes into account the current climb rate and is generally reluctant to change the direction or even to choose a so called "crossing" RA, where the aircraft must cross the current path of the intruder, but it will do that if it determines this necessary. It also takes into account the typical (but clearly defined) pilot's reaction to the first and subsequent RAs).
So, at each point of time TCAS will "try its best" to handle immediate threats. It will (generally) not demand impossible manoeuvres nor superhuman abilities: all it can demand is known and accounted for. Yet, in theory I bet it is possible to create a scenario where TCAS would lead to very close encounters, or will select a sub-optimal strategy - even if we assume that all the aircraft obey their RAs precisely, which in itself is not given. In aviation, the engineers (so far) generally favour clear deterministic algorithms over something very clever but unpredictable and hard-to-test.