You ask the right question. Given the amount of misinformation out there, I can understand why you are confused.
First the tip vortices: They are neither a source of downward force not do they cause drag. They are only a consequence of lift creation by wings of limited span. This answer tries to explain this in more detail. If, however, you mean the wake vortex, then yes, you can view them as a source of drag. But still not of a downforce.
Now to the vortex of a delta wing at high angle of attack. This is caused by flow separation at the leading edge and produces a low pressure area over the wing. While the tip vortex picks up its rotation speed while air is flowing over the wingtip, and then continues into the free airstream of the wake, the vortex of the delta wing directly affects pressure on the upper side of the wing. Without it, the wing would stall at a much lower angle of attack, so it helps a delta configuration airplane to land at a much lower speed. But it also causes a lot of drag: Flow separation at the leading edge means that leading edge thrust is lost and the lift force is essentially orthogonal to the wing surface. Given the high angle of attack, the drag of a delta wing with vortex lift is very high (lift force times the tangent of the angle of attack).
But this is not all bad: It helps to burn off energy quickly which is useful during approach and landing. Certainly, you do not want to rely on vortex lift during cruise: Then the lift of a delta wing will be created much like it is on unswept wings, tip vortices included.