Short answer: Yes.
Maximum speed is correlated with wing loading. A high wing loading shifts all speeds up. A flying wing will always have a lower wing loading in comparison to a conventional design when payload, range and landing speed of both are identical. So you will start at a disadvantage.
Due to the high lever arm of the conventional tail, the conventional configuration can use high-lift devices (or highly cambered airfoils) for a much higher maximum lift coefficient than what is possible in regular low-pitching-moment airfoils. This allows to keep the wing small for the same landing speed. Unfortunately, the flying wing is restricted to those low camber or even reflex airfoils because its surfaces for creating a pitch balance have a small lever arm and, consequently, a poor moment-to-lift ratio.
The design of the Prandtl wing leads to a bit of downforce at the tips when in normal flight in order to lower the wing's root bending moment. But only a bit. Sweep places this downforce in the aft region of the wing so the inner, more forward region has to create all the lift for carrying its own weight, the payload and, of course, compensate for that downforce, too. Since the low wing bending moment will allow for a light wing, the total lift requirement is rather low, too, but still encumbered by the aforementioned loads.
Now to the maximum speed: A flying wing will allow for very low drag coefficients. There are no intersections between parts and no tailboom which adds friction drag, so for a given wing loading a flying wing will allow to reach high speeds with limited power. However, the lower wing area of the conventional design with the same landing speed has less wetted area and so can easily accommodate that tailboom drag. Once the details like part intersections are well designed, the conventional design should come out ahead.
What you need to watch out at high speed is flutter: That wide, light, swept wing will start to flutter much earlier (at lower speeds) than the rigid, strong, small, straight conventional wing. With sweepback you will encounter a speciality of flying wings, a coupling between the short period mode and the wing bending eigenfrequency. Since the short period mode frequency goes up with speed and the bending eigenfrequency is constant over speed, flying fast will bring both together and require a stiff wing in order to shift that point up. Don't forget that pitch damping of flying wings is an order of magnitude lower than that of conventional designs, so the short period mode will show up in a Prandtl wing if you go fast enough.