Is there a formula that can be reliably used to determine when turboprops become the better option for domestic flights in a plane similar to a 737 in capacity and size? Also could this formula be applied to trans-continental/ trans-oceanic flights and larger planes? Should Southwest Airlines only be fling turboprops to save money or should every international airline be looking to switch at some point?
Is there a formula that can be reliably used to determine when turboprops become the better option
A formula not but an indicative plot yes. The 5th slide of this presentation (slide that I repost here under and that has been taken from ¹), shows a comparison of the specific fuel consumption for several propulsion technologies:
As visibile, the fuel consumption of a typical modern turboprop overcomes the one of turbofan at some Mach 0.5 and that is the point where a turbofan becomes a better option.
¹ Daniel P. Raymer, Aircraft Design: A Conceptual Approach
There isn't a one-fits-all formula. It would depend mostly on the distance between the city pairs you want to fly between. Turboprops have lower seat-mile costs, but are slower (the fastest one, the Q400, can do 360 kt true) and limited to the mid 20s flight levels. This means they can't get above much of the bad weather in the temperate latitudes. Jets just transition through icing conditions climbing or descending most of the time; turboprops may have to cruise in them.
For shorter distances, turboprops make sense, say up to 1000 miles (Q400 max range is about 1500), but people don't like to ride in them more than a couple hours, and for longer distances you need the ability to go above FL290 to get the high cruising speeds and get into the stratosphere to avoid the weather, and turboprops aren't certified to go that high, (and even if they could they would get in the way of jets on busy airways).
There's also passenger appeal. In the regional business, operators would use 50 and 70 seat jets to service 300-500 mile city pairs that would be more cheaply served by Dash 8s or ATRs, mostly because the passengers demanded jet service and were happy to pay a higher ticket price.
A Boeing 737 equivalent turboprop would be the Bristol Britannia 139 seat airliner, designed for transatlantic service in the late 50s, but it's service ceiling of 24000 ft and speed about the same as a Q400 meant it was a long haul across the Atlantic, through a lot of the weather, was quickly replaced by jets in transatlantic operations.