Is the "weather premium" worth it?

I was watching a CityNerd post on Youtube recently, and he mentioned what was (in his opinion) the ridiculous premium people pay to live in places with good weather. His argument was, you could live somewhere with less than perfect weather, such as the midwest, save huge amounts of money on housing and cost of living, and use the difference to travel for an entire month somewhere warm every winter if you wanted, and still come out ahead.

Got me thinking since I've long wanted to retire to CA mainly for the scenery and beautiful weather, and have just accepted I'll need to save more/work longer in order to do it. I am wondering if people, especially those who have moved primarily for the good weather, think the "weather premium" is worth it?