Perhaps you're right. Maybe it's me caveating the shit out of a stance
But seriously though, and I don't know the specifics of Luton's new ground*, but I think the challenges of our situation mean that it's an all or nothing/lick of paint approach and a lifetime of the latter has conditioned me (any many others I suspect) to think that half a billion investment is completely off the table.
As others have said, a complete new build would mean a temporary home. This, to my mind, would be the biggest issue. The Idea of playing in Derby is mental. We'd be better taking the hit and go somewhere much more local but smaller.
*Are they moving from Kenilworth Road?