It is largely irrelevant that he is hurting tbh - as a fan, or indeed as a dedicated manager, he should be hurting. If he wasn't, that would be the concern.
As you say, for me he's a safe pair of hands and if he stays we'll do alright - the question is do we want to move on or not.
I keep flip-flopping on this, but the reason I'm ultimately for not changing manager too soon is less to do with the positives of/faith in Wilder; it's far more the worry about who the Club will appoint next, given the complete clusterfuck with Selles in the Summer...