Dean’s lack of belief in Sam is something that has been building. As a kid, Dean was protective of Sam, and this continues into adulthood. The sad reality is that if you feel the need to protect someone, it implies that you don’t think much of their ability to protect themselves. At first glance, S2 Dean at least seems to have faith in Sam's goodness - he professes as much to Gordon Walker, and in BUaBS, he refrains from killing Meg-possessed Sam due to his faith in his brother.
But on closer glance, it's a bit more complicated. Dean's winning argument in Hunted is that Sam feels guilty watching porn...not the most convincing of material. Even in BUaBS, the reason Dean gives for not killing Sam is this: “Sam, when Dad told me ... that I might have to kill you, it was only if I couldn't save you. Now, if it's the last thing I do I'm gonna save you.” It’s not that he has faith in Sam being good. It’s that he is going to save Sam from himself. Dean’s insistence that Sam must be good isn't faith - it's desperation.
Any faith in Sam Dean may have had in S2 seems to have slipped away by the end of S3. At that point, he would rather go to Hell then let Sam even risk using his demonic powers.
In S4, Dean expresses skepticism that Sam knows the difference between right and wrong, and in TMatEoTB (that’s a keyboard-full), spends an entire episode trying to undermine Sam’s decision because he thinks Sam will sleep with Lilith, or do something equally stupid. Sam's failure to rescue Dean from Hell, combined with his failure to be better than Dean and avoid demon deals, seems to have cemented Dean's lack of faith. I do think Dean actually hoped for both things from Sam: That Sam would save him, and that Sam would do it without falling like Dean did. Instead, Sam failed spectacularly at both.
So when Dean admits that he doesn't believe in Sam, doesn't believe he will be able to keep saying "no" to Lucifer, it's been awhile in coming. That Sam would declare faith in Dean, that Dean would change his mind about saying "yes" to Michael because of that, is also believable. Because I may have overstated my case: Sam and Dean had been partners in hunting for so long because they trusted each other. And Dean was not kidding when he admitted, in Scarecrow, that he admired his younger brother's willingness to stand up for himself. Dean's faith in Sam is selective, but it is there. And we see both Dean's belief and faithlessness convincingly on display in PoNR.