Looking at the quiet scene between John and Sam in Dead Man's Blood. John tells Sam the college fund story. He talks to Sam, about Sam, but everything he says (up until he talks about Sam announcing that he wants to go to college) applies to both Sam and Dean, and John's relationship with both of them. There's nothing unique to Sam's situation.
And very little of this conversation is about Sam, as a person. It’s about what John wanted for Sam, and what he wanted for Dean. An explanation, which we all know Sam craves. Not quite an apology - but an acknowledgement of his failings as a human being.
John mentions only two things that are specific to Sam as a person. One is Sam’s desire to go to college. Given that Sam and John had a huge screaming fight about it (and possibly multiple screaming fights thought Sam's adolescence) and then, Sam, y’know, went off to college for four years and caused a lot of angst, it would be very hard for John not to have noticed by now that Sam had a desire to go to college. The other Sam-specific statement is John’s simple statement that, “We're just different.” Which Sam immediately disagrees with.
John concedes that, "It never occurred to me what you wanted." He's talking about the period of time after Sam started expressing a desire to go to school. The stories we've heard from Sam's childhood, especially adolescence, imply that Sam was not shy about expressing what he wanted. From high school plays to soccer, Sam had wants. He told his family about them, and if his family disapproved, he went out and got them anyway. No matter how often Dad yelled at him or Dean guilted him. The fact that John didn't think about Sam's wants until after Sam ran off to Stanford, after Sam had literally spent years screaming in John's face about what he wanted, shows to me that John had a particular emotional disconnect from his second son.
We see this also when John knows he is going to die, in In My Time of Dying. He sends Sam away, saying, "Can we not fight? You know, half the time we're fighting, I don't know what we're fighting about. We're just butting heads. Sammy, I, I've made some mistakes. But I've always done the best I could. I just don't want to fight anymore, okay?"
In just about every scene I remember where John and Sam are together, Sam is very upfront about what he wants from John (typically more information, or more involvement from Sam in what John is doing, or that John focus on Dean instead of revenge - in short, a closer and more open relationship). But John doesn't even know what they fight about, and dismisses Sam's anger as "just butting heads".
In Devil's Trap, the YED famously tells Dean that Sam is "clearly" John's favorite son. Fandom has different opinions on the veracity of that (I think John's behavior is undeniable proof that the YED was lying), but there's one thing that I haven't seen any fans touch on: The only proof or example that the YED offers for his statement. "Even when they fight, it’s more concern than he’s ever shown you."
Of all of the examples of all the memories of John and Sam that the YED-in-John's-body now has in his brain, he goes for John and Sam fighting. That's his big example of emotional closeness. And that's really sad. And I think that was a big factor in why Sam was a teenager yelled at has dad so much: He wanted attention and concern, and he wasn't getting it from Dad any other way.
Dean, of course, got more emotional attention from his father, but at a deep cost to himself. He got it by becoming the family emotional caretaker at far too young an age, and being a little adult. His dad's "man", as John put it in Something Wicked.
Because John was closer to Dean, he also had a better sense of Dean as person. We see this in their last scene together in In My Time of Dying.
As with John and Sam's scene in Dead Man's Blood (where John starts off by talking about the college fund) John begins with an anecdote. But the story he tells Dean is not about his wishes for his babies whose personalities he does not yet even know. Instead, he starts off with a story that’s very specific to Dean: “You know, when you were a kid, I'd come home from a hunt, and after what I'd seen, I'd be, I'd be wrecked. And you, you'd come up to me and you, you'd put your hand on my shoulder and you'd look me in the eye and you'd... You'd say ‘It’s okay, Dad.’” Then he apologizes - which he never does to Sam.
When he acknowledges his own wrongdoing, he does so in a way that specific to Dean and the situation he put Dean in: “You took care of Sammy, you took care of me. You did that, and you didn't complain, not once. I just want you to know that I am so proud of you.” Even though Dean was far less likely to voice complaints than Sam, John had a greater awareness of Dean, of what his life was like, and of what John's horrible choices were doing to him. By contrast, he seemed to be utterly clueless about Sam, despite Sam's many attempts to express his point of view to his father and establish a connection.