Judging by what you told us about him he doesn't mind being alone. And people who don't mind being alone can't relate to those who want to just "be" with someone else. Plus...he's a man and men just think differently than we do.
I used to get all bent out of shape by some of the things my DH would say....until one day I realized that he was just never going to understand how I think. Same way I was not going to understand his thought process. When he'd ask me why I wanted to do something or why I reacted a certain way to something he said all I had to say was "because that's how I
feel" and he would back off.

He knew if he pursued it he'd get a whole conversation about feelings and no man wants to go through that!!