Nick,

I find I have this same struggle the most when there are more than one other voice in the room or conversation. It puts too much strain on me to be spontaneous in conversation. I CAN NOT chit chat. I do not have an ability to talk abstractly. When people are just making conversation rather than discussing ideas of substance, I have to just back away and listen.

There is an old adage.

Simple people talk about people.
Smarter people talk about things.
Wise people talk about ideas.

Rather than feeling inadequate, gauge whether the conversation is important.