I can empathise with your situation. I too cannot tolerate simple conversations, even within my family. Extraordinarily frustrating.

I'm not sure what the answer is to getting better as I've yet to find it. However, after 3.5 yrs of trying, I do know that the "push it" philosophy is a sure road to hell. On two separate time periods, I've tried the immersion philosophy/approach to desensitizing myself, and in both scenarios it has taken months and months to recover. During this time, I have reduced tolerance, increased headache intensity, and a host of other things.

I bring this up because there is a lot of talk about anxiety. I think for some people, this is discussed from the wrong perspective. Anxiety is not the cause of our problems, though it can exacerbate concussion-related symptoms. For me, anxiety is the result of over-stimulation and generally occurs during the early phase of recovering from a given event. If I am pushed during the time-period, the anxiety gets worse; if I recovery and am careful, the anxiety dissipates. At my basal level with no overstimulation, anxiety isn't ever an issue.

Overstimulated brains have altered physiology, which leads to a host of various things... worse symptoms, darker mood, anxiety, depression, etc...

I guess the key is to find that boundary that lets you attempt to push things but not get over-stimulated. For me, this seems virtually impossible but I've come to accept this and seem to be getting by.