I have the same issue as you, may a little more mild. After a couple of concussions, every time I bump my head (a bump that would normally give someone a headache) gives me a concussion for a few weeks and I have this terrible exercise intolerance for months where I can't even walk for more than 5-10 minutes without eliciting symptoms.
After exercising fine for like 6 months I recently dropped a tall hiking stick and it slightly bumped my head (very little force) and this seems to be aggravated it back up. I am praying that I won't be out of the gym another 6+ months.
I am not convinced this is an anxiety thing because I have had this happen to me, and then I forget about it and mind my day, then a few days later the symptoms creep up on me and I realize what happened. Anxiety may be related but I think it may be related to a phenomenon seen in chronic pain patients known as central sensitizaiton where the nervous system essentially programs itself to be in pain (in this case an inflammatory mess). It programs a response for itself that it thinks it will protect itself but is clearly actually maldaptive.
I also heard a theory that perhaps a prior brain injury may have caused a cerebral spinal fluid (CSF) leak (the fluids that cushions your brain). This could be why the brain is so sensitive.
Either way the best treatment I have found so far is cranial osteopathic manipulative medicine (in the US). Also anti-inflammatory supplements like fish oil and cover-three (very expensive) may be of help.
I have tried the hormone therapy, but this seemed to mostly shock my system. The only two I would recommend would be some sort of estrogen inhibitor (red supplements has natural ones just google this), or growth hormone stimulator (secretropin), I have had some success w/ these.