I have the same exact problems.

I played (non-contact) hockey last week and someone ran into me, contacting my helmet/ face-mask...more of a "nick" than full-on contact. There was no pain, but a second of vertigo....and a WHOLE LOT of anxiety. But I was fine the rest of the night.

Next morning, my symptoms, which are usually a "5", went all the way up to a "9" the entire day.

Deep down, I can't possibly imagine it was a concussive force, but it definitely did exacerbate my symptoms.
For years, I flip-flop if this ordeal is psychological or physical. Incredibly frustrating.