I know exactly how you feel since this has happened to me a lot of times. Everytime i get hit in the head (even very slightly) i tend to become very anxious about it and that worsens my initial symptoms.
Luckily, symptoms do not worsen from the hit itself and i tend to get better longterm even though i hit my head lightly here and there.
Best of luck,
Jones