I think it's because when you sense confrontation your body goes into 'fight or flight' mode ready for action, part of which involves putting your brain onto high alert for any signs of threat, ready to react quickly to any danger.

So you involuntary start concentrating much harder on what is happening, and the brain has to work much harder to achieve this. That's how it felt to me anyway.

I found the best way to handle this is to try to stay relaxed, so that this fight or flight response doesn't happen. Deep breaths etc. Sometimes easier said than done of course!