From what I understand, part of the problem is that your brain loses the ability to automatically filter out things you don't need to pay attention to. So, when you are in a busy setting, your brain is working extra hard trying to figure out what you need to attend to and what you don't.

I believe this is why just BEING in a busy or highly multisensory setting can induce symptoms, even if you are not really interacting with it in any way.