I have been in that same situation! I never thought of it as a panic attack but more of a 'my brain can't take this and I gotta get out NOW!'
i am happy to say that this gets better for me with time. I can endure much more noise now, and even some multiple types of noise. I have to be rested though.
I have talke to a lot of audiologists about the hyperacusis (damaged noise filter) and they say that gentle and very gradual exposure to noise (sound

) is the only way to address this. I have made an effort to put some happy sounds into my world, and have tried to change my emotional reaction to them. We need to live in a noisy world eventually, so doing it in a controlled manner is a good strategy. It seems to be working for me!
Hang in there. It does get better. Slowly. I need to hear from the longer term people the same thing, as I still get discouraged. But it does! VERY slowly. Baby steps. Steps backwards. But eventually forwards!