Not every sound causes the same level of stimulation. Sound frequency/pitch and loudness have different impacts. The farther away the sound source is, the more that sound has echoing reverb. This reduces the impact on the ear. Think of it as a fuzzy loud sound. Outdoor sounds are often the easiest to tolerate because of this.

Think of fingers on a chalk board. The wrong frequency can be a struggle.

I know that certain frequencies of buzzing are a problem for me while other frequencies that may be louder are not.