I think you should continue to do this. The brain will memorize a response and continue to react to that event long after the physiological reason for that response is past.

So, get a hat and go outside. If the sun is too bright, you can use the hat to reduce the glare.