I think any kind of social interaction takes a significant amount of brain power. When you are home surrounded by family there is always something going on, you are carrying on conversations, theres activity going on - its probably noisy to some degree. You are trying to prepare meals, tidy up with numerous distractions all around.

I live by myself so I really notice all the ramped up sensory stimulation when I visit my family. What I do is take lots of little time outs in a quiet room with my noise cancelling headphones (a girl's best friend) and some relaxing music - even 5 minutes helps