It's a tough thing to figure out, but sometimes you have to just be honest and say you need some quiet time...
Maybe suggest to guests/family that they can go sightsee for a few hours in the midday?

Maybe they felt like they had to stay and visit the whole time, but perhaps going off for awhile might have been nice for them too?
Doesn't hurt to suggest it, so they know it is OK with you that they take off for awhile.