I think what your plan to do is fine. If it hurts their feelings then they really don't know you do they?

Even as a caregiver I have a similar problem. People think I am superwoman at times and that frustrates me. Explaining the why is always brought back with a comment like "Girl's night out!" or "Come on, you need to get out and take time for yourself" or Can't Jim stay alone and you get out for a couple of hours?" Stuff like that. But many times it's just because I am tired and don't feel like going out. I am grateful to be asked but people don't get it sometimes. Only my true close friends and family get it and honestly that's all that matters to me.