I've been through this one twice. Once when Jim got out of the military and then when I retired to take care of him fulltime. What an adjustment! It wasn't hard, but it was trying to get used to not having my "own" time. But in the long run, I've never been happier and he says the same thing. Retirement brought us closer than ever, no pun intended. I've learned to like things he likes that I thought I hated and visa versa.
Just give it time and yes you'll go through the apology stage but not for long. He'll especially need adjusting since he's used to having the house to himself.