I'm so sorry about your situation Renee
I too went through a divorce when my oldest son was just a baby! I know its hard on us and we are the adults but the children....It's just not fair.
I agree with the other posters. There has to be boundaries, He can't just waltz in and out as he pleases. It's a disruption and its not fair for you or your kids.
Take care of yourself. Stress is not good for us!