Set boundries and stick with them no matter what. As for taking Olivia, if I were in your shoes, no way would I allow him unsupervised visits with his parents or by himself.
You may think he is not a danger and would never do harm, but the news reports are littered with stories of how a divorce situation turned tragic in regards to the children. From what I've read of your saga, his mental state is not stable, and thus I believe you should not trust him to take Olivia out of a controlled environment.
As for your Mom, how sad she cannot be supportive in a healthy way. It will be good when you can move out to be on your own again. Hang in there and be careful of what you allow him in privileges.
Carolyn