a simple solution is to put the wedge style (old fashion rubber wedges) door stops under each door. You don't have to shove them under to the point of making them difficult to remove, just slide it under door bottom so the cat cannot get the door open. We had to do this (and I frequently use my WC and can still get em out). Normally, while on my crutches I just pop it out of the way with my foot and get what I need, then reinsert.

My cat was also a real stinker with my handles, so on 2 doors in particular we put on door stops (on the outside of door). I just flip them up to open door and pop em down when door is shut.

As silly as it sounds, your cat may benefit from a fellow house-mate. They often will play, stalk and entertain each other. Sounds like your cat it really bored. If he has nothing to entertain him he will find something to do (and not to your pleasure...)

Good luck!