If you can stand a holiday, or a day off here and there, I'd do it.

All the medications we have that affect neurotransmitters, affect receptor concentration over time. The body tends to want to keep some receptors open, for sensory protection. It is a biological adaptation that evolved to allow us to flee, or fight, and to avoid pain.

So even if you have a life long need for a Benzo, you are going to slowly reach a dose where your kidneys cannot excrete it well, and it will build up in your system causing CNS depression and respiratory depression. (this is more common in the elderly and does not need high doses). This is why Benzos are no longer recommended for elderly patients.

Valium is a long acting Benzo. The only other longer acting one is Klonopin (clonazepam). Sudden withdrawal after long time everyday use of either, can result in seizures, even in people without epilepsy. Much of this risk is time dependent and dose dependent. So taking only when needed, will help prevent tolerance issues and need for higher doses. What I see in the general population with people on a Benzo (which is a controlled substance BTW), is that if they are maintained on it for a long time (because of special need), they can be caught short on a refill, lose them, get robbed, or a natural disaster interrupts their ability to get more, and then the sudden discontinuance becomes very serious. It is this that can be a serious medical consequence.

Ativan is a moderately long acting Benzo and Xanax is short acting.