I have been experiencing issues with my vision and glasses/contacts as well.
I find contact lenses are easier on the eyes, but not perfect.

I also find some days my actual visual acuity is better than other days, so I switch to a different correction on those days.

Can a correction that is too strong or too weak amplify my nystagmus or vertigo (actual eye movement), or will it just cause subjective dizziness?

In other words, do I need to have a perfect prescription in order to reduce nystagmus/vertigo, or is that outside of my control?