Have you tried vestibular exercises? My doctor has me doing them, and they've been slowly helping my vestibular system. The idea seems to be to initiate the system (my symptoms flare when doing it, but then go away afterwards) without over-taxing it. You should be able to find some online or from a physical therapist (just start slow). My symptoms have been different than what you described, but it might be worth looking into.