You really have to go with your gut instinct on whether or not to go back to the neuro. If you don't have confidence in his ability to care for you, then it's kind of hard to have him care for you!
Sounds like your primary IS taking good care of you. There are some differing thoughts on this withdrawal stuff. Some say to even go as low as 2-1/2 mg. for 4 - 5 days and then to go every other day to taper. That's what my Dad's doc did. But, again, it's different for everyone. You have to go by how you are FEELING during the taper.
Again, any new symptoms - call your primary.
You should be fine! It sounds like you are in good hands.
Annie