Sounds like you are safer with the GP, for now, until you can get a new Neuro. I learned the hard way, not paying attention to warning signs of a bad doctor, or one that could give a crap. They can really screw you up bad.
Go with what you feel is safe for you.
Mary