a great neuro I was referred to by my PCP once told me the following (the specialist I was seeing kept suggesting another specialist, and sent me back to my PCP... similar story to your current scenario)

"Let me let you in on a little doctor secret: when your doctor refers you to another doc in their chosen specialty, it usually means they don't know what else to do for you & perhaps the other doc can better treat you. Either that or he just doesn't want to treat you anymore."

If I were you I would grab that referral and run to another doc/neuro. Good luck! It's completely unfair to put the patient in the middle when all they want/need is help...