I don't have that sx but your post did make me want to reply.
So your neuro thinks it can be panic attacks? Now what? You are still having a huge scare, that is very real. What did he recommend you do for the panic attacks? Did he blow you off with his comments?
If the answer is yes, you have to be a better advocate for yourself. Do you bring anyone in with you when you have a consultation? That may help, especially if you believe the neuro is not being as thorough as he/she should be. Two concerned people can work their magic on the situation. Plus your "other" might perceive what is happening differently, interpret whether you are being blown off or not understanding what is being said.
It should not be adversarial. You are both on the SAME side and he gets paid a lot of money!

I would pursue the panic attack with my PCP, no doubt. Let the PCP have a crack at it and see what is said.
If panic attack is ruled out then go back to the neuro.
Whatever you do, never let the doctor stand with one hand ready to walk out the door.
One of my favorite ways to pleasantly manipulate the situation is saying, "Ok, you think it is such-n-such. We need to work together to come up with a solution. I am open for suggestions."
smile, relax. He is the expert and you are the body!