Finding a neurologist that actually listens to what their patients have to say has been a huge frustration. In my opinion, many neurologists and surgeons are extremely egotistical. Your doctor questioning why you came back to him after going to someone else seems like ego to me.
I had one neurologist considered to be the top of his field that told me my symptoms were caused by me crossing my legs. He swore if I stopped crossing my legs my symptoms would go away. The next neurologist read my file and although he wouldn't speak against the doctor (that seems to be a big no-no) he never pursued that diagnosis.
I have found that as long as I behave like a responsible, mature adult, most doctors follow in the same pattern. If they don't move on to the next