Remain very calm and ask him "If you were not sure I had M.S., why did you not refer me to someone who would be able to find out?"
You can't blame this doc for your M.S., but I would let him know that you will always live with the nagging thought "what would early intervention have done for me?"
I wish you peace with this awful frustration.......