It is interesting that he said you don't get large fiber involvement until later in the game with hereditary neuropathy. I waited a very long time to see a neurologist, at least 15 years after the numbness started in my early 30's. By then my NCS was abnormal. My skin biopsy showed no discernible nerve fibers so it was pretty far gone.
I had no idea that all this was connected with my childhood coordination problems, as there was no access to family history.
It sounds like you have found a good neurologist. As I recall a lot of your symptoms fit with CMT.