My neuro told me once that if the fatigue I experience is increased as the day progresses, it's probably from MS. If it begins in the morning, it might be depression. Yours sounds like it's MS related.
Make sure you have your thyroid numbers checked; my fatigue is usually a warning that my thyroid isn't working right and my medication needs to be adjusted.
If that's okay, then definitely you should at least get a second opinion from a different neuro.
Keep us up to date about what's going on...