I don't know if this is helpful or not but I have similar issues. My writing remains fluent and easy while my speech not so much. I have trouble with defining and explaining, while talking about nothing (shooting the s***t I think you call it!) is fine. When under sensory demands I

stutter, my speech slows, and I can lose the thread mid thought. I find that any extra demand on the brain effects my speech. I would suggest you not worry about it getting worse (i know easier said than done). I've been at this a long time and worrying about something that may or may not happen is exhausting and takes from any quality of life you have right NOW. Just the freaking out over your speech worsening can cause your speech to worsen.

It sounds like you have a great neurologist which can be very hard to find
Besides self care, both physically and emotionally, we really have very little control.