I do the same thing! I actually had the same experience as far as when I went to the doc they said my speech was fine. *Howeverrrr* my parents and my friends notice the speech decline, and the fact that it takes longer to have a sentence come out.
I can be in mid conversation and completely forget what I was talking about, even without distraction. It can be scary and frustrating too. (scary because I have to control the obsessive unfounded thoughts that it could turn into alzheimer's) I'm sure it wont!
I am just going to hold onto the fact that it will get better, but like you, I notice some decline in areas that were not a problem when I initially was injured.
I hope that the program doesn't kick you out. It's too bad that they are doing it based only on improvements, and not based on what you need help with!!!

Maybe you can re-enroll if that happens? (I know, probably not, but worth a try)
Good luck...