I've been at this TBI thing for over 3 years now. I have not returned to work, but I am taking graduate classes. (My doctors feel if I finish my degree, I have less of a chance re-injury, due to the nature of my job).
Anyway, are you doing harm by pushing through? Technically no. However, your brain is only going to let you push so hard for so long before you are dealing with really severe symptoms. (I've done this a few times and ended up in bed for like a month). The long and short of it is, each time you over do it (i.e. your symptoms increase), you are adding on to your recovery time.
Believe me, I know how horrible it is to not be able to do the job you love. (I've dealt with huge depression over this since my accident.) My hope is that as I continue to take care of myself with various therapies, that when I finish my degree my brain and body will be ready for me to go back to what I love.
For the dizziness from patterns that you mentioned, I too have issues with this. It triggers seizures in me

Have you had your vestibular system's function and your vision checked since this started? Both can really play a part in causing dizziness and disorientation caused by seeing such patterns.