I wonder if you are trying to ramp up too fast.

The damage to your brain happened. It left you with a weakness. It is no different than an ankle I broke 30 years ago. I can do everything but it will feel sore and stiff after. But, the brain does not recover like an ankle does.

The doctor who said, No restrictions, was likely just saying he did not see anything that indicated you could not or that resuming them would put you at risk. But, there is no way he could say that you can tolerate all of your previous activities without a return of any symptoms.

You may do better to only increase one of the three activities and work to increase endurance in that activity. Then, try another activity a week later. It may only be one of the three activities that is causing problems.