Marijuana doesnt damage your brain the first time you do it..a year ago i smoked half an ounce with friends every weekend for the entire weekend until i stopped for about 2 months. I ended up going back to it and like you did, had a really bad trip.
I panicked and ended up thinking my heart was beating way to fast and almost walked myself to the hospital but come to find out, i started getting allergic to it.

I didnt know it at the time but i tried again a month later and the same thing happened.
As far as i know you have to smoke quite a bit before it damages your brain, you're getting the idea that it might the marijuana thats causing you to have anxiety because, you panicked the last time you smoked.

Derealization is a common symptom of anxiety and anxiety is a most definitely a common symptom of PCS.
Trust me my first week of PCS was terrible, i had the same symptoms you did, i was scaring myself for 3 days straight and had bad insomnia for the first 3 days.
Just know that Concussions can trigger anxiety and attacks, i've had a few attacks since the incident and it was because i was doing exactly what you're doing. Worrying to much.

Just chill out a little bit, DO NOT GO SEARCHING around on the internet for things that might be wrong with you, you're only going to make your anxiety worse.
The best way to approach anxiety is, know that its there and ignore it, be okay with it.
Brain injuries take awhile to heal, the symptoms may come and go and one day you might wake up and come to a full recovery. Just relax and get some rest.