It is the energy of the impact or more accurately, the acceleration or deceleration of the brain within the skull. It is the brain impacting the inside of the skull that is the trauma. The skull does not have to move very much for the brain to be subjected to trauma. It all depends on the difference between the rate of acceleration/deceleration of the skull versus the acceleration/deceleration of the brain within the skull.

Think of it like driving in congested traffic. The car 1 in front is the skull. Car 2 following is the brain. If the front car slows faster than car 2 slows, car 2 will impact car 1, even if both are still moving. It is the differential between the two speeds or rate of deceleration that causes the impact.

The question is. What is the minimum force required to cause concussion trauma ? Even the experts do not know the answer. Some think it is 60 G's meaning your head would feel like it had a 180 pound brain banging into the side of the skull. Others think it is much higher. But, the brain has fluid surrounding it that allows for some of the impacting force to happen without causing any trauma.

In your case, it does not sound like there were forces strong enough to be concerned.