The hardest hit would sting but not be concussive. Surface pain has no relation to head impact trauma.

My concern would be falling or getting hit by your playing partner's paddle. I think that risk is low.

Choose the intensity of play.

I played ping pong and chose not to play with the intense players.

It is entertainment, not Death Ball.