I'm not sure if there is any scientific explanation for contacts not being good for people with vision issues, but I can tell you about my experience with wearing contacts Vs. Glasses.

Before my injury I only ever wore contacts. I hate glasses. But for about 4 years now I've only worn glasses because it makes that much of a difference to my symptoms. (And that's normal glasses that make that much difference for me not even special, prisms glasses. I would assume the prisms glasses would help more)

I have vision issues (light sensitivity, convergance insufficiency, Accomodative dysfunction and tracking problems)

I don't know if my intolerance to contacts is because of these problems or something else. My head is generally, very sensitive to any kind of pressure on it, including glasses that are too tight, putting my hair in a ponytail and wearing goggles to swim. All of these things irritate my symptoms, so it makes sense that contacts would too I guess.

But when I started to see my current optimetrist for vision therapy, she did also say that I probably shouldn't wear contacts. I never asked her why, unfortunately.

Anyways, I'm not sure how helpful this will be, but I hope it helps a bit at least.