I wonder from the way you describe it, whether your physician is thinking of studies in recreational ketamine users when he says there is memory loss in people who took a lot of it. That phrase 'took a lot of it' sounds like the way recreational users rather than clinical patients might be described.

Ask him for copies of the studies he is talking about and tell him you want to understand how those studies apply (or don't) to your own personal circumstances.

That said, cognitive deficit and memory loss is indeed extremely common in ketamine users both recreational and legitimate in a hospital or prescribed context. One of the other especially nasty side effects from extensive use is severe bladder damage.

Ketamine has been around since the 50s and 60s so there is a vast amount of published research which I just can't trawl through. You should be able to get enough reputable medical information from Googling to determine the likely potential memory loss risks for the particular ketamine treatment that is being proposed by your doctor for you. I imagine most ketamine treatment for CRPS is well controlled and managed but if it is to be given by infusion, over multiple days and/or sessions at increasing doses, there is always a risk of hallucination and memory loss side effects. You need to speak to your doctor, research thoroughly and weigh up the potential benefits against the potential risks for your specific situation.