Ok ~ Spondylolisthesis is the forward displacement of the lumbar vertebra on the one below it, especially the L5 over the sacrum (S1) which produces pain by compression of the nerve roots.

And since he said it's "degenerative" it sounds like it's just going to get worse.
Walking is often the BEST therapy in the world. The longer you do it, the easier it becomes, and the farther you can go. So don't give up on it. It's VERY good for you!
And yes, you SHOULD be doing your physical therapy exercises.

I know they're a pain, literally, but are really beneficial.
Have they given you anything for pain? Does it work? If not, you might tell your doc about it and ask for something that DOES work because I'm SURE this is a very painful condition!! You MUST advocate for yourself because no one else will.
Best of luck and I hope your pain eases soon. God bless! Hugs, Lee