You are doing the right thing.
I would never stand for some snot nosed punk to tell me that I'd been passed over cause I didn't go to bowling, or to after work gatherings.
Your health (mental and physical) is most important.
You'll never get into a peaceful state of mind with all that stress.

Stress is a #1 factor in exacerbating PN.
Try to get your neuro to sign off on disability,
and fight with SSI and SSDI.
Get out of that work environment !!

You are still young and there are many opportunities for someone with drive, to work at home, or even start your own marketing business on the internet.
You said:
"......the credit union I work for as the collector ....."
Don't you know that in this bad economy, that your profession is recession proof!
Your skills are needed, if you are good at it.
You can find work elsewhere, if need be.
A place that knows up-front that you are theirs 8 hrs a day and the rest of the time ..... you aren't !

PN does not have to be an end-all, unless you become totally incapacitated by it.
The meds keep me going too.
Without themI'd be a cripple.
But with them, I'm 64, self employed (30 yrs), and have a retail shop and do in-home, road-service work.

Don't take any crap from a young punk that's so hyped up about 'company' and 'interaction' on your own time.
There's a time for work and a time for personal requirements.
They should never cross the line.
Any employer who requires that, is full of crap.
Get out of there. It isn't worth it.