Did you report the inappropriate comment by the doctor anywhere else besides the office manager?
That kind of comment is totally out of line and shows very bad judgement by that dr.
I would find out where else to report this sort of thing as he could be doing it and more to others.
Medical board for your state?

[How to Complain About Your Doctor
If your doctor behaved in a way that felt inappropriate or unprofessional, speak up.]
http://www.realsimple.com/health/fir...ut-your-doctor