Any man can have that trouble in certain circumstances it may be considered normal infact.
Although, I have read many reasons for this to occur in abnormal situations. Like MS, spinal cord injuries, drug side effects, penile nerve injury via surgery, and improperly formed and adjusted bicycle seat. Also diabetic neuropathy, viral or infectous damage, and pyschological complications.
If you can I would consider seeing and discussing your concerns with a doctor (preferably a urologist) but not excluding a neurologist as well.

I have a lot of trouble in that same location of the body. So I really value even a faint sense of orgasm.