Every state's WC laws are different. Generally speaking, you can't refuse to see a WC doc. They will frequently be adversarial. If you don't have an attorney, you probably should hire one.

I haven't had an issue with requiring WC needing to provide me with a driver when it's a distance or in heavy traffic. I only drive short distances and have discussed the scenarios in which I will and won't drive with my doc, and he agrees I'm safe to drive close to home. Will your doc notify the DMV that he's concerned with you driving?