Does the letter give the Drs name ?
If so do a web search of the name and find out as much as you can.
maybe you'll find that dr is highly rated by other patients.

How far out is the appt?
Your lawyer should have details on how best to proceed and may have knowledge of the Dr in question also.

The letter should say something about if a need to reschedule or if distance is too far.. but atty should help on that ,
but I"m pretty sure you can't just blow it off..

It's reasonable that they would want a independent reviewer - and hopefully it is not like work comp where the so called independent docs are not independent at all. They often fashion the report to more fit the wc ins side and not the injured claimant.

try not to worry too much