I'd follow my doctor's advice.
It may be that you will feel better waiting until you get home from your trip.
You seem to be very nervous and being in a comfortable place for any unexpected adverse effects is a good thing.

I know others have had aftereffects, but for me, it's just a normal day when I have my infusion, and always has been. Other than a slight sore throat for a few infusions, I had no problems. It's normal to be nervous before the first one, but the infusion team will be watching you and know what to do now, unlike when the first infusion centers opened and had no idea what to expect from this new drug.

Let us know how it goes.