Twofish, don't listen to them. It's kind of like women who have a tilted cervix are told they can't have kids. It's like some old myth people run with sometimes. Sure, there are risks - but there are risks with everything we do.
Before my partner and I moved in together, I flew every 3 weeks from Ohio to California and I did this for FOUR years straight. The worst thing that happened to me was the first flight I got drunk (I was nervous, first time flying) and threw up in my purse of all places. Another time I got a nose bleed. Never had a seizure, I did feel some 'weird' pressure, but I'm sure it was just stress and the pressure changes.
We do not need to travel with a doctor, at best - maybe let someone on the staff know that you do have Epilepsy and if you have any special type of seizures that would require a non-standard approach, fill them in on what to do. It doesn't take a doctor to put you on your side and protect your head.
Go fly!!