I have no idea why they are attracted to you but just taking a wild guess, perhaps one of your neighbors has a one or more fruit trees that is attracting them, or a composs bin close to your fence that you may walk past every morning and perhaps they jump on you then and then the little buggers come for a ride to work with you.
I would be very tempted to lay some bait in plastic containers for them along the neighbours fence that you may think there coming from, and also bait in your car so their attracted more so to your bait, than you, i found this on the net were people had a big problem with fruit flies.
.................................................. .................................................. .................................................. ............................

I work in a restaurant and we get them bad in the summer, and we use a mixture of cider vinegar, and water. about 70 percent vinegar, and 30 percent water, in a glass, and set them where you have the most fruit flies. a few days of that, and your glasses will have fruit flies floating on top. But make sure to use CIDER vinegar. The cider attracts them. change the mixture everyday, and do it for a couple weeks. It may not be the most pleasant smell, but hey, No fruit flies. Give it a try! It really works! Good luck!

good luck anyway
Brian