I am getting to this thread really late, James is already off the show, but I can give you a clue as to why he might have aspergers!

As a person who was dx with autism as a kid, I understand it very well. Not only have I grown up around people on the autism spectrum, went to school with them, friends with them, but I have also met thousands of people, online and offline, with aspergers/autism, and have been friends with some of those "famous" people on the spectrum.
One of the biggest things I have learned over the years, from meeting people on the spectrum, is that EVERYONE is different! You will NEVER find two people on the spectrum that are exactly alike. Autism is defined as a disorder in social skills/social interaction, and this can present in so many different ways, there is no exact way to pinpoint a exact list of symptoms that everyone on the spectrum has.
The whole point of a diagnoses of autism/aspergers is so a person can get help with the issues they are facing. If a person isnt seen to be having issues, then they usually dont get the diagnoses. With proper help, most people on the spectrum progress threw out there life, and get much better with there issues with time. This is especially true for many people who are higher functioning, and people with aspergers. Many people who have major issues as a kid, progress to the point that they dont have many issues as adults. One thing is that everyone is different, so it is different for everyone on the spectrum, some dont have many issues as adults, while some people with autism are never able to talk or care for themself!

We dont know what life was like for James Durbin as a kid, we dont know what he went threw, what help he got, or what issues he still has.
We only seen him on a tv show, we dont know what issues he might/might not have in everyday life.

You cant NOT pinpoint a few things and say that this person is not on the spectrum.

Eye contact - there are just as many people on the spectrum that dont have a issue with this, as there are that do. You might find a person who cant communicate, who has no problem keeping eye contact, yet you might find a person with aspergers who cant keep eye contact. Everyone is different

Friends and girlfriends - there is nothing saying that people on the spectrum dont and cant have friends and families of there own! It totally is not uncommon for adults on the spectrum to have family, kids. And that is not just confined to the "mild" aspergers folks, many people with major issues that have the average family and kids.

Everyone one is different, we all have different issues, different strength and weaknesses