The only thing that would be criminal is if it was intentionally defrauding the SS administration. If they find that you are not disabled you may have to pay them back, but it sounds like you have some pretty bad back problems. Do they have any video of you doing the yard work? Be careful in the future, with people like that around you don't need enemies.

This is a case where a women defrauded SSD, of course this is the extreme from your case but this is just one of many cases of what can happen.

Cessation of Disability Benefits Paid to Apparently Healthy Young Woman
Our investigation of a 24 year-old Title II disability beneficiary, who was entitled based
on asthma, seizures, and stress, revealed that she was attending a 4-year college and was
participating in school sports during the period she was receiving benefits. In fact, the
woman received a Bachelor of Science degree in Accounting in December 2002 and was
an active member of the basketball team for 4 years, also playing soccer and running track.
Her basketball coach stated the woman received a physical exam each year which showed
she was in good physical and mental health. e coach also stated she participated in daily
practices with no apparent problems. Videotapes of the claimant playing in college basketball
games were obtained. CDI investigators interviewed the woman, who admitted making false
statements to the doctors to continue receiving her disability benefits. e woman’s benefits
were ceased and, because fraud was detected, there was a reopening/reversal of the initial
allowance in 1999, which resulted in an overpayment of $30,018. In June 2004, the woman
was indicted on one count of Social Security fraud, to which she pled guilty. In January 2005,
she was sentenced to 5 years’ probation.