I hope you saved the voice mail messages. This guy sounds like a "piece of work" (as my Dad would say). He can't sue you....he has no contract.
I'd at least let the neighbors know he did the work (and the name of his company) so if he comes back they can tell you someone was up on your roof if they happen to see him. I wouldn't put it past him to come back and take the repaired part of the roof off!
If he threatened you in any of the messages you can file a police report just to have a paper trail.
Some people can be so ignorant. He puts a whole new meaning behind the word.
BTW....I wouldn't pay him a dime. Or even a penny!