I had something similar. Went to an appointment, nobody in, but spoke to two blokes removing an emergency generator. The place was off the beat and track and would only ge there if you were going there sort of thing. Anyway, bloke phones a couple of days later, its disgusting that you didnt turn up, etc , etc. Said I was there, but nobody home, spoke to the blokes removing the emergency generator etc. He told me that he would tell everybody that I was useless and didnt turn up as planned. Touch of red mist hit me, said I would rip his head off and crap down his neck and actually went down there but he wouldnt open the door. Two days later I had plod on the phone about my threatening behaviour. Apparently the bloke had left all the detail out about the conversation with regard to the bad mouthing. Plod asked me to be more diplomatic in future and were going back to give him a bulling.
Upshot is, keep it firm, keep it, clean and keep it legal. But chase it down.