Can Contractor keep harassment up 4 years latter

The contractor used sand to fill 8-foot slope to build a pole barn garage upon. Contract called for gravel, compaction prior to building erection . Original contract for $21k, I gave $8K prior .Contractor dumped sand built on it and was gone in 2 days, then demanded balance. Sand washed out I withheld monies owed to repair, I had all sand around build removed as best could be and dirt and gravel brought in properly compacted and errosion blanketed and seeded. The contractor filed lein which fell off after 1 yr , now 4 yrs later the contractor is insisting on inspecting property and demanding money and having his lawyer demand 21K. I still need the floor jackhammered up and replaced. My questions: Does the contractor have a right to keep coming on property 4 years later and harrassing me. I live in Oneida County NY and contractors do not need to be licensed.Have I done the right thing ,to withhold and use that money to fix his wrongs? Thank you