Can I legally abandon my construction contract in California?

I am a general contractor in the state of California and began work with a nonprofit organization in June 2020. The director of the nonprofit has failed to honor the payment schedule that was included in our contract from the beginning of me working with him. At times he would write me a check for half the sums stipulated in the payment schedule, and eventually would give me the rest. Since the first week of July I was due a check of $10, 000.00. when I requested to check he only gave me $4, 000.00 of that sum. I did not want to display distrust in my client and I've been patient with him and the breaches of payment schedule before so I went ahead and continued constructing. When I was in need of the remainder $6,000 of the $10,000 schedule payment he began denying me the funds saying that he was unhappy with our attendance to his site based on what he had already paid me. You never proceeded to say that I must be siphoning the money to other projects and that because of this he wanted to buy materials personally and pay myself and my employees our wages daily. I send four emails informing him that the state of California allows me to schedule my work and my staff as I see fit as long as I do not breach our contractual agreement. I made it very clear that despite the offensive accusations and treatment of distrust that my intentions were the same as they've always been to finish his construction and provide him with the highest quality construction possible. I also offer him my expenses record and my Google GPS location history so that you can see that his distrust was based on non-factual ideas. Despite me offering him these unalterable unbiased data sets for his convenience and my attempt at trying to convey that my interest and intentions were to resolve this in an amicable way he has refused to pay me and has never taken me up on the offer of the information that would exonerate me and asked me to remove my tools from his site. At this point I really do not want a headache with this man and would be happy if I can just not deal with him or his project any longer but I'm worried about being liable for illegally abandoning the construction contract. In your professional opinion would I be liable for any charges against me if I decided to not go forward with this contract any longer? Your advice is greatly appreciated thank you very much!