I’m a general contractor. Business is in Virginia.
The short answer is that it depends on the contract. The longer answer is that there is a 3 day recission period required on residential construction. Also, if you haven't ordered anything for which the deposit money was paid, then you haven' t delivered anything to the homeowner and yes you would have to return the money, in my opinion, if you wanted to cancel the contract. All of this is general but these are the basic principals.