How to let a Contractor go?

Absent any contract, you can tell the contractor to take a hike at any time, and he will have no success pursuing any employee claims, such as unemployment, against you because he wasn't an employee. All you have to do is tell your side of the story if it becomes necessary.