Asking a non-employee to leave the premises of a business isn't treating them like a "potential criminal". If you find a random visitor wandering around your office, you may ask them to leave. It's not a "shitty" thing to do as long as it's done politely.

Let's keep in mind this is also someone who is leaving voluntarily, not being laid off. They are picking their moment when they become a non-employee.