Restricting freedom of bad actors means enhancing freedom of everyone else.

Say a a kid started throwing tantrums at school. By not punishing/ removing him you restrict the freedom of everyone else.