I also think the argument isn't put forward in the best logical way, but for a different reason: another potential reason why aliens might not choose to come to visit us is that they physically can't: imagine a lifeform whose chemistry is based on liquid methane instead of water. They'd literally vaporize in Earth's atmosphere!

They could also operate on a completely different timescale. For example, shrews' metabolism is way faster than ours. Sequoias are way slower. There's no reason an alien should perceive time on the same scale we do.