If anything is different between the two resulting configurations of possibly affected particles, such as the state of the particles of the detector, then interference can't happen. It's not just about whether the individual particle going through one of the slits is in an identical location.

An important thing to realize is that interference is a thing that happens between whole configurations of affected particles, not just between alternate versions of a single particle going through the slit.