Consider that the people who behave like that might think they're the polite ones. They listen actively and feel deeply engaged. They see conversation as something everyone participates in; contributes to. From the coworkers' perspective it might seem rude for someone to apparently not listen and just wait their turn to speak, acting as if they think what they have to say is more important than whatever conversation is happening around them. I'm not saying it is that way. I'm just trying to suggest one more charitable way of looking at it.