That's what I'm saying. Pronouns (of all types, not just gender) should convey the speakers perception and intent, so there's nothing to keep track of. Think of vous/tu or thou/you. If you have kids, it's the difference between saying to your spouse "our kid drew all by himself" and "your kid drew on the wall".
The pronoun selection conveys meaning perceived by the speaker, and not always the literal categorical noun substitution. But it's for the speaker to decide.
I'll leave you with this. An older gay friend was involved in gay rights stuff back in the 1970's. He said there were two kinds of activists in the organizations. Those that wanted to achieve goals and those that wanted to involve themselves in internecine politics. His experience is the latter win.
Because no one sane can keep up outside of a small personal group of friends.