> A utf8 decoder should reject portions of utf8 streams that don't use the shortest possible encoding
so you would say that there should be no file names using the cyrillic o? So if a russian-speaking person wants to save a file, that file name should be rejected? Or translated into a mish-mash between cyrillic and roman characters?
How will that work if that filename is reused on a system on which the default font doesn't contain the roman characters (I'm sure such a thing exists) and thus font substitution needs to happen?
The fix definitely isn't this easy. Maybe one could disallow homoglyphs of a different language than the one dominating the current file name. But this might be a lot of work and I doubt it's fool-proof.
> A utf8 decoder should reject portions of utf8
> streams that don't use the shortest possible encoding
so you would say that there should be no file
names using the cyrillic o?
I'm sorry, I was unclear. I should have said "don't use the shortest possible encoding for a code point". Cyrillic 'o' is code point U+043E while roman 'o' is code point U+006F. The canonicalization attack relies on overly liberal utf8 decoders that would allow multiple binary streams to be interpreted as, say, code point U+006F.
This looks like the canonicalization attack, but is a different problem, one that is not solved by fixing decoders.
That's not the point. The attack relies on some nonconforming decoders exhibiting a many-to-one mapping of bitstreams to codepoints, changing the semantics of the bistream.
you shouldn't restrict anything - just give the users enough hints to be more informed.
I would highlite the background of any character that is not from the users codepage in red.
for eg. if your local settings are us-en, any character not from that codepage will have a red background (or even in italics, some way to signify that the character is 'foreign')
so you would say that there should be no file names using the cyrillic o? So if a russian-speaking person wants to save a file, that file name should be rejected? Or translated into a mish-mash between cyrillic and roman characters?
How will that work if that filename is reused on a system on which the default font doesn't contain the roman characters (I'm sure such a thing exists) and thus font substitution needs to happen?
The fix definitely isn't this easy. Maybe one could disallow homoglyphs of a different language than the one dominating the current file name. But this might be a lot of work and I doubt it's fool-proof.