> The Winapi (aka Win32) is the only commonly used API that regularly requires something other than UTF-8 (the Windows Unicode APIs use UTF-16 – not UCS-2 as indicated in the Spolsky article).
Why has Microsoft still not fixed this? They have the various functions [Abc] which call [Abc]A and [Abc]W depending on whether UNICODE is defined, it seems obvious that they should add [Abc]U for UTF-8 and provide a similar define to make those the typedefs for the undifferentiated functions. Even if all the functions did was convert between UTF-8 and UTF-16 and call [Abc]W it would save everyone from having to write and debug their own implementation of the same thing.
> There are a few language+environment combinations that literally can't open Unicode filenames. These include MinGW C++ which has no platform independent way of opening file streams with unicode filenames.
This seems like much the same issue. I don't know if this somehow is the fault of the C++ standard or not, but it seems like there should be a way to specify (if not for the default to be) that any C or C++ standard library functions taking a const char* or std::string should Do The Right Thing when provided with a null-terminated UTF-8 string. If the OS needs something different on the bottom then let the library do the conversion -- half the point of standard libraries is to abstract away things like that instead of making everybody futz with them all the time.
>> The Winapi (aka Win32) is the only commonly used API that regularly requires something other than UTF-8 (the Windows Unicode APIs use UTF-16 – not UCS-2 as indicated in the Spolsky article).
> Why has Microsoft still not fixed this?
Is there a way they can fix it without breaking backward compatibility?
They can, by introducing a few thousand stub functions which convert and delegate to ★W functions, but there'd be little use to do so. Every sane program out there uses the ★W functions and some insane still use the ★A ones. So it would only be beneficial for new code while all existing code remains the same, with the same encoding bugs if there are any.
I'm also not sure whether there are that many cases where it really helps. UTF-8 only on Windows is painful and so is using UTF-8 only with all other things that use UTF-16 (Qt, Java, etc.). Usually in those cases you use a library/framework/whatever that handles the platform abstraction or just conform to what's expected.
> Every sane program out there uses the ★W functions and some insane still use the ★A ones. So it would only be beneficial for new code while all existing code remains the same, with the same encoding bugs if there are any.
Every sane program uses the undifferentiated functions, defines UNICODE so that they map to the ★W functions, and uses TCHAR which defining UNICODE causes to map to a wide char. Older programs don't define UNICODE and often use char (or CHAR) instead of TCHAR. If they would create a different define (e.g. '#define UTF8') which would map to the new UTF8 functions and would define TCHAR as CHAR then anything doing it either way would do the right thing just by defining UTF8 and recompiling. Only programs that explicitly call the ★W functions (which they never should have exposed) wouldn't be "fixed" to use UTF8, but neither would they be broken.
> UTF-8 only on Windows is painful
...because Microsoft hasn't fixed it.
> Usually in those cases you use a library/framework/whatever that handles the platform abstraction or just conform to what's expected.
That's a cop out. You're just deferring to the frameworks, which also shouldn't be using anything other than UTF8, and who may have more difficulty in fixing it because the transition mechanism Microsoft used to unicode is well adaptable to another transition. Not every library you have to use will use the same encoding as the framework and you're back to a huge pain. The only way to fix it is for everything to always use UTF8, and deprecate everything else going forward.
Why has Microsoft still not fixed this? They have the various functions [Abc] which call [Abc]A and [Abc]W depending on whether UNICODE is defined, it seems obvious that they should add [Abc]U for UTF-8 and provide a similar define to make those the typedefs for the undifferentiated functions. Even if all the functions did was convert between UTF-8 and UTF-16 and call [Abc]W it would save everyone from having to write and debug their own implementation of the same thing.
> There are a few language+environment combinations that literally can't open Unicode filenames. These include MinGW C++ which has no platform independent way of opening file streams with unicode filenames.
This seems like much the same issue. I don't know if this somehow is the fault of the C++ standard or not, but it seems like there should be a way to specify (if not for the default to be) that any C or C++ standard library functions taking a const char* or std::string should Do The Right Thing when provided with a null-terminated UTF-8 string. If the OS needs something different on the bottom then let the library do the conversion -- half the point of standard libraries is to abstract away things like that instead of making everybody futz with them all the time.