Shortly after the Jews escaped from Egypt, Moses went up to Mt. Saini to converse with God and returned carrying two stone tablets on which was engraved the ASCII character set.
Later on, we had computers and those computers ran MS-DOS and Unix (we'll ignore the other operating systems at the moment) and they handled strings as a series of eight-bit bytes. The system used standard ASCII characters for positions 0x00 to 0x7F, and whatever it wanted for 0x80 to 0xFF. Eventually the "whatever it wanted" ended up in logical sets called code pages.
In the late 80's, some people got together to solve this and in the early 90's they released a 16-bit encoding scheme called Unicode. The main problem is that it was binary-incompatible with all existing 8-bit systems, including ASCII. The second problem is that since most computer-readable text files use only ASCII characters, this unnecessarily doubled the size of source code files, configuration files, networking protocol messages, etc.
Microsoft, however, was just beginning work on a brand new operating system kernel called NT, and decided to use the nascent 16-bit encoding scheme in lieu of the older 8-bit character pages.
Eventually though, the Unicode people realized that 216 characters wouldn't be enough for everything they wanted, but the only user of Unicode so far, NT, was already locked into 16-bit characters, so they came up with UTF-16, so that multiple 16-bit characters can specify a code point above the 16-bit range.
The Unicode character set was fairly complete for most common tasks, but the problem was that is still required 16-bit characters.
Then someone got the idea that they could do the same thing as UTF-16 with 8-bit characters, using the bytes above 0x80 to specify Unicode characters and use the lower bytes as ASCII. That way, the encoding would be ASCII-compatible and older systems would just see it as some weird code page. The end result was UTF-8, and it was implemented on Unix as the solution for implementing Unicode.
Now, here's the problem... remember how hard of a conceptual change implementing 16-bit characters on an existing 8-bit operating system was? The same holds true in reverse. NT obligately used 16-bit characters, and with so many other gains over DOS, it was on course to completely replace DOS in the computing world. Every Windows product since XP is based on NT; MS-DOS is gone.
So there we have it, a clbuttic case of jumping on an unproven technology for a major project.
I am not sure how this is going to render for you, but "classic" is showing up as "clbuttic" here. I'm guessing you have an over-eager extension of some sort running?
In Microsoft's defense, the goal was to have better internationalization support, which is a noble goal. At the same time that *nix systems basically just didn't support internationalization, and wouldn't until UTF-8 was adopted. In hindsight UTF-8 is a much better system, but hindsight is 20/20 and I can't blame Microsoft for jumping on UTF-16.
Microsoft, however, was just beginning work on a brand new operating system kernel called NT, and decided to use the nascent 16-bit encoding scheme in lieu of the older 8-bit character pages.
Eventually though, the Unicode people realized that 216 characters wouldn't be enough for everything they wanted,
I think this is the important bit. MS adopted Unicode before most Unix / Linux / systems, and it was hurt by that. When MS first implemented it, Unicode codepoints were 16-bit, there wasn't really an encoding here; it wasn't UTF-16, it was just raw codepoints. (I think this must have also happened to the Java java.lang.String class and the Java char datatype.)
Unix / Linux stayed stuck in the codepage / locale madness that gave us stuff like Shift_JIS and Big5 encodings. When, they finally realized Unicode was available, it has 32-bit codepoints, which no one ('cept AIX) was crazy enough to use directly. And UTF-8 was a great way to avoid changing widely dispersed C code, so they mostly stuck with that.
Then someone got the idea that they could do the same thing as UTF-16 with 8-bit characters, using the bytes above 0x80 to specify Unicode characters and use the lower bytes as ASCII. That way, the encoding would be ASCII-compatible and older systems would just see it as some weird code page. The end result was UTF-8, and it was implemented on Unix as the solution for implementing Unicode.
I think you got the chronology wrong. Isn't UTF-8 older than UTF-16?
You’re correct. UTF-8 was developed first. However, NT was also being separately developed at the same time (NT started in 1989 and was released 1993, UTF-8’s precursor FSS-UTF was first proposed in 1992 and ratified early 1993, far too late in NT’s development to change from 16 bit characters.
Ok so story time.
Shortly after the Jews escaped from Egypt, Moses went up to Mt. Saini to converse with God and returned carrying two stone tablets on which was engraved the ASCII character set.
Later on, we had computers and those computers ran MS-DOS and Unix (we'll ignore the other operating systems at the moment) and they handled strings as a series of eight-bit bytes. The system used standard ASCII characters for positions 0x00 to 0x7F, and whatever it wanted for 0x80 to 0xFF. Eventually the "whatever it wanted" ended up in logical sets called code pages.
In the late 80's, some people got together to solve this and in the early 90's they released a 16-bit encoding scheme called Unicode. The main problem is that it was binary-incompatible with all existing 8-bit systems, including ASCII. The second problem is that since most computer-readable text files use only ASCII characters, this unnecessarily doubled the size of source code files, configuration files, networking protocol messages, etc.
Microsoft, however, was just beginning work on a brand new operating system kernel called NT, and decided to use the nascent 16-bit encoding scheme in lieu of the older 8-bit character pages.
Eventually though, the Unicode people realized that 216 characters wouldn't be enough for everything they wanted, but the only user of Unicode so far, NT, was already locked into 16-bit characters, so they came up with UTF-16, so that multiple 16-bit characters can specify a code point above the 16-bit range.
The Unicode character set was fairly complete for most common tasks, but the problem was that is still required 16-bit characters.
Then someone got the idea that they could do the same thing as UTF-16 with 8-bit characters, using the bytes above 0x80 to specify Unicode characters and use the lower bytes as ASCII. That way, the encoding would be ASCII-compatible and older systems would just see it as some weird code page. The end result was UTF-8, and it was implemented on Unix as the solution for implementing Unicode.
Now, here's the problem... remember how hard of a conceptual change implementing 16-bit characters on an existing 8-bit operating system was? The same holds true in reverse. NT obligately used 16-bit characters, and with so many other gains over DOS, it was on course to completely replace DOS in the computing world. Every Windows product since XP is based on NT; MS-DOS is gone.
So there we have it, a clbuttic case of jumping on an unproven technology for a major project.
hyouko 13 points · Oct 2018
I am not sure how this is going to render for you, but "classic" is showing up as "clbuttic" here. I'm guessing you have an over-eager extension of some sort running?
[deleted] 8 points · Oct 2018
Oh no I did it on purpose because I thought it was funny. https://www.telegraph.co.uk/news/newstopics/howaboutthat/2667634/The-Clbuttic-Mistake-When-obscenity-filters-go-wrong.html
[deleted] 7 points · Oct 2018
That originally came from The Daily WTF.
[deleted] 6 points · Oct 2018
[deleted]
[deleted] 1 points · Oct 2018
There's no 'cloud' in 'classic' :D
Kered13 10 points · Oct 2018
In Microsoft's defense, the goal was to have better internationalization support, which is a noble goal. At the same time that *nix systems basically just didn't support internationalization, and wouldn't until UTF-8 was adopted. In hindsight UTF-8 is a much better system, but hindsight is 20/20 and I can't blame Microsoft for jumping on UTF-16.
[deleted] 6 points · Oct 2018
You completely glossed over EBCDIC.
bss03 6 points · Oct 2018
I think this is the important bit. MS adopted Unicode before most Unix / Linux / systems, and it was hurt by that. When MS first implemented it, Unicode codepoints were 16-bit, there wasn't really an encoding here; it wasn't UTF-16, it was just raw codepoints. (I think this must have also happened to the Java java.lang.String class and the Java char datatype.)
Unix / Linux stayed stuck in the codepage / locale madness that gave us stuff like Shift_JIS and Big5 encodings. When, they finally realized Unicode was available, it has 32-bit codepoints, which no one ('cept AIX) was crazy enough to use directly. And UTF-8 was a great way to avoid changing widely dispersed C code, so they mostly stuck with that.
It's not always best to be a first-mover. :/
jmesmon 2 points · Oct 2018
UTF-32?
peterfirefly 1 points · Dec 2018
I think you got the chronology wrong. Isn't UTF-8 older than UTF-16?
[deleted] 1 points · Dec 2018
You’re correct. UTF-8 was developed first. However, NT was also being separately developed at the same time (NT started in 1989 and was released 1993, UTF-8’s precursor FSS-UTF was first proposed in 1992 and ratified early 1993, far too late in NT’s development to change from 16 bit characters.