and before you say something about Asian languages, the space savings of UTF-16 aren't as significant as you might think, once you mix in the markup/JS syntax and compression
So, in other words, the space savings for Asian languages aren't significant if you don't use Asian languages? Not an argument, dumbass.
What? HTML, JavaScript, and CSS use ASCII characters for all of their syntax and standard identifiers. If you're making a webpage with CJK content, the fact is that you're going to have a ton of ASCII characters in it, each of which is twice as compact in UTF-8 compared to UTF-16.
Let's look at some examples, measuring the size of the homepage's HTML source, before compression:
yahoo.co.jp in Japanese: In UTF-16, it's 471.0 kB. In UTF-8, it's 240.6 kB.
nicovideo.jp in Japanese: In UTF-16, it's 72.0 kB. In UTF-8, it's 38.9 kB.
qq.com in Chinese: In UTF-16, it's 426.1 kB. In UTF-8, it's 237.2 kB.
naver.com in Korean: In UTF-16, it's 330.2 kB. In UTF-8, it's 174.6 kB.
the fact is that you're going to have a ton of ASCII characters in it
So fucking what? That HTML is biased towards Latin1 is not an argument to penalize CJK even more. Presumably CJK people want to read books n'shiet too, or are you going to deny them that as well? (Not that that's an argument for UTF16, of course. UTF16 is a shit idea.)
Using just the text contents of this article as an example, and applying traditional deflate compression, gives me these results:
UTF-8: 13.6 kB
UTF-16: 12.5 kB
Here's the same content, using bz2 compression instead:
UTF-8: 10.4 kB
UTF-16: 10.9 kB
This is pretty much a best-case scenario for UTF-16: pure Japanese text. And yet, once you apply some modern compression, UTF-8 still ends up smaller than UTF-16. This is because UTF-8 is generally more compressible than UTF-16.
So, in other words, the space savings for Asian languages aren't significant if you don't use Asian languages? Not an argument, dumbass.
Nanobot 4 points · Dec 2018
What? HTML, JavaScript, and CSS use ASCII characters for all of their syntax and standard identifiers. If you're making a webpage with CJK content, the fact is that you're going to have a ton of ASCII characters in it, each of which is twice as compact in UTF-8 compared to UTF-16.
Let's look at some examples, measuring the size of the homepage's HTML source, before compression:
yahoo.co.jp in Japanese: In UTF-16, it's 471.0 kB. In UTF-8, it's 240.6 kB.
nicovideo.jp in Japanese: In UTF-16, it's 72.0 kB. In UTF-8, it's 38.9 kB.
qq.com in Chinese: In UTF-16, it's 426.1 kB. In UTF-8, it's 237.2 kB.
naver.com in Korean: In UTF-16, it's 330.2 kB. In UTF-8, it's 174.6 kB.
diggr-roguelike2 0 points · Dec 2018
So fucking what? That HTML is biased towards Latin1 is not an argument to penalize CJK even more. Presumably CJK people want to read books n'shiet too, or are you going to deny them that as well? (Not that that's an argument for UTF16, of course. UTF16 is a shit idea.)
Nanobot 4 points · Dec 2018
Using just the text contents of this article as an example, and applying traditional deflate compression, gives me these results:
UTF-8: 13.6 kB
UTF-16: 12.5 kB
Here's the same content, using bz2 compression instead:
UTF-8: 10.4 kB
UTF-16: 10.9 kB
This is pretty much a best-case scenario for UTF-16: pure Japanese text. And yet, once you apply some modern compression, UTF-8 still ends up smaller than UTF-16. This is because UTF-8 is generally more compressible than UTF-16.
diggr-roguelike2 0 points · Dec 2018
I never advocated for UTF-16.