Rendered at 19:20:36 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
joshdavham 2 hours ago [-]
The author Paul McCann (polm) is one of my favourite programmers out there!
He’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects.
He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2].
I think there’s evidence found for the origin of “彁” as the result of a poor scan of a newspaper article. Look up “彁 新聞” to find some japanese sources about this.
sedatk 1 hours ago [-]
Fascinating. But, I guess it's better to have superflous invalid characters than missing real ones.
hnfong 2 hours ago [-]
Well, vast swaths of the Kangxi dictionary (which serves as "sources" for probably most of the CJK characters) are such "ghost" characters as described in the article...
The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
CodesInChaos 2 hours ago [-]
> apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism
Do you mean Han Unification? I.e. that conceptually equivalent characters which are written differently in Japan and China received only a single unicode code-point, and are rendered the Chinese way by default on most computers?
hnfong 2 hours ago [-]
Yes. As far as I understood it, the result was an incoherent mess.
I think some characters got different code points, while others were merged. And apparently the Japanese complained bitterly over the ones that were merged. If you had read any articles about that, this is probably what you have in mind right now.
And I'm also a bit tilted by the ones that had different code points, because when processing CJK text now we have to deal with characters that are (in my native Cantonese) essentially the same, looks similar (to my undiscriminating eyes), yet having different code points so that things like text search sometimes don't work.
Of course I'm not "blaming" the Japanese, if anything the simplified vs traditional Chinese thing is much more of a practical problem, and the conflicting code points I deal with on a routine basis are more of a Hong Kong vs Taiwan thing, but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode (which I think really is some kind of Aristotelian essentialism...) mostly due to vocal objections from the Japanese.
pamcake 1 hours ago [-]
> I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode
What would that philosophy be about? Sounds apocryphal. Unicode has never done "unification" like that for other languages/scripts?
i/ı/i, ö/ø/ø̈/oͤ: Same same, different codepoints.
Search and sorting is a mess everywhere. Depending on your locale, ö sorts either after o or after z. Sometimes it's semantically and phonetically equivalent to o wrt search but moreoften not. https://en.wikipedia.org/wiki/%C3%96
yorwba 16 minutes ago [-]
Well, Unicode didn't unify 門/门 either, but the appearance of the second glyph changes a lot depending on whether you use a Chinese font or a Japanese one. Meanwhile they did unify all the graphical variants of Latin characters with identical semantics, it's just that everyone is so used to the result that you hardly notice not being able to control whether g has a hook or a looped tail, and nobody expects to copy some German blackletter text into a French document and have it show up visually distinct, without explicitly setting the font.
muvlon 13 minutes ago [-]
Meanwhile, Latin a and Cyrillic а, which look identical, have the same origin and are often pronounced identically too, get separate codepoints. CJK unification was not based on any clear preestablished principles but simply an attempt to cling to 16-bit codepoints, Asian languages be damned.
RicoElectrico 57 minutes ago [-]
The Han unification probably was about fitting CJK into the 16 bits aka BMP.
It was a blunder indeed.
philipov 2 hours ago [-]
"- the spectre of communism. All the powers of old encoding have entered into a holy alliance to exorcise this spectre..."
schoen 13 minutes ago [-]
Well, Communism made it into Unicode (U+262D). I tried to show it here but it gets stripped.
He’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects.
He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2].
[0] https://github.com/polm/fugashi [1] https://www.japanesenlp.com/ [2] https://spacy.io/
The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
Do you mean Han Unification? I.e. that conceptually equivalent characters which are written differently in Japan and China received only a single unicode code-point, and are rendered the Chinese way by default on most computers?
I think some characters got different code points, while others were merged. And apparently the Japanese complained bitterly over the ones that were merged. If you had read any articles about that, this is probably what you have in mind right now.
And I'm also a bit tilted by the ones that had different code points, because when processing CJK text now we have to deal with characters that are (in my native Cantonese) essentially the same, looks similar (to my undiscriminating eyes), yet having different code points so that things like text search sometimes don't work.
Of course I'm not "blaming" the Japanese, if anything the simplified vs traditional Chinese thing is much more of a practical problem, and the conflicting code points I deal with on a routine basis are more of a Hong Kong vs Taiwan thing, but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode (which I think really is some kind of Aristotelian essentialism...) mostly due to vocal objections from the Japanese.
What would that philosophy be about? Sounds apocryphal. Unicode has never done "unification" like that for other languages/scripts?
i/ı/i, ö/ø/ø̈/oͤ: Same same, different codepoints.
Search and sorting is a mess everywhere. Depending on your locale, ö sorts either after o or after z. Sometimes it's semantically and phonetically equivalent to o wrt search but moreoften not. https://en.wikipedia.org/wiki/%C3%96