While one can make the case that ত্য is simply "'to' - 'o' + 'ya' = 'to'"[0][1], it's rather confusing mental acrobatics, and it doesn't reflect either how the writing system is taught, or how native speakers use it and think of it on a day-to-day basis.

If anything, your comment makes a stronger argument for consolidating ই and ি (they are literally the same letter and phoneme, but written differently in different contexts) than for combining the viram and য into the jophola.

[0] To non-Bengali speakers reading this, yes, this is how that construction would work, and yes, I am aware that the arithmetic doesn't appear to add up (which I guess is part of the point).

[1] Also, now that I think about it, the য is a consonant, not a vowel, so using it in place of a vowel is doubly awkward. This is particularly an issue in Bengali, where sounds that might be consonants in English (like "r" and "l") can be either consonants or vowels in Bengali, depending on the word.

Yes.

> [...] it's rather confusing mental acrobatics, and it doesn't reflect either how the writing system is taught, or how native speakers use it and think of it on a day-to-day basis.

Mental acrobatics are part-and-parcel of the language, either in digital or non-digital form. If I were to spell out your name aloud, I would end with "ত-এ য-ফলা", which doesn't really say anything about how ত্য is pronounced. While writing on paper, we say "ক-এ ইকার", and then we reorder what we just said to write the ইকার in front of the ক. Even more complicated mental acrobatics - we say ক-এ ওকার, and then proceed to write half of the ওকার in front of the ক and then the other half, after the ক. We don't necessarily think about these when we carry out these acrobatics in our head, but they exist, and we have made the layer on top of the encoding system (rendering, and to some extent, input) deal with these acrobatics as well. My point in the original comment (and to some extent in the preceding one) was to emphasize that a lot of these issues are at the input method level - we should not have to think about encoding as long as it accurately and unambiguously represent whatever we want it to represent.

Just out of curiosity - I would be interested to know more about your learning experience that you feel is not well aligned with the representation of jophola as it is currently.

I might be sympathetic to this, except that keyboard layouts and input (esp. on mobile devices) is an even bigger mess and even more fragmented than character encoding. Furthermore, while keys are not a 1:1 mapping with Unicode codepoints, they are very strongly influenced by the defined codepoints.

It'd be nice to separate those two components cleanly, but since language is defined by how it's used, this abstraction is always going to be very porous.

> Just out of curiosity - I would be interested to know more about your learning experience that you feel is not well aligned with the representation of jophola as it is currently.

I have literally never once heard the jophola referred to as a viram and a য, except in contexts such as this one. Especially since the jophola creates a vowel sound (instead of the consonant য), and especially since the jophola isn't even pronounced like "y". (I understand why the jophola is pronounced the way it does, but arguing on the basis of phonetic Sanskrit is a poor representation of Bengali today - by that point, we might as well be arguing that the thorn[0] is equivalent to "th" today, or that æ should be a separate letter and not a dipthong[1])

I'm not going to claim that it's completely without pre-digital precedent, but it certainly is not universal, and it's inconsistent in one of the above ways no matter how one slices it, especially when looking at some incredibly obscure and/or antiquated modifiers that are given their own characters, despite being undeniably joined of other characters that are already Unicode codepoints[2].

(Out of curiosity, where in Bengal are you from?)

[0] https://en.wikipedia.org/wiki/Thorn_%28letter%29

[1] As it was in Old English.

[2] Such as the ash (æ)!

For Japanese (and Chinese and a few others) things are even more complicated. It's tricky to fit ~5000 symbols on a keyboard, so typically in Japan one types on either a phonetic layout, or a latin layout, and translate to Kanji as needed (eg: "nihongo" or "にほんご" is transformed to "日本語" -- note also that "ご" itself is a compound character, "ko" modified by two dots to become "go" -- which may or may not be entered as a compound, with a modifier key).

As I currently don't have any Japanese input installed under xorg, that last bit I had to cut and paste.

It is entirely valid to view ø as a combination of o and a (short) slash, or å as a combination of "a" and "°" -- but if one does that while typing, it is important that software handles the compound correctly (and distinct from ligatures, as mentioned above). My brother's name, "Ståle" is five letters/symbols long, if reversed it becomes "elåtS", not "el°atS" (six symbols).

So, yeah, it's complicated. Remember that we've spent many years fighting the hack that was ascii, extended ascii (which may be (part of) why eg: Norwegian gets to have å rather than a+°). You still can't easily use utf8 with neither C nor, as I understand it C++ (almost, but not quite -- AFAIK one easy workaround is to use QT's strings if one can have qt as a dependency -- and it's still a mess on Windows, due to their botched wide char hacks... etc).

All in all, while it's nice to think that one can take some modernized, English-centric ideas evolved from the Gutenberg press, and mash it together with what constitutes a "letter" (How hard is it to reverse a string!? How hard is it to count letters!? How hard is it to count words?!) -- that approach is simply wrong.

There will always be magic, and there'll be very few things that can be said with confidence to be valid across all locales. What is to_upper("日本語"), reverse("Ståle"), character_count("Ståle"), word_count("日本語") etc.

This turned into a bit more of an essay than I intended, sorry about that :-)

    for(int i = 0; i < string.length();) {
        final int codepoint = string.codePointAt(i);
        i += Character.charCount(codepoint);
    }

The correct version is:

  for (int i = 0; i < string.length(); i = string.offsetByCodePoints(i, 1))
  {
     int codepoint = string.codePointAt(i);
  }

But this just adds to the fact, proper Unicode string processing is a pain :).

Encoding was in a similar place 10-15 years ago. Almost every publisher in Bengali had their own encoding, font, and keyboard layout - the bigger ones built their own in-house systems, while the smaller ones used systems that were built or maintained by very small operators. To make things even more complicated, these systems needed a very specific combination of operating system and page layout software to work. Now the situation is quite better with most publishers switching to Unicode, at least for public facing content.

With input methods, I expect to see at least some consolidation - I don't necessarily think we need standards here, but there will clear leaders that emerge. Yes, keyboard layouts are influenced by Unicode code-points, but only in a specific context. Usually when people who already have experience with computers start to type in Bengali (or any other Indic language), they use a phonetic keyboard, which is influenced mostly by the QWERTY layout. Then, if they write a significant amount, they find that the phonetic input is not very efficient (typing kha everytime to get খ is painful), and they switch to a system where there's a one-to-one mapping between commonly used characters and keys. This does tend to have a relationship between defined codepoints and keys, but that's probably because the defined codepoints cover the basic characters in the script (so in your case, ্য would need to have a separate key, which I think is fine). There will still be awkward gestures, but that's again, a part of adjusting to the new medium. No one bats an eyelid when hitting "enter" to get a newline - but when we learn to write on paper, we never encounter the concept of a carriage-return.

> I have literally never once heard the jophola referred to as a viram and a য

Interesting - I guess we have somewhat different mental models. For me, I did think of jophola as a "hoshonto + jo", possibly because of the "jo" connection, and this was true even before I started to mess around with computers or Unicode. I always thought about jophola as a "yuktakshar", and if it's a "yuktakshar", I always mentally broke it down to its constituents.

> [...] especially when looking at some incredibly obscure and/or antiquated modifiers that are given their own characters

I think those exist because of backwards compatibility reasons. For Bengali I think Unicode made the right choice to start with the minimum number of code points (based on what ISCII had at that time). As others have pointed out elsewhere in the thread - it is an evolving standard, and additions are possible. Khanda-ta did get accepted, and contrary to what many think, non-consortium members can provide their input (for example, I am acknowledged in the khanda-ta document I linked to earlier, and all I did was participate in the mailing list and provide my suggestions and some evidence).

> Out of curiosity, where in Bengal are you from?

কলকাতা

A better question is, Are there any native Bengali speakers creating character set standards in Bangladesh or India? If not, why not? If so, did they omit your character?

I ask, because although you prefer to follow the orthodox pattern of blaming white racism for your grievance du jour, the policy of the Unicode Technical Committee for years has been to use the national standards created by the national standards bodies where these scripts are most used as their most important input.

Twenty years ago, I spent a lot of time in these UTC meetings, and when the question arose as to whether to incorporate X-Script into the standard yet, the answer was never whether these cultural imperialists valued, say, Western science fiction fans over irrelevant foreigners, but it was always, "What is the status of X-Script standardization in X-land?" Someone would then report on it. If there was a solid, national standard in place, well-used by local native speakers in local IT applications, it would be fast-tracked into Unicode with little to no modification after verification with the national authorities that they weren't on the verge of changing it. If, however, there was no official, local standard, or several conflicting standards, or a local standard that local IT people had to patch and work around, or whatever, X-Script would be put on a back burner until the local experts figured out their own needs and committed to them.

The complaint in this silly article about tiny Klingon being included before a complete Bengali is precisely because getting Bengali right was more complex and far more important. Apparently, the Bengali experts have not yet established a national standard that is clear, widely implemented, agreed upon by Bengali speakers and that includes the character the author wants in the form he/she wants it, for which he/she inevitably blames "mostly white men."

(Edited to say "he/she", since I don't know which.)

It's a bit telling that folks in the software industry[1] seem to assume that techies are male (a priori), but those who write articles of this kind are female.

Not blaming you for it, but it's something you should try to be conscious about and fix.

[1] I've been guilty of this myself, though usually in cases where I use terms like "guys" where I shouldn't be.

However, I clearly do need to question my assumption that since this was a female name before, it's a female name now, so I should change it to "he/she".

Oh, sorry about that. Not sure if you're joking about the assumption of assumptions, but asking people to take note of their behavior based on something that they _might_ have assumed is not dangerous. Assuming gender roles is. Apologies for making that assumption, but IMO it's a rather harmless one so I don't see anything to fix about it :P

This is factually incorrect. It seems you missed both the factual point about the Klingon script in the article as well as the broader point which that detail was meant to illustrate.

> although you prefer to follow the orthodox pattern of blaming white racism for your grievance du jour, the policy of the Unicode Technical Committee for years has been to use the national standards created by the national standards bodies where these scripts are most used as their most important input.

There's a huge difference between piggybacking off of a decades-old proposed scheme which was never widely adopted even in its country of origin, and which was created under a very different set of constraints than Unicode, and which was created to address a very different set of goals than Unicode, versus making native speakers an active and equal part of the actual decision-making process.

Rather than trying to shoehorn the article into a familiar pattern which doesn't actually fit ("orthodox pattern of blaming white racism for your grievance du jour"), please take note that the argument in the article is more nuanced than you're giving it credit for.

As I explained, native speakers are the primary decision makers, and not just any native speakers but whoever the native speakers choose as their own top, native experts when they establish their own national standard. For living, natural languages, you don't get characters into Unicode by buying a seat on the committee and voting for them. You do it by getting those characters into a national standard created by the native-speaking authorities.

So, I repeat: What national standard have your native-speaking authorities created that reflects the choices you claim all native speakers would naturally make if only the foreign oppressors would listen to them? If your answer is that the national standards differ from what you want, then you are blaming the Unicode Technical Committee for refusing to override the native speakers' chosen authorities and claiming this constitutes abuse of native Bengali speakers by a bunch of "mostly white men".

No, the ultimate decision makers of Unicode are the voting members of the Unicode Consortium (and its committees).

> For living, natural languages, you don't get characters into Unicode by buying a seat on the committee and voting for them. You do it by getting those characters into a national standard created by the native-speaking authorities

As referenced elsewhere in the comments, there are plenty of decisions that the Unicode Consortium (and its committees) take themselves. Some of these (though not all) take "native-speaking authorities" as an input, but the final decision is ultimately theirs.

There's a very important difference between being made an adviser (having "input") and being a decision-maker, and however much the decision-makers may value the advisers, we can't pretend that those are the same thing.

But the character set WAS designed by native speakers, by experts chosen not by the UTC but by the native speaking authorities themselves. The UTC merely verified that these native speaking experts were still satisfied with their own standard after using it for a while, and when they said they were, the UTC adopted it.

You go on about how the real issue is the authority of these white men and how the native speakers are restricted to a minor role as mere advisers, and yet the native speakers, as is usually the case, had all the authority they needed to create the exact character set that THEY wanted and get it adopted into Unicode. That's the way the UTC wants to use its authority in almost all cases of living languages.

Unfortunately for your argument, these native speakers didn't need any more authority to get the character set they wanted into Unicode. They got it. You just don't like their choices, but you prefer to blame it on white men with authority.

The former group would give each "semantic character" its own codepoint, even when that character is "mappable" to a character in another language that has the same "purpose" and is always represented with the same grapheme (see, for example, latin "a" vs. japanese full-width "a", or duplicate ideograph sets between the CJK languages.) In extremis, each language would be its own "namespace", and a codepoint would effectively be described canonically as a {language, offset} pair.

The latter group, meanwhile, would just have Unicode as a bag of graphemes, consolidated so that there's only one "a" that all languages that want an "a" share, and where complex "characters" (ideographs, for example, but what we're talking about here is another) are composed as ligatures from atomic "radical" graphemes.

I'm not sure that either group is right, but trying to do both at once, as Unicode is doing, is definitely wrong. Pick whichever, but you have to pick.