Internationalise The Fediverse
By @edent
· ActivityPub fediverse i18n mastodon unicode · 35 comments · 750 words · read ~266 times.
We live in the future now. It is OK to use Unicode everywhere.
It seems bizarre to me that modern Internet services sometimes "forget" that there's a world outside the Anglosphere. Some people have the temerity to speak foreign languages! And some of those languages have accents on their letters!! Even worse, some don't use English letters at all!!!
A decade ago, I was miffed that GitHub only supported some ASCII characters in its project names. There's no technical reason why your repo can't be called "ഹലോ വേൾഡ്".
Similarly, I'm frustrated that Mastodon (the largest ActivityPub service) doesn't allow Unicode usernames and has resisted efforts to change.
So I built a small ActivityPub server which publishes content from an Actor called @你好@i18n.viii.fi - it is only a demo account, but it works!
Some ActivityPub clients report that they are able to follow it and receive messages from it. Others - like Mastodon - simply can't see anything from it. Take a look at the replies on Mastodon to see which services work. You can also see some of its posts on the Fediverse.
What Does The Fox Spec Say?
The ActivityPub specification says:
Building an international base of users is important in a federated network. Internationalization
I can't find anything in the specifications which limits what languages a username can be written in. But there are a few clues scattered about.
The user's @ name is defined by preferredUsername which is:
A short username which may be used to refer to the actor, with no uniqueness guarantees. 4.1 Actor objects
There's nothing in there about what scripts it can contain. However, later on, the spec says:
Properties containing natural language values, such as
name,preferredUsername, orsummary, make use of natural language support defined in ActivityStreams. 4. Actors
So it is expected that a preferred username could be written in multiple scripts. Which implies that the default need not be limited to A-Z0-9.
The ActivityStreams specification talks about language mapping.
Finally, the ActivityPub specification has some examples on non-Latin text in names.
So, I think that it is acceptable for usernames to be written in a variety of non-Latin scripts.
But What About...?
There are usually a few objections to "Unicode Everywhere" zealots like me. I'd like to forestall any arguments.
What about homograph attacks?
Well, what about them? ASCII has plenty of similar looking characters. I doubt most people would notice when a capital i is replaced by a lower L - and vice-versa. Similarly the kerning issue of an r and n looking like an m is well known. Are mixed language homographs more dangerous? I don't think so.
What if people make names that can't be typed?
Well, what if they do? Maybe not being found by people who can't type your language is a feature, not a bug. But, anyway, clients can let users search for other people, or copy and paste their names.
What about weird "Zalgo" text?
It is up to a client to decide how they want to render text input. The "problems" of strange Unicode combinations are well known. This is not a hard computer-science problem.
What about bi-directional text?
The spec makes clear this is allowed.
Do people even want a username in their own script?
I have no evidence for this. But I bet you'd get pretty frustrated if you had to switch keyboard just to type your own name, wouldn't you? In any case, why can't I have a username of @😉
What's Next?
If you build ActivityPub software, give some thought to the billions of people who don't have names which easily fit into ASCII.
If your software can see @你好@i18n.viii.fi and its posts, please let me know.
@Terence Eden’s Blog @nĭ hăo Can connect, at least. Perhaps mention works too?
From Hubzilla.
@blog Just to make the answer to "Do people even want a username in their own script?" official:
Yes. Yes we do.
Great work and I hope it catches on! 🙂
@blog Yes! English may have an elevated status in software development, but that should absolutely not translate into any kind of favouritism on the user side. I don’t have enough insight into the technical side of international username support to say if there might be issues you haven’t addressed, but I know that custom emoji gets the ASCII treatment as well, and for no good reason whatsoever. :gutkato_malĝojeta:
#Mastodon and #Firefish have these issues open for the emoji:
Unicode in custom emoji for MastodonUnicode in custom emoji for FirefishUnicode in custom emoji reactions for Pleroma
#Pleroma has basic support the emoji, but lacks support for post language. There are two pull requests to add it, but their importance seems severely underestimated:
Setting post language in PleromaMulti-language posting in Pleroma
Might be a good idea to have open issues and track their status in all the relevant software. It also probably helps if more people talk about this and express support (in appropriate channels!) to show that yes, it is indeed worth it. :sandviĉo:
@blog Yes! Im so annoyed by the arbitrary #anglocentrism!
@blog @Edent Good point. We had to fix one thing (URL encoding the webfinger request) but it now works for remote actors in Bonfire.
@Edent I agree on the issue of homograph attacks - this is bad when you might be communicating/logging in/paying online and you want to make sure you send data to the right place. There, the domain is all you really have to go from and it needs be to be absolutely right, especially when clicking links.
But when it's social media, what's the worst that could happen if you follow olly instead of oIIy?
If there is a vital security issue, punycoded domains names while leaving unicode account names seems like a reasonable compromise (that's why thinks like mastodon domain verification exists).
On the issue of emoji account names though, emoji is an absolute mess and I hate all of it. But you do you. 😤
@blog Tusky on cathode.church (Glitch-social) can't doesn't automatically reply to @你好@i18n.viii.fi, and can't find, when using the search
@Edent unless I have missed something, it is a shame we don’t have a newer ctype.h that can take a char* and isalpha a utf8. You need a similarly low level (that I end up writing myself every time) next char function and so on.
Now tell me I have missed a universal utf8.h that has existed for decades?!?!
My guess though is nothing as light weight as ctype.h though, sadly.
Personally I would mostly be happy to consider anything >= 0x80 as an ongoing identifier character.
@Edent I 100% support this effort. The issue you have is not with ActivityPub but with Webfinger. We are working on the formal specification of AP x WF and I'd love to get your help on i18n here:
https://github.com/swicg/activitypub-webfinger/issues/9 Allowed characters in preferredUsername · Issue #9 · swicg/activitypub-webfinger
@Edent sadly we aren't where we should be... Zoom invites from me still say: join Ren<random crap>e's zoom meeting. And like é seems pretty straight forward.
@blog I have no problem with something like original ASCII for localized English-speaking application or database use. For anything general, or applicable internationally or even worldwide I see no excuse for anything less when we have something suitable for generating bad translations into almost every language! I see no excuse for making anyone code or script in a language foreign to them. This is 2024, we have international solutions for this!
We support utf-8 usernames but I've still got it hidden behind a feature toggle after 10 years. The main reason is that you can't easily mention somebody without having their keyboard available or finding a previous occurrence you can copypasta.
Though I' reminded that we made aliases available for this purpose. so you could make a personal alias to ഹലോ വേൾഡ് called 'joe' and let the mentions autocomplete.
@blog gotosocial 0.13.2 with elk 0.10.3 at least shows the profile and allows me to follow it. the link to the profile in your post functions just fine too.
@blog
hi, hi
just reporting in to say that current versions of GoToSocial can see the account, and probably could see it's posts if not for the lack of backfill
@blog in the Ice Cubes Mastodon client on iOS, I just get this JSON response when tapping the user name:
{"subject":"acct:%E4%BD%A0%E5%A5%BD@i18n.viii.fi","links":[{"rel":"self","type":"application\/activity+json","href":"https:\/\/i18n.viii.fi\/%E4%BD%A0%E5%A5%BD"}]}
@blog I generally agree. Homographs do produce problems in science, though, even in articles written in 'English'. For instance β-carotene is not the same as the non-existing ß-carotene. (The latter, the sz ligature, can all too often be found in the scientific literature, where β is meant. Not a big problem for the human eye, but a big one for machine-readability.)
@blog Not to forget confusing fonts. Fraktur, for example:
@blog
Yeah, the amount of times I ended up having a square in the middle of my surname made me really wary of putting my real name on official documents in the west. Instead I operate under a fake name "Kielinski" instead.
My home-grown ActivityPub server Illuminant managed to follow and receive the follow Accept fine, however it said:
so it failed to fetch the recent posts by @你好@i18n.viii.fi.
@blog
Pleroma, via toot, says:
@blog "This is not a hard computer-science problem."
😂
There is, or at least was for decades, a Cambridge computer science exam question: "Explain why even experienced programmers sometimes have difficulties with character codes."
When that question was originally written the expected answers would have been around things like escape sequences on five track paper tape.
When I did the exam the sort of answer expected might have been to do with whether your code was portable between ASCII and EBCDIC (with the gaps in the middle of the letters, remember?).
These days, your toot would be an answer.
@blog sorry, but text is pretty hard after you start thinking about anything else but the latin alphabet, that's the primary technical motive for why even basic support is lacking.
@blog Tusky opens a webpage with some JSON in it instead. Fantastic. :blobfoxannoyed:
@blog i’m not a fan of this idea tbh solely because of accessibility - not everything supports copypasting fedi handles and there is a reason why most communities demand at least a portion of alphanumeric characters for usernames
BTW before you yell at me; i’m not part of the "anglosphere" and to be frank as someone who’s one of these native speakers of "language with accented letters" this is more useless than a blue checkmark emoji and only brings more issues than the positives
(also lol saying the latin script is "english alphabet" ggwp that’s a nice self own)
@onemuri @blog
Thanks for your repy. Re your comment about a "self own".
The purpose of hyperbole in written text is to convey the ridiculous nature of a statement by making it obviously extreme. For example, I used multiple exclamation marks and preceded it with a couple of other statements of a similar nature.
In doing so, I hoped to lead my reader into understanding that I disagreed with the proposition - as set out by the rest of the post.
I'm sorry if that wasn't clear.
@blog I'm using husky against pleroma. The username is parsed as a link to https://i18n.viii.fi/.well-known/webfinger . It didn't appear in the reply window when typing this up.
@blog in @Tusky when I click on the account link it takes me to the webfinger URL.
@blog I think you're aiming too high when half the payment processors and reservation systems I com into contact with can't even accept a hyphenated name.
@blog No, please don't internationalize usernames.
It will open a whole area of phishing and other kinds of vulnerabilities.
@blog
Homographs are a big security problem, also an easily printable id is needed in many protocols for development, debugging and bug reports. Unless you want to replace ids with qrcodes or similar...
@federico3 @blog
As I mention in the post, ASCll aIready has a H0M0GRAPH problem.
You also pre-suppose that all programmers are able to read A-Z as well as their own alphabet.
But, even if that's not the case, the IDs can be URl encoded.
@blog Another apparent i18n limitation in the Fediverse is that hashtags have an extremely limited character set.
@blog hmph.
Clicking on it just goes to the webfinger URL.
Searching for the
@-form shows nothing.Searching by URL gives:
(this is GotoSocial
mainas of yesterday or so)Personally, I’ve got mixed feelings on this one.
I agree that the localpart should be able to contain Unicode codepoints. Some should be excluded. I don’t know the exact set offhand, but those allowed in URLs (after the server and
/, i.e. in thepathcomponent) should probably be fine.The domain part, however, I’m rather firm on it not deviating from ASCII, i.e. to internationalise it the punycode representation (
xn--something) must be used, not the Unicode representation.So, no complaint against
@☻@example.combut I consider@foo@example.еюinvalid because it needs to be spelt@foo@example.xn--e1a4cinstead. (What clients make of this is up to them, as usual with IDNs… sigh)More comments on Mastodon.