And, as of last week, features a small contribution by me!

Look, I'm not an experienced bit-twiddler. I can't micro-optimise algorithms or spot intricate C-based memory leaks. What I can do is get annoyed at poor documentation!

You see, documentation and code comments are vitally important. Poor spelling might trip up non-native speakers, bad examples confuse learners, and ambiguous wording is a barrier to understanding.

As was written by the sages:

a computer language is not just a way of getting a computer to perform operations but rather that it is a novel formal medium for expressing ideas about methodology. Thus, programs must be written for people to read, and only incidentally for machines to execute. Abelson, Sussman, and Sussman Structure and Interpretation of Computer Programs

So, what did I fix? A few years ago, I noticed Google's documentation used example domains it didn't control. The same thing was happening in the curl source code.

One example domain used was "HTTPS://your.favourite.ssl.site" - when that code was written 23 years ago, the .site TLD didn't exist. Now it does.

Is there a serious risk that someone will register ssl.site and use it to take over the machine of anyone who unthinkingly follows that example? Probably not. But it also isn't terribly clear that it is an example. So I changed it to secure.site.example which uses the reserved .example TLD.

That should make it clear to everyone that it is a placeholder example and will prevent anyone from misusing that domain.

Similarly, there were a few comments which used domain.com as an example. However, that's a real website - so I updated those to example.com.

I was delighted to see the changes accepted.

And I was only slightly disappointed to have narrowly missed out on being contributor 1337, but being number 1342 ain't so bad 😄

You can see the full list of changes on GitHub.

Much like my patch to the Linux Kernel this might be considered a trivial matter - but I honestly believe that clear and accurate documentation can be as important as the code itself.

Huge thanks to Daniel for creating curl, for making such a welcoming environment for new contributors, and for handing out such brilliant stickers at FOSDEM!

Drill a hole in part A and insert part B once you have ensured part C has been aligned after its connection to A.

Most people can handle reading a whole sentence to figure out what's going on. But, after a tiring day of building, it is somewhat annoying having to juggle instructions into actions.

Most readers will assume that instructions are written in linear time. Do this, then that. But that example is non-linear. What it is trying to say is:

Connect part C with part A. Then align part C and part A. Then drill the hole in part A. Then insert part B into part A.

It is slightly less interesting writing. But it presents all the actions in the order they need to be taken.

I see this temporally-mixed anti-pattern all the time. A typical example of this in technical documentation is:

Select Print from the File menu.

A simpler, clearer, and less ambiguous way of writing that is:

Open the File menu. Select Print.

Another similar example of confusing writing is:

Go to File → Print → Settings if you need to change the paper size.

Again, this places cognitive burden on the reader. If they want to understand if the instruction is relevant to them, they have to read the entire sentence. When faced with dozens of sentences, this can become confusing. The solution is:

If you want to do X, then do Y...

Immediately the reader knows that they can skip this sentence because they don't want to do X.

As technical writers, we sometimes want to craft eloquent prose. We long for glorious and intricate sentences. We tire of the monotony of linear writing.

Tough. We need to get over ourselves. Go write that epic fantasy novel you've been thinking about. The job of a technical writer isn't to entertain, enliven, or delight the reader. The job is to give them instructions in an easy to follow format, reducing the amount of cognitive burden they have, and making it quick to find the information they need.

In truth, Discord is no harder to sign up to than Slack, Matrix, Gitter, IRC, or whatever. And of course Open Source projects will follow the maxim of "go where your audience are". There's no point posting everything to MySpace when everyone's already on Facebook.

Do I care that Discord isn't open source? Well, kinda. But I can open it in Firefox and it works just fine.

Discord is perfect for ephemeral communications.

But it is not a fucking substitute for documentation!

I'm currently getting started, and increasingly frustrated, with the Watchy development platform. They've effectively said "here's a barebones guide to setting it up - anything else, ask on Discord" - and it fucking sucks.

There's no API documentation - I have to scroll through a million messages to find anything.

I can't use search, because people don't know how to thread. So I can see questions but not replies.

When I do find replies, it's hard to know how relevant they are. A typical Discord chat looks like:

• Alice: What's the command to go fullscreen?
• Bob: Anyone know how I irrevocably format my disk without confirmation?
• Carol: Oh, yeah, it's easy. Just pass the -f flag.

Errrr...

And then you get the people who get snippy with newbie for asking a question which is frequently seen! So infuriating.

I'm not necessarily advocating for the Four-Document Model - which has some critics - but I just don't understand why wouldn't at least collate all of the common questions and put the answers in one place.

Look, writing a FAQ is probably not the right way to approach comprehensive documentation. But if you can't even be bothered to do that, perhaps you shouldn't be releasing a product in the first place?

/rant

Previously I'd been copying and pasting the instructions as I went. One step said "Make sure the bauxite configuration command is set to true" but the code provided said ./configure --magic M --more-magic QxZp --bauxite false --turnip green -z

And there is was! I changed a false to a true and everything started working. Being the good netizen that I am, I sent a pull request to fix the documentation.

And then it struck me.

There's no CI/CD for documentation.

Oh, don't get me wrong. Things like OpenAPI can auto-generate documentation based on your code - but it can't write a "getting started" tutorial.

Looking back through the documentaion I'd encountered, it was clear that the tutorial had been wrong for a few years. It was a small project, so it wasn't hugely surprising that hordes of users hadn't complained. But, to me, it points to a general problem. I find this issue all the bloody time! Whether it is a big established project or a little indie gadget - I follow a tutorial only to find out it is missing a step, or assumes I have a library installed, or hasn't been updated for the latest version.

This is my plea to all developers. Spin up a fresh machine - without anything installed other than the base OS - and see if you can follow your own tutorial.

If you can't, rewrite it step-by-step until it works.

I run OpenBenches - a site which collects memorial benches. When a user adds a bench, the inscription is automatically Tweeted. If the inscription is longer than 280 characters, it is truncated.

The PHP code to truncate text to a specific length is pretty simple:

$tweet_inscription = mb_substr($inscription, 0, 280);

I use Multibyte String operations, because a single character can take up more than one byte. Especially if the text contains characters outside of the "Latin" range.

But this is not sufficient for Twitter.

My friend, Beck Strickland, recently added some gorgeous photographs of benches in Hiroshima, Japan. The inscriptions were pretty long, and were not being Tweeted.

The error I was getting back from Twitter was "186: Tweet needs to be a bit shorter."

Here's the Japanese text:

まち を 緑でい っ ぱ い に ! 広島ゾンタクラブ 医療法人あさだ会 有設計工房岩重老沼薬品 広島市立山田小学校広島県立皆実高校 小田耳鼻咽喉科医院 翠清会梶川病院 広島大学総合科学部 片桐眼科 広島歯科医院 しいのレディースクリニック 石橋三千男事務所 Akemi Engish School 高橋内科小児科 広島総合病院 川村,大迫法律事務所 (株)山本薬品 ㈲)メディカルサービス 老人保険施設ベルローゼ スタッフ·トゥー·ワン 原田病院 広島大学医学部 ひろしま通訳·ガイド協会 有森信パーキングビル (株)幸房

Go paste that into any character counter. Even after newlines, it's a heck of a lot less than 280. But Twitter disagrees!

What this problem is not

I wasted a lot of time looking at esoteric Unicode Normalisation Forms as directed by the Twitter documentation on Counting Characters.

Tweet length is measured by the number of codepoints in the NFC normalized version of the text.

That wasn't the answer.

The documentation did point to a page about the Twitter Text Parsing Library. Which, helpfully, has this to say on the matter:

The Configuration defines Unicode code point ranges, with a weight associated with each of these ranges. This enables language density to be taken into consideration when counting characters.

Aha! Japanese is a fairly dense language. The English word Monday has six characters, the Japanese translation 月曜 has two.

When Twitter announced the increase to 280 characters they published a blog post saying:

We want every person around the world to easily express themselves on Twitter, so we're doing something new: we're going to try out a longer limit, 280 characters, in languages impacted by cramming (which is all except Japanese, Chinese, and Korean).

やった!

Loving the algorithm

When I first wrote this post, there was nothing that I could find in the official Twitter Developer Documentation which explained how these weights are calculated. Aside from the above blog post, I couldn't find anything which explained that CJK characters are counted differently.

The Twitter Text library is open source, but doesn't explicitly spell out how the weightings are calculated. There are implementations in Ruby, JS, Objective C, and Java. Thankfully, Takashi Nojima maintains a PHP version.

Because Tweets can contain a mixture of languages, Twitter has developed a "Weighted Length" algorithm. For example "石橋三千男事務所 Akemi Engish School" looks to have 28 characters but, because some of them are in a higher density script, Twitter gives a weightedLength of 36.

This makes truncating Tweets automatically somewhat difficult!

10 Is weightedLength less than 280?
20 Yes - Tweet it.
30 No - remove one character from the end.
40 GOTO 10

OK, it isn't quite that bad. The PHP implementation lets you find the valid length of the string using:

$data = \Twitter\Text\Parser::create()->parseTweet($tweet_inscription);
echo $data->validRangeEnd;

Is everything you know about Twitter character counting wrong?

An inflammatory clickbait title? I don't think so. There was literally no way of knowing how the character count algorithm works from reading the developer documentation.

How do you count the characters in an emoji? Do you count the skin tone and gender modifiers separately? Again, the developer documentation didn't say anything.

Developer documentation is essential. And it needs to be kept up to date with the latest changes. I'm friends with some of the folk on the Twitter Developer Team (with friends like this, right?!) and I shared my concerns with them.

Since writing this post, I've been working with them to improve the developer documentation. I'm happy to say that the documentation now reflects reality.

The counting characters page now explains how to count CJK characters and complex emoji. Enjoy!

This sort of knowledge is common in life - but is fatal in computing and design. Take the following tweet I received.

Mark Hawkins

@Mawkins

Replying to @edent @edent @dabr you folks aware ampersands / &s don't seem to work as part of hashtag links?

---

The complaint was that #tfm&a should be rendered as #tfm&a not #tfm&a.

Everyone knows that's how hashtags work!

On Twitter's website, find the page which discusses hashtag syntax. Find where they explain how they should be styled.

You can't.

And thus implicit knowledge is born. Dabr only looks at letters and numbers in a hashtag. It assumes that any other character is the end of the tag.

Dabr's Hashtag

Without official guidance - implicit knowledge develops.

Has Dabr Got It Wrong?

No. I don't think so. Take a look at how Twitter on the web renders hashtags...

Twitter's Web Site

...and on the mobile.

Twitter Mobile

So Where Does Render The Full Tag?

Several applications don't render tags in the same way as Twitter. Take a look at SocialScope

SocialScope Hashtags

Tweetie2

I'll upload more screenshots if I find examples of "badly behaved" hashtags.  Please let me know if you find any.

What Does Twitter Say?

Twitter has one page devoted to hashtags.  It is a support page for hashtags.  This explains to people what hashtags are.  There's no detail on valid characters, maximum length, or any of the things which might be useful for a developer or designer.

Edit 2010-02-25

David Dorward has pointed out that there is an official resource. On the Twitter Engineering blog - which isn't linked to from the developer site - there is a page discussing hashtags and how to validate them. You'll notice that they are rather circumspect on what should constitute a hashtag.

Conclusion

Standards and guidelines allow developers to create compatible applications.

Without explicit recommendations, developers will diverge as widely as possible.  Twitter - and everyone with an interest in compatibility and usability - needs to ensure that the knowledge they impart is explicit.

Letting people make it up as they go along leads to confusion.