Fifteen years ago (fuck, I'm old) I started documenting what Twitter's nascent hashtags could and couldn't do.

Back in 2010, this is how the official Twitter site linked hashtags.

Notably, punctuation symbols didn't "count" as part of a tag.

How does modern social media handle something like #Fish&Chips?

• Mastodon links directly to #Fish&Chips
• BlueSky links directly to #Fish&Chips
• Threads links to a search for Fish & Chips

What about normalisation?

Should #Romeo link to #ROMEO and #rOMeO?

On all three of the major social networks, case is insensitive.

But what about the vagueries of Unicode normalisation?

Is #Ŕöméø&Jülíèt the same as #Romeo&Juliet?

Both Threads and Mastodon do some form of decomposition - turning the various accents into their accentless versions.

But BlueSky links to the literal version.

Is that the right thing to do? I don't know.

This literal interpretation of the text in hashtags allows for some interesting steganography - which can be fun, but I wonder if it is what users expect?

And that's what it comes down to. What is technically correct isn't always the same as what users need.

Perhaps most users prefer #ROMEO to link to the same posts as #romeo. Perhaps they think #Romeó should link there too. But no social network, as far as I am aware, has done any user research into the behaviour that users want when interacting with hashtags.

I'd love someone to do some actual research on how people expect a folksonomy to work.

#hashtag – As long has there has been a way to search Tweets* people have been adding information to make the easy to find. The #hashtag syntax has become the standard for attaching a succinct tag to Tweets.

<

p>The Twitter Engineering Blog

That's all well and good, but as I discovered yesterday, without standardisation the ability to search falls apart.

I'm not talking about whether you should use the #LondonFire tag rather than #FireOfLondon or #LDNfire. Rather; how does a computer recognise what a valid tag is?

Why Does This Matter?

Search and tracking quickly break down if they are inconsistent. For example, if you are using #Romeo&Juliet to mark all your conversations about the play you are watching, different Twitter clients will link through to either #Romeo, #Romeo&, or #Romeo&Juliet.  Each search returning potentially different conversations.

What's The Convention?

Twitter's website ought to be the definitive source of how hashtags work. This is their main site.

Yet, when we visit their mobile site - we get a completely different experience.

Application Confusion

Because there aren't any widely publicised definitions for what hashtags are, some applications have a significantly different attitude to hashtags

SocialScope Hashtags

UberTwitter's Hashtag Support

Standardisation

To be fair, the Twitter team do have a standard.  Even if they don't use it themselves.

They even have some limited test cases and libraries in Ruby and Java.

So, given that Twitter, their implementation and apps all disagree on what a hashtag is, let's try to work our what they should be.

Anatomy of a Tag

To begin at the beginning.  A hashtag starts with a hash. #.  Simple, no? No.

There are two different hash symbols! There's the # we all know and love, and there's ＃.  Looks pretty similar, but in fact it's the unicode symbol [U+FF03]

Actually, that's not the beginning.  What comes before the # of the hashtag?

Consider the following examples - which should be hashtags?

• #tag - the # starts off the Tweet
• This is my tweet #test - the # comes after a space.
• This is it.#tag - the # is pushed against some punctuation, perhaps for reasons of space.
• Here we go-#LiftOff - the # is pushed against a -
• I've run out of space#OhNo - the # is pushed against some text
• &#nbsp; - the # is part of an HTML entity
• text　#hashtag - the # comes after a "wide space" (U+3000)
• Should I use #tag/#hashtag? The # comes after a /
• Is this valid ##tag - there are two #s

So, we can see it's a little more complicated than we first thought.

The End

Let's skip over what's in a hashtag and as "how do we know that a tag has finished?"

Consider the following examples -

• New album #OMG! - should the ! be part of the hashtag?
• #BreakingNews: dog bites man - should the : be part of the hashtag?
• (is this a #tag) - should the ) be part of the hashtag?
• I like #tags#

We probably don't want to have any punctuation at the end of our tag.  Can you think of any counter examples?

Yummy Filling

Our language is more than just the letters A-Z. We've got punctuation, numbers, symbols and all manner of other glyphs.  Which of them count as part of a hashtag?

Take a look at these examples

• Vote Bush! #Don't
• My dog died #:-(
• Einstein #e=mc^2
• I'm on bus #123
• I'm giving #110%

Using Twitter's standards, none of the above render as complete tags.

Foreign Languages

We've mentioned accents above.  As we can see in the first example, "funny" characters can cause problems.  Broadly speaking, there are three issues.

1. Accents.  Should the é on #Café be linked?
2. Accents.  Is #Romeo the same as #Ŕöméø?
3. Japanese, and some other languages, don't use spaces.  Is #tagの valid? What about # 会議中 ?

Exhausted

These are a fraction of the possible problems.  It's exhausting trying to find all the possible textual combinations and permutations which could lead into a hashtag.  No wonder there is confusion!

Search is a complex, profitable, and useful business.  It's of vital importance that there is a legitimate, comprehensive standard which all sites and applications can follow.

This sort of knowledge is common in life - but is fatal in computing and design. Take the following tweet I received.

Mark Hawkins

@Mawkins

Replying to @edent @edent @dabr you folks aware ampersands / &s don't seem to work as part of hashtag links?

---

The complaint was that #tfm&a should be rendered as #tfm&a not #tfm&a.

Everyone knows that's how hashtags work!

On Twitter's website, find the page which discusses hashtag syntax. Find where they explain how they should be styled.

You can't.

And thus implicit knowledge is born. Dabr only looks at letters and numbers in a hashtag. It assumes that any other character is the end of the tag.

Dabr's Hashtag

Without official guidance - implicit knowledge develops.

Has Dabr Got It Wrong?

No. I don't think so. Take a look at how Twitter on the web renders hashtags...

Twitter's Web Site

...and on the mobile.

Twitter Mobile

So Where Does Render The Full Tag?

Several applications don't render tags in the same way as Twitter. Take a look at SocialScope

SocialScope Hashtags

Tweetie2

I'll upload more screenshots if I find examples of "badly behaved" hashtags.  Please let me know if you find any.

What Does Twitter Say?

Twitter has one page devoted to hashtags.  It is a support page for hashtags.  This explains to people what hashtags are.  There's no detail on valid characters, maximum length, or any of the things which might be useful for a developer or designer.

Edit 2010-02-25

David Dorward has pointed out that there is an official resource. On the Twitter Engineering blog - which isn't linked to from the developer site - there is a page discussing hashtags and how to validate them. You'll notice that they are rather circumspect on what should constitute a hashtag.

Conclusion

Standards and guidelines allow developers to create compatible applications.

Without explicit recommendations, developers will diverge as widely as possible.  Twitter - and everyone with an interest in compatibility and usability - needs to ensure that the knowledge they impart is explicit.

Letting people make it up as they go along leads to confusion.