There are a wide range of social media logins provided by Auth0 - including the usual suspects like Facebook, Twitter, WordPress, Discord, etc. Sadly, there's no support for Mastodon⁰.

All is not lost though. The Auth0 documentation says:

However, you can use Auth0’s Connections API to add any OAuth2 Authorization Server as an identity provider.

You can manually add a single Mastodon instance, but that doesn't work with the decentralised nature of the Fediverse. Instead, I've come up with a manual solution which works with any Mastodon server!

Background

Every Mastodon¹ server is independent. I have an account on mastodon.social you have an account on whatever.chaos. They are separate servers, albeit running similar software. A generic authenticator needs to work with all these servers. There's no point only allowing log ins from a single server.

Fortuitously, Mastodon allows app developers to automatically create new apps. A few simple lines of code and you will have an API key suitable for read-only access to that server. You can read how to instantly create Mastodon API keys or you can steal my PHP code.

User Experience

The user clicks the sign-in button on OpenBenches. They're taken to the Auth0 social login screen:

The user clicks on Mastodon. This is where Auth0's involvement ends!

The user is asked to provide the URl of their instance:

In the background, my server contacts the Mastodon instance and creates a read-only API key.

The user is asked to sign in to Mastodon.

The user is asked to authorise read-only access.

The user is now signed in and OpenBenches can retrieve their name, avatar image, and other useful information. Hurrah!

Auth0

Once you have created a service to generate API keys, it will need to run on a publicly accessible web server. For example https://example.com/mastodon_login.

Here's what you need to do within your Auth0 tennant:

• Authentication → Social → Create Connection
• At the bottom, choose "Create Custom".
• Choose "Authentication" only.
• Give your connection a name. This will be visible to users.
• "Authorization URL" and "Token URL" have the same value - the URl of your service.
• "Client ID" is only visible to you.
• "Client Secret" any random password; it won't be used for anything.
• Leave everything else in the default state.

It should look something like this:

Click the "Create" button and you're (almost) done.

Auth0 Icon

You will need to add a custom icon to the social integration. Annoyingly, there's no way to do it through the web interface, so follow that guide to use the command line.

Done!

I'll admit, this isn't the most straightforward thing to implement. Auth0 could make this easier - but it would still rely on users knowing the URl of their home instance.

That said, the Mastodon API is a delight to work with and the read-only permissions reduce risk for all parties.

A user writes a status. The user can choose to make their statuses quotable or not. What happens when a quoter quotes that post?

I've read through the specification and tried to simplify it. Quoting is a multi-step process:

1. The status must opt-in to being shared.
2. The quoter quotes the status.
3. The quoter's server sends a request to the status's server.
4. The status's server sends an accept message back to the quoter's server.
5. When other servers see the quote, they check with the status's server to see if it is allowed.

I'm going to walk you through each stage as best as I understand them.

Opting In

An ActivityPub status message is JSON. In order to opt-in, it needs this additional field.

"interactionPolicy": {
  "canQuote": {
    "automaticApproval": "https://www.w3.org/ns/activitystreams#Public"
  }
}

That tells ActivityPub clients that anyone is allowed to quote this post. It is also possible to say that only specific users, or only followers, or no-one is allowed.

The QuoteRequest

Someone has hit the quote post button, typed their own message, and shared their wisdom. Their server sends the following message to the server which hosts the quoted status. This has been edited for brevity.

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    {
      "QuoteRequest":   "https://w3id.org/fep/044f#QuoteRequest"
    }
  ],
  "type": "QuoteRequest",
  "id":     "https://mastodon.test/users/Edent/quote_requests/1234-5678-9101",
  "actor":  "https://mastodon.test/users/Edent",
  "object": "https://example.com/posts/987654321.json",
  "instrument": {
    "id":           "https://mastodon.test/users/Edent/statuses/123456789",
    "url":          "https://mastodon.test/@Edent/123456789",
    "attributedTo": "https://mastodon.test/users/Edent",
    "quote":          "https://example.com/posts/987654321.json",
    "_misskey_quote": "https://example.com/posts/987654321.json",
    "quoteUri":       "https://example.com/posts/987654321.json"
  }
}

All this says is "I would like permission to quote you."

The Stamp

The quoted server needs to approve this quote. First, it generates a "stamp".

This is a file which lives on the quoted server. It is proof that the quote is allowed. If it is deleted, the quote permission is revoked. When the stamp's ID is requested the stamp must be returned.

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    {
      "gts": "https://gotosocial.org/ns#",
      "QuoteAuthorization": {
        "@id": "https://w3id.org/fep/044f#QuoteAuthorization",
        "@type": "@id"
      },
      "interactingObject": {
        "@id": "gts:interactingObject"
      },
      "interactionTarget": {
        "@id": "gts:interactionTarget"
      }
    }
  ],
  "type": "QuoteAuthorization",
  "id":                "https://example.com/quote-987654321.json",
  "attributedTo":      "https://example.com/users/username",
  "interactionTarget": "https://example.com/posts/987654321.json",
  "interactingObject": "https://mastodon.test/users/Edent/statuses/123456789"
}

If the quoted status is viewed from a different server, that server will query the stamp to make sure the share is allowed.

The Accept

This is the message that the quoted server sends to the quoting server. It references the request and the stamp.

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    {
      "QuoteRequest": "https://w3id.org/fep/044f#QuoteRequest"
    }
  ],
  "type": "Accept",
  "to":    "https://mastodon.test/users/Edent",
  "id":    "https://example.com/posts/987654321.json",
  "actor": "https://example.com/account",
  "object": {
    "type": "QuoteRequest",
    "id":         "https://mastodon.test/users/Edent/quote_requests/1234-5678-9101",
    "actor":      "https://mastodon.test/users/Edent",
    "instrument": "https://mastodon.test/users/Edent/statuses/123456789",
    "object":     "https://example.com/posts/987654321.json"
  },
  "result": "https://example.com/quote-987654321.json"
}

The "result" must be the same as the stamp's URl.

And then?

You can follow and quote @colours@colours.bots.edent.tel on your favourite Fediverse platform.

I've written an ActivityPub server in a single file which is designed to teach you have the protocol works. Have a play with ActivityBot.

For a traditional social-media site like Twitter or Facebook, you would create an OAuth app on the service that you want. But there are hundreds of Mastodon servers. So you need to create a new app for each one. That sounds hard, but it isn't. Well… not too hard.

Here's some code adapted from Infosec.press. It's all written using cURL on the command line - so you should be able to adapt it to your preferred programming language.

Register an app on the user's Mastodon instance

Let's assume the user has given you the name of their Mastodon server - example.social

You then send a request for an app to be created on example.social with your website's details. All it requests is the ability to read a user's details, nothing else.

curl -X POST \
 -F "client_name=Login to your_website.tld" \
 -F "redirect_uris=https://your_website.tld/oauth/mastodon?server=example.social&" \
 -F "scopes=read:accounts" \
 -F "website=https://your_website.tld" \
 -A "user-agent/0.1"
 https://example.social/api/v1/apps

You can set the User Agent to be anything suitable. Some servers won't work if it is omitted.

If the request was successful, example.social will send you this JSON in response:

{
  "id": "12345",
  "name": "Login to your_website.tld",
  "website": "https://your_website.tld",
  "scopes": [
    "read:accounts"
  ],
  "redirect_uris": [
    "https://your_website.tld/oauth/mastodon?server=example.social&"
  ],
  "vapid_key": "qwertyuiop-asdfghjkl-zxcvbnm",
  "redirect_uri": "https://your_website.tld/oauth/mastodon?server=example.social&",
  "client_id": "qw_asdfghjkl_zxcvbnm",
  "client_secret": "qwertyuiop1234567890"
}

Save the server's address, the client_id, and the client_secret. You will need all three later.

The user logs in to their Mastodon instance

You need to redirect the user to their server so they can log in. You need to construct a Mastodon URl using the data you received back. Don't forget to URl encode the redirect_uri.

For example, redirect the user to:

https://example.social/oauth/authorize
?client_id=qw_asdfghjkl_zxcvbnm
&scope=read:accounts
&redirect_uri=https://your_website.tld/oauth/mastodon%3Fserver=example.social%26
&response_type=code

When the user visits that URl they can then log in. If they're successful, they'll be redirected back to your server using your specified redirect URI:

https://your_website.tld/oauth/mastodon?server=example.social&code=qazwsxedcrfvtgbyhnujm

Get a Bearer token

Your website has received a GET request with the user's server name and an authorisation code. As per the Mastodon documentation, your app uses that code to request a Bearer token:

curl -X POST \
 -F "client_id=qw_asdfghjkl_zxcvbnm" \
 -F "client_secret=qwertyuiop1234567890" \
 -F "redirect_uri=https://your_website.tld/oauth/mastodon?server=example.social&" \
 -F "grant_type=authorization_code" \
 -F "code=qazwsxedcrfvtgbyhnujm" \
 -F "scope=read:accounts" \
 -A "user-agent/0.1"
 https://example.social/oauth/token

If that's worked, the user's server will return a Bearer token like this:

{
    "access_token": "abcdefg_123456",
    "token_type": "Bearer",
    "scope": "read:accounts",
    "created_at": 1732916685
}

Get the user's details

Finally(!) you can use that token to verify the user's credentials with the server:

curl \
 -H "Authorization: Bearer abcdefg_123456" \
 -A "user-agent/0.1"
 https://example.social/api/v1/accounts/verify_credentials

If that works, you'll get back all the user's details. Something like this:

{
    "id": "7112",
    "username": "Edent",
    "acct": "Edent",
    "display_name": "Terence Eden",
    "url": "https://mastodon.social/@Edent",
    "avatar": "https://files.mastodon.social/accounts/avatars/000/007/112/original/37df032a5951b96c.jpg",
...
}

Putting it all together

1. User providers their Mastodon instance's domain name
2. Your service looks up the domain name in its database
   • If there are no results, request to create a new app on the Mastodon instance and save the returned client_id and client_secret
3. Redirect the User to their Mastodon instance, using a URl which contains the client_id & callback URl
4. User logs in to their Mastodon instance
5. The User's Mastodon instance redirects the User to your service's callback URl which includes an the instance's domain name and User's authorisation code
6. Your service reads the User's domain name and authorisation code
7. Your service exchanges those details for a Bearer token
8. Your service uses the Bearer token to get the User's account details

Next steps?

This basic code works. For my next trick, can I integrate it into Auth0?

First up, you'll need a file called config.py to hold all your API keys:

instance = "https://mastodon.social"
access_token          = "…"
write_access_token    = "…"
untappd_client_id     = "…"
untappd_client_secret = "…"

Then a file called untappd2mastodon.py to do the job of grabbing your data, finding your latest check-in, then posting it to the Fediverse:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from mastodon import Mastodon
import json
import requests
import config

#  Set up access
mastodon = Mastodon( api_base_url=config.instance, access_token=config.write_access_token )

#       Untappd API
untappd_api_url = 'https://api.untappd.com/v4/user/checkins/edent?client_id=' + config.untappd_client_id + '&client_secret='+ config.untappd_client_secret

r = requests.get(untappd_api_url)

untappd_data = r.json()

#       Latest checkin object
checkin = untappd_data["response"]["checkins"]["items"][0]
untappd_id = checkin["checkin_id"]

#       Was this ID the last one we saw?
check_file = open("untappd_last", "r")
last_id = int( check_file.read() )
print("Found " + str(last_id) )
check_file.close()

if (last_id != untappd_id ) :
        print("Found new checkin")
        check_file = open("untappd_last", "w")
        check_file.write( str(untappd_id) )
        check_file.close()
        #       Start creating the message
        message = ""

        if "checkin_comment" in checkin :
                message += checkin["checkin_comment"]

        if "beer" in checkin :
                message += "\nDrinking: " + checkin["beer"]["beer_name"]

        if "brewery" in checkin :
                message += "\nBy: "       + checkin["brewery"]["brewery_name"]

        if "venue" in checkin :
                if "venue_name" in checkin["venue"] :
                        message += "\nAt: "       + checkin["venue"]["venue_name"]
        #       Scores etc
        untappd_checkin_url = "https://untappd.com/user/edent/checkin/" + str(untappd_id)
        untappd_rating      = checkin["rating_score"]
        untappd_score       = "🍺" * int(untappd_rating)

        message += "\n" +  untappd_score + "\n" + untappd_checkin_url + "\n" + "#untappd"

        #       Get Image
        if checkin["media"]["count"] > 0 :
                photo_url = checkin["media"]["items"][0]["photo"]["photo_img_lg"]
                download = requests.get(photo_url)
                with open("untappd.tmp", 'wb') as temp_file:
                        temp_file.write(download.content)
                media = mastodon.media_post("untappd.tmp", description="A photo of some beer.")
                mastodon.status_post(status = message, media_ids=media, idempotency_key = str(untappd_id))
        else:   
                #       Post to Mastodon. Use idempotency just in case something went wrong
                mastodon.status_post(status = message, idempotency_key = str(untappd_id))
else :
        print("No new checkin")

You can treat this code as being MIT licenced if that makes you happy.

Yes!

As we all know, the Fediverse is one big chain mail. I don't mean that in a derogatory way.

When I write a post, it appears on my server (called an "instance" in Mastodon-speak).

Everyone on my instance can see my post.

My instance looks at all my followers - some of whom are on completely different instances - and sends my post to their instances.

As an example:

• I am on mastodon.social
• John is on eggman_social.com
• Paul is on penny_lane.co.uk
• Both John and Paul follow me. So my post gets syndicated to their servers.

With me so far?

What happens when someone shares (reposts) my status?

• John is on eggman_social.com
• Ringo is on liverpool.drums
• Ringo follows John
• John reposts my status.
• eggman_social.com syndicates my post to liverpool.drums

And so my post goes around the Fediverse! But can I see where it has gone? Well... sort of! Let's look at how.

A note on privacy

People on Mastodon and the Fediverse tend to be privacy conscious. So there are limits - both in the API and the culture - as to what is acceptable.

Some people don't share their "social graph". That is, it is impossible to see who follows them or who they follow.

Users can choose to opt-in or -out of publicly sharing their social graph. They remain in control of their privacy.

In the example above, if Ringo were to reshare John's reshare of my status - John doesn't know about it. Only the original poster (me) gets notified. If John doesn't share his social graph, it might be possible to work out where Ringo saw the status - but that's rather unlikely.

Mastodon has an API rate limit which only allows 80 results per request and 1 request per second. That makes it long and tedious to crawl thousands of results.

Similarly, some instances do not share their social data or expose anything of significance. Some servers may no longer exist, or might have changed names. It's impossible to get a comprehensive view of the entire Fediverse network.

And that's OK! People should be able to set limits on what others can do with their data. The code you're about to see doesn't attempt to breach anyone's privacy. All it does is show me which servers picked up my post. This is information which is already shown to me - but this makes it slightly easier to see.

The Result

I looked at this post of mine which was reposted over 100 times.

It eventually found its way to… 2,547 instances!

Ranging from 0ab.uk to թութ.հայ via godforsaken.website and many more!

And that's one of the things which makes me hopeful this rebellion will succeed. There are a thousand points of light out there - each a shining beacon to doing things differently. And, the more the social media giants tighten their grip, the more these systems will slip through their fingers.

The Code

This is not very efficient code - nor well written. It was designed to scratch an itch. It uses Mastodon.py to interact with the API.

It gets the instance names of all my followers. Then the instance names of everyone who reposted one of my posts.

But it cannot get the instance names of everyone who follows the users who reposted me - because: The only way to get a list of followers from a user on a different instance is to apply for an API key for that instance. Which seems a bit impractical.

But I can get the instance name of the followers of accounts on my instance who reposted me. Clear?

I can also get a list of everyone who favourited my post. If they aren't on my instance, or one of my reposter's follower's instances, they're probably from a reposter who isn't on my instance.

My head hurts.

Got it? Here we go!

import config
from mastodon import Mastodon
from rich.pretty import pprint

#  Set up access
mastodon = Mastodon( api_base_url=config.instance, access_token=config.access_token, ratelimit_method='pace' )

#   Status to check for
status_id = 111040801202691232
print("Looking up status: " + str(status_id))

#   Get my data
me = mastodon.me()
my_id = me["id"]
print("You have User ID: " + str(my_id))

#   Empty sets
instances_all        = set()
instances_followers  = set()
instances_reposters  = set()
instances_reposters_followers  = set()
instances_favourites = set()

#   My Followers
followers = mastodon.account_followers( my_id )
print( "Getting all followers" )
followers_all = mastodon.fetch_remaining( followers )
print("Total followers = " + str( len(followers_all) ) )

#   Get the server names of all my followers
for follower in followers_all:
    if ( "@" in follower["acct"]) :
        f = follower["acct"].split("@")[1]
        instances_all.add( f )
        if ( f not in instances_followers):
            print( "Follower: " + f )
            instances_followers.add( f )
    else :
        instances_all.add( "mastodon.social" )
        instances_followers.add( "mastodon.social" )
total_followers  = len(instances_followers)
print( "Total Unique Followers Instances = " + str(total_followers)  )

#   Reposters
#   Find the accounts which reposted my status
reposters     = mastodon.status_reblogged_by( status_id )
reposters_all = mastodon.fetch_remaining(reposters)

#   Get all the instance names of my reposters
for reposter in reposters_all:
    if ( "@" in reposter["acct"]) :
        r = reposter["acct"].split("@")[1]
        instances_all.add( r )
        if ( r not in instances_followers ) :
            print( "Reposter: " + r )
            instances_reposters.add( r )
total_reposters  = len(instances_reposters)
print( "Total Unique Reposters Instances = " + str(total_reposters)  )

# Followers of reposters     
# This can take a *long* time!   
for reposter in reposters_all:   
    if ( "@" not in reposter["acct"]) :  
        reposter_id = reposter["id"]
        print( "Getting followers of reposter " + reposter["acct"] + " with ID " + str(reposter_id) )
        reposter_followers = mastodon.account_followers( reposter_id )   
        reposter_followers_all = mastodon.fetch_remaining( reposter_followers )  
        for reposter_follower in reposter_followers_all:    
            if ( "@" in reposter_follower["acct"]) : 
                f = reposter_follower["acct"].split("@")[1]
                instances_all.add( f )
                if (f not in instances_reposters_followers) :
                    print( "   Adding " + f + " from " + reposter["acct"] )
                    instances_reposters_followers.add( f )   
total_instances_reposters_followers  = len(instances_reposters_followers)
print( "Total Unique Reposters' Followers Instances = " + str(total_instances_reposters_followers)  )

#   Favourites
#   Find the accounts which favourited my status
favourites     = mastodon.status_favourited_by( status_id )
favourites_all = mastodon.fetch_remaining(favourites)

#   Get all the instance names of my favourites
for favourite in favourites_all:
    if ( "@" in favourite["acct"]) :
        f = favourite["acct"].split("@")[1]
        instances_all.add( f )
        if ( f not in instances_favourites ) :
            print( "Favourite: " + f )
            instances_favourites.add( r )
total_favourites = len(instances_favourites)

print( "Total Unique Favourites Instances  = " + str(total_favourites) )
print( "Total Unique Reposters Instances = " + str(total_reposters)  )
print( "Total Unique Followers Instances = " + str(total_followers)  )
print( "Total Unique Reposters' Followers Instances = " + str( len(instances_reposters_followers) ) )
print( "Total Unique Instances = " + str( len(instances_all) ) )

If I moved to a different server, my "birthday" would be irrevocably lost 😢

But… what if I moved to a self-hosted Mastodon instance? Why! Then the database would be under my complete control and I could put whatever data I wanted in there. I could even lie about things!

Surely no one would be that silly though?

The other day I was chatting with someone whose follower count was so high that it temporarily broke my client.

To be clear, the account @admin@mastodon.adtension.com is being deliberately provocative here. I don't think that they expect anyone to believe that they have more followers than the entirety of humanity, nor that they started using Mastodon last century.

This isn't a problem limited to Mastodon and ActivityPub. Twitter controls its own database and could, if it wanted to, inflate follower numbers for insecure people.

And, before you get too excited, this isn't a usecase for BlockChain! In theory, multiple servers could write statistics to a ledger (1,000 people on example.social follow @edent@whatever.social) but they have no way of verifying each others' statistics. So a determined user could have multiple fake instances writing fake data to the chain.

Chasing status is a mug's game. Anyone can hire a sports car for the afternoon and rent a fancy suit - that doesn't mean they're a celebrity. Similarly, anyone can lie on the Fediverse and make you believe they're a social media superstar.

Don't believe the hype!

Search is complex - expectations

I don't mean the act of searching a database - that's routine - but I mean it is socially complex.

Lots of people left Twitter because it was too easy to search for them. For example, if you really hate people who support the wrong football team, it's trivial to search Twitter for people posting about that team. Then you can send them abuse. Or you can drag up old arguments. Or you can find something said years ago which sounds silly today.

So Mastodon was a haven. A place where it wasn't easy for twunts to pick a fight with you. Naturally, that means there has been some resistance to the idea of search - and that has informed its design decisions. By default, accounts cannot be searched. You have to make the positive choice to opt-in to search. And you can opt-out at any time.

That strikes me as a reasonable position. It adds a new feature for those that want it and keeps people safe if they'd rather not be bothered.

Search is complex - federation

The Fediverse is a collection of servers which share information with each other. You can only search the posts which your server knows about.

Here's what I mean. I post on mastodon.social and you post on whatever.int - because I subscribe to you, all your posts are sent to mastodon.social. That means anyone on my server can search for your posts. No one from fursuits.example follows me, so my posts aren't searchable by their members. But let's say @jeff@whatever.int is followed by @sue@fursuits.example - if Jeff reposts me, then that specific post is syndicated to the fursuits instance and that specific post can then be found in search results.

A bit confusing - but necessary. This isn't a monolithic service like Twitter with a single database. It is a loose collection of distributed entities which share data. You cannot search the entire Fediverse.

Search is complex - images

Six years ago, I noted that you couldn't search Twitter for Alt-Text. But, I am pleased to say, Mastodon allows for that! Search results include the description of images.

This can be a little confusing. You might search for a phrase and see dozens of posts which don't contain it in their body. It's only when you hover over the image that you'll find what you're looking for.

I don't know if this is something which could be improved with a better UI? Perhaps some way of highlighting that it is the image which is the result?

Search is complex - domains

Searching text is complicated. Should a search for "mastodon" find posts with the words "mastodons"? Probably. Should a search for the word "the" find "there", "their", or "them"? Probably not.

Should a search for "example.com" find "example.com/blog"? Maybe. At the moment, it doesn't. So you can't use search to find who shared your website. Nor can you find people who are talking about a specific page on the web.

Update! This works! A search for shkspr.mobi -from:edent finds all mentions of my website by people other than me.

Search is complex - operator discovery

As well as searching for words and phrases, you can also search metadata. For example, adding has:video to your query will find posts with an attached video. Using is:reply will find replies. And after:2023-08-27 will find posts after a specific date.

At the moment, there's no way to know what those options are. There's a discussion on GitHub which enumerates them but nothing for the casual user. Perhaps there needs to be an "Advanced Search" page? That's how most social networks do things.

Update! There's now a drop-down on the search box which shows you all the options.

Search is complex - what's missing?

Twitter let users search by number of likes & retweet - which was helpful for finding your most popular Tweets. That's not (yet) available on Mastodon.

Similarly, there's no way to search by geography. Twitter let users tag posts with their geolocation which could then be queried. Neither of those features are on Mastodon's roadmap.

I don't think it is possible to search previous edits of a post. That's probably a sensible decision.

Search is complex - unknown unknowns

One valid criticism of the Fediverse is that it is hard to search for interesting people, discussions, or posts. Hopefully this opt-in search makes it easier for people to meet new friends and form new ideas.

And, hopefully, the controls around it are strict enough that it will prevent abuse from running rampant.

I think searching is going to be primarily a good thing for the Fediverse. But what second and third order effects will this bring?

In one sense, this is sad. A set of useful public services are being cut off from their audience. My friend, Bill Thompson, described this as "unnecessary disruption" I, on the other hand, think that creative destruction is sometimes a necessity.

You can listen to more of the discussion on the Gareth and BillCast about 19 minutes in.

Look, first off, it makes sense for organisations to put warning messages where their audience is likely to see it. No one sensible begrudges that. Twitter is a channel just as legitimate as radio, TV, or loudhailers in the street.

The problem comes when that's the only channel.

Twitter, somewhat accidentally, set itself up as a replacement for the public sphere. Now it has made it clear exactly what the cost of "free speech" is - alert bots are abandoning it.

Almost all the bots I mentioned above have websites. You can go check them, if you remember - but that isn't the same as having their warnings mixed in with your daily scrolling.

Some of those bots also have RSS feeds. But the RSS model doesn't lend itself to your peer group reposting urgent content into your feed.

And none of those websites and feeds are easily discoverable if you're in a new city.

So, how could this be solved with ActivityPub / Mastodon / Fediverse?

On the one hand, it's pretty easy for any organisation to set up a decentralised service with their alerts. Visiting https:// mass_transit.ak will show you all of Arstotzka's transport bots. You could subscribe to @flood-warnings@alerts.gov.ak for Arstotzka's official warnings.

On Twitter, it's easy to search "Train Delays London" and see everything. Hopefully including one of the accounts which gives you official alerts.

But, one of the interesting problems with a decentralised service is that search is (deliberately⁰) difficult. Unless someone you're already following happens to share an alert, it can be very difficult to find relevant posts and accounts.

There is something of a solution which, I think, could be helpful. ActivityPub defines a way to add location to a post.

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Note",
  "name": "Arstotzka' Flood Service:",
  "content": "Heavy rains! Expect flooding and major disruption.",
    "location": {
       "name": "Arstotzka Central District,
       "type": "Place",
       "longitude": 12.34,
       "latitude": 56.78,
    }
}

At the moment, there's no way to search for geo-tagged posts - but it doesn't have to be that way. It is relatively simple to use the Haversine formula to select database entries which are within a geographic area.

But, the key problem is too much decentralisation. It goes against the credo of redecentralization to have a single server called official-alerts.com - having a single destination to search is convenient, but provides for a single point of failure.

A country might have alerts.ak which has multiple accounts, or might have single-use instances like flood.ak and fire.ak and traffic.ak. They might be federated to, for example, official-alert-service.eu.

At which point, I get stuck.

How does a user discover these accounts?

There are three options, as I see it:

1. Advertising. The organisation promotes these accounts via other channels.
2. Serendipity. A user sees that another user has shared one of these accounts.
3. Better discovery tools. A user's instance could recommend accounts to follow based on geography or other factors.

Thoughts?

After checking in, the app automatically posts to Twitter. But who wants to prop up Alan's failing empire? Not me! So here's some quick code to liberate your data and post it elsewhere.

There are two ways - APIs and Screen Scraping.

API

First up, a big disclaimer. Untappd had an API - but aren't accepting new users:

Thank you for your interest in Untappd’s API. At this time, we are no longer accepting new applications for API access as we work to improve our review and support processes. We do not have a planned date to begin accepting new applications, so please check back soon.

If you already have an API key (and I do) you can call your own data. This code saves the ID of the most recent checkin to a file. Each time it runs, it checks in the ID in the file is the same as what's returned by the API. If they are different, the post is published and the new ID is saved.

This is rough and ready code:

#!/usr/bin/env python
from mastodon import Mastodon
import json
import requests
import config

#  Set up access
mastodon = Mastodon( api_base_url=config.instance, access_token=config.write_access_token )

#   Untappd API - grab the most recent checkin
untappd_api_url = 'https://api.untappd.com/v4/user/checkins/edent?limit=1&client_id=' + config.untappd_client_id + '&client_secret='+ config.untappd_client_secret

r = requests.get(untappd_api_url)

untappd_data = r.json()

#   Latest checkin object
checkin = untappd_data["response"]["checkins"]["items"][0]
untappd_id = checkin["checkin_id"]

#   Was this ID the last one we saw?
check_file = open("untappd_last", "r")
last_id = int( check_file.read() )
print("Found " + str(last_id) )
check_file.close()

if (last_id != untappd_id ) :
    print("Found new checkin")
    check_file = open("untappd_last", "w")
    check_file.write( str(untappd_id) )
    check_file.close()
    #   Start creating the message
    message = ""

    if "checkin_comment" in checkin :
        message += checkin["checkin_comment"]

    if "beer" in checkin :
        message += "\nDrinking: " + checkin["beer"]["beer_name"]

    if "brewery" in checkin :
        message += "\nBy: "       + checkin["brewery"]["brewery_name"]

    if "venue" in checkin :
        if "venue_name" in checkin["venue"] :
            message += "\nAt: "       + checkin["venue"]["venue_name"]
    #   Scores etc
    untappd_checkin_url = "https://untappd.com/user/edent/checkin/" + str(untappd_id)
    untappd_rating      = checkin["rating_score"]
    untappd_score       = "🍺" * int(untappd_rating)

    message += "\n" +  untappd_score + "\n" + untappd_checkin_url + "\n" + "#untappd"
    print( message )
    #   Post to Mastodon. Use idempotency just in case something went wrong
    mastodon.status_post(status = message, idempotency_key = str(untappd_id))
else :
    print("No new checkin")

This doesn't do media or badges, but it's good enough to start with.

Screen Scraping

The Untappd HTML is pretty uniform, so using something like Beautiful Soup is possible.

Here's some code to get you started

from bs4 import BeautifulSoup
import requests

#   Set the URL and headers
url = "https://untappd.com/user/edent"
headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0'}

#   Get the HTML
response = requests.get(url)
page_html = response.text

#   Parse the HTML into soup
soup = BeautifulSoup(page_html, 'html.parser')

#   Grab the data from the first checkin

untappd_id = int( soup.find("div", class_="item")["data-checkin-id"] )

comment = soup.find("p", class_="comment-text").text

#   Steal the beer from the icon's alt text
beer = soup.find("a", class_="label").find("img")["alt"]

#   Rating is a data element
rating = soup.find("div", class_="caps")["data-rating"]

#   Was this ID the last one we saw?
check_file = open("untappd_last", "r")
last_id = int( check_file.read() )
check_file.close()

if (last_id != untappd_id ) :
    check_file = open("untappd_last", "w")
    check_file.write( str(untappd_id) )
    check_file.close()
    message = .....

And from there you should have enough to start posting your checkins everywhere. Stick that code in a crontab and have it run periodically.

Cheers!

1. cache invalidation
2. naming things
3. off-by-one errors

Let's talk about how we name unique items in Federated services - for example, posts on a social media service.

If you have only one service, it's pretty easy. Every time a new entry is created in a database, give it a sequential number.

This becomes a problem at scale. If you have millions of users on hundreds of different shards of a database, eventually you'll get a clash of IDs. To that end, Twitter invented Snowflake IDs.

Snowflakes are pretty clever. They are a 64 bit ID. The first part of the ID is a timestamp, and the second part is some information about the server which generated the ID. This means that IDs can be sorted by time, and will globally unique.

So, does the Fediverse use Snowflake? Yes. AND NO!

The Mastodon service uses Snowflake for its IDs. For example, one of my posts has the ID 109347703064222520. If you bitshift that into a 64 bit number, you'll see it is a UNIX timestamp with a few bits of extra data.

But here comes the problem. On a Federated service, there's no guarantee of uniqueness. A naughty server could deliberately generate a duplicate Snowflake ID. You can't trust anyone on the Internet!

So here's how Mastodon - and the wider Fediverse - deals with unique IDs. It cheats.

• When I make a post, my server gives it a Snowflake ID which is unique to my server - e.g. 123456
• This generates a URl which is globally unique - e.g. https:// my_server.xyz/@username/123456
• My post is delivered to your server.
• Your server gives it an ID which is unique to your server - e.g. 4d7b70c7
• Your server turns that into a URl - e.g. https:// your_server.lol/incoming/from/@username@my_server.xyz/4d7b70c7

For example, here's the JSON response I get when I look up the Snowflake ID of a reply someone sent me. My server replies with:

{
 'created_at': datetime.datetime(2022, 11, 13, 0, 52, 37, tzinfo=tzutc()),
 'id': 109347716491680514,
 ...
 'uri': 'https://queer.af/@erincandescent/109347716173491502',
}

The original ID is 109347716173491502 on the sender's server. But the unique ID on my server is 109347716491680514.

Their unique ID could be a GUID, a series of emoji, or a sequential number. It doesn't matter. To prevent clashes, the receiving server generates its own ID.

This could all be solved a lot more easily if every server minted each post using a Proof of Work transaction against a centralised BlockCha… *sound of gunshot*

Ahem.

It is a pretty reasonable solution. A remote server may or may not have a globally unique ID. It doesn't matter. Once you see a post, it is given its own locally unique ID. And that's good enough.

When you click on a link on my website which takes you to another website, your browser sends a Referer⁰. This says to the other site "Hey, I came here using a link on shkspr.mobi". This is useful because it lets a site owner know who is linking to them. I love seeing which weird and wonderful sites have linked to my content.

It is also something of a privacy nightmare as it lets sites see who is clicking and from where they're clicking. So Mastodon sets a noreferrer¹ attribute on all links. This tells the browser not to send the Referer.

This means sites no longer know who is sending them traffic.

That's either a good thing from a privacy perspective or a disaster from a marketing perspective. Or a little bit of both.

Here's a related issue. When a user posts a link to your website on Mastodon, the server checks your page to see if there are any oEmbed tags for a rich link preview. But, at the moment, it doesn't check your website's robots.txt file - which lets it know whether it is allowed to scrape your content.

In the case of something like Twitter or Facebook, this is fine. If a million users post a link, the centralised social network checks the link once and caches the result.

With - potentially - thousands of distributed Mastodon sites, this presents a problem. If a popular account posts a link, their instance fetches a rich preview. Then every instance which has users following them also requests that URL. Essentially, this is a DDoS attack.

I can fix you

So here's my thoughts on how to fix this.

When a user posts a link to Mastodon, their instance should send a WebMention to the site hosting the link. This informs the website that someone has shared their content. Perhaps a user could adjust their privacy settings to allow or deny this.

The instance would check the site's robots.txt and, if allowed, scrape the site to see if there were any Open Graph Protocol metadata elements on it.

That metadata should be included in the post as it is shared across the network.

For example, a status could look like this:

{
  "id": "123",
  "created_at": "2022-03-16T14:44:31.580Z",
  "in_reply_to_id": null,
  "in_reply_to_account_id": null,
  "visibility": "public",
  "language": "en",
  "uri": "https://mastodon.social/users/Edent/statuses/123",
  "content": "<p>Check out https://example.com/</p>",
  "ogp_allowed": true,
  "ogp": {
      "og:title": "My amazing site",
      "og:image:url": "https://cdn.mastodon.social/cache/example.com/preview.jpg",
      "og:description": "A long description. Perhaps the first paragraph of the text."
      ...
   }
   ...
}

When a post is boosted across the network, the instances can see that there is rich metadata associated with the link. If there is an image associate with the post, that will be loaded from the cache on the original Mastodon instance - avoiding overloading the website.

Now, there is a flaw in this idea. A malicious Mastodon server could serve up a fake OGP image and description. So a link to McDonald's might display a fake image promoting Burger King.

To protect against this, a receiving instance could randomly or periodically check the OGP metadata that they receive. If it has been changed, they can update it.

Perhaps a diagram would help?

What other people say about the problem

David Gerard

@davidgerard@circumstances.run

yes, you should put a cache in front of a blog. nginx and wp-supercache do well. but.

mastodon's auto-DDOS feature is still obnoxious. and in a social network, technically designed in obnoxiousness is incompetent.

i realise it'd need extension of activitypub, but is anyone working on sending prerendered cards with the URL? just to save 1000 servers hammering the URL to generate their own cards locally.

2022-11-28, 14:44 7 boosts 23 favorites

Feedback?

Is this a problem? Does this present a viable solution? Have I missed something obvious? Please leave a comment and let me know 😃

When you scroll through the website, you normally see a list of replies. It looks like this:

Because it acts as a one-dimensional list, there's no easy way to figure out which post someone is replying to.

The data structure underlying the conversation is quite different. It actually looks like this:

Concepts

In Mastodon's API, a post is called a status.

Every status on Mastodon has an ID. This is usually a Snowflake ID which is represented as a number.

When someone replies to a status on Mastodon, they create a new status which has a field called in_reply_to_id. As its name suggests, has the ID of the status they are replying to.

Let's imagine this simple conversation:

1. Ada: "How are you?"
2. Bob: "I'm fine. And you?"
3. Ada: "Quite well, thank you!"

Message 2 is in reply to message 1. Message 3 is in reply to message 2.

In Mastodon's jargon, message 1 is the ancestor of message 2. Similarly, message 3 is the descendant of message 2.

  → Descendants →
1--------2-------3
   ← Ancestors ←

Branches

Now, let's imagine a more complicated conversation - one with branches!

1. Alice: What's your favourite pizza topping?
├── 2. Bette: Pineapple
│   ├── 4. Chuck: You make me sick!
│   └── 7. Dave: Yeah, I love pineapple too
└── 3. Chuck: Mushroom are the best
    ├── 5. Alice: Really?
    │   └── 6. Dave: Button mushrooms are best!
    └── 8. Elle: I like them too!

As you can see, people reply in threads. In this example, 2 is a different "branch" of the conversations than 3.

It looks a bit more complicated with hundreds of replies, but that's it! That's all you need to know!

API

If you want to download a single status with an ID of 1234 the API call is /api/v1/statuses/1234

If you want to download a conversation, it is a little bit more complicated. Mastdon's API calls a conversation a context

Let's take the above simple example - Ada and Bob speaking. Ada's first status has an ID of 1. To get the conversation, the API call is /api/v1/statuses/1/context

That returns two things:

• A list of ancestors. This is empty because 1 is the first status in this conversation.
• A list of descendants. This contains statuses 2 and 3.

You will note, the context does not return the status 1 itself.

Let's suppose that, instead of asking for the context of status 1, we instead asked for 2. This would return:

• A list of ancestors. This contains status 1.
• A list of descendants. This contains status 3.

What about if we asked for 3? This would return:

• A list of ancestors. This contains status 1 and 2
• A list of descendants. This is empty because 3 is the last message in this conversation.

Branches

When it comes to complex threads - like the pizza example - things become a bit more difficult. Let's see the example again:

1. Alice: What's your favourite pizza topping?
├── 2. Bette: Pineapple
│   ├── 4. Chuck: You make me sick!
│   └── 7. Dave: Yeah, I love pineapple too
└── 3. Chuck: Mushroom are the best
    ├── 5. Alice: Really?
    │   └── 6. Dave: Button mushrooms are best!
    └── 8. Elle: I like them too!

Suppose we ask for the context of the message with ID 5. This would return:

• A list of ancestors. This contains statuses 1 and 3
• A list of descendants. This contains status 6.

That's it!?!? Where are the rest? They are part of a different conversation branch. Even status 8 isn't returned because it's a reply to 3, not 5.

In order to get the full conversation, we need to be sneaky!

The list of ancestors contains the first message in the conversation. So we can grab that, and then call context again for its ID.

Let's dive into some Python code to see how it works.

Code

This uses the Mastodon.py library for calling the Mastodon API and the Python treelib to create a conversation tree data structure.

This code connects to Mastodon and receives the status for a single ID.

from mastodon import Mastodon
from treelib import Node, Tree

mastodon = Mastodon( api_base_url="https://mastodon.example", access_token="Your personal access token from your instance" )

status_id =  109348943537057532 
status = mastodon.status(status_id)

Getting the conversation means calling the context API:

conversation = mastodon.status_context(status_id)

⚠ Note: Calling the context on a large thread may take a long time. The longer the conversation, the longer you'll have to wait.

If there are ancestors, that means we are only on a single branch. The 0th ancestor is the top of the conversation tree. So let's get the context for that top status:

if len(conversation["ancestors"]) > 0 :
   status = conversation["ancestors"][0]
   status_id = status["id"]
   conversation = mastodon.status_context(status_id)

Next, we need to create a data structure to hold the conversation. We'll start by adding to it the first status in the conversation:

tree = Tree()

tree.create_node(status["uri"], status["id"])

Finally, we add any replies which are in the descendants. It is possible that some earlier statuses have been deleted. So we won't add any status which are replies to deleted statuses:

for status in conversation["descendants"] :
   try :
      tree.create_node(status["uri"], status["id"], parent=status["in_reply_to_id"])
   except :
      #  If a parent node is missing
      print("Problem adding node to the tree")

That's it! Let's show the tree:

tree.show()

Here's what it should look like:

2022-11-14 20:02 Edent: Today I was meant to be flying in to San Francisco to attend Twitter's Developer Conference - Chirp.Twitter had paid for my flights and hotel, because I was one of their developer insiders. I planned to spend the week meeting friends old and new.Instead, Alan the Hyperprat canceled the conference. So I'm staying in the UK.So I'm going to spend the week hacking on Mastdon's #API and building cool shit.  That'll show him!You can see what I'm working on at https://shkspr.mobi/blog/2022/11/building-an-on-this-day-service-for-mastodon/ https://mastodon.social/users/Edent/statuses/109343943300929632
├── 2022-11-14 20:10 Edent: Oh! And I was meant to be attending a Belle & Sebastian gig tonight. I canceled those tickets for I could fly to SF.So far, I reckon Alan's acquisition of Twitter has cost me close to £190.Wonder if he's good for the money? https://mastodon.social/users/Edent/statuses/109343972435801664
│   ├── 2022-11-14 20:14 thehodge: @Edent reminds me of the time I was booked to speak at a conference in Munich and I excitedly booked a behind the scenes tour of the worlds largest miniature city!Then the company went under!Gutted. https://mastodon.social/users/thehodge/statuses/109343989481494630
│   ├── 2022-11-14 21:16 Janiqueka: @Edent the way my bill for him keeps increasing https://mastodon.online/users/Janiqueka/statuses/109344233355230523
│   ├── 2022-11-14 21:19 henry: @Edent I was due to be at B&S tomorrow but it’s been postponed again.. not sure if that makes it better or worse for you! https://social.lc/users/henry/statuses/109344244402822729
│   │   └── 2022-11-15 04:53 Edent: @henry again!? Ah well!Hope you get to see them soon. https://mastodon.social/users/Edent/statuses/109346031194446940
│   ├── 2022-11-15 09:18 Amandafclark: @Edent send him an invoice :) https://mastodon.social/users/Amandafclark/statuses/109347071811426672
│   └── 2022-11-15 11:29 Edent: One of the #MastodonAPI projects I'm working on is a better way to view long & complex threads.You may have seen me build something similar for the other site a while ago - demo at https://shkspr.mobi/blog/2021/09/augmented-reality-twitter-conversations/ - so I'm hoping I can do something similarly interesting.Main limitation is getting *all* of the conversation threads. It looks like the context API isn't paginated. But I might be being thick. https://mastodon.social/users/Edent/statuses/109347587353822637
│       ├── 2022-11-15 11:36 bensb: @Edent Excellent project. You might have seen, but there's also this feature request for better 🧵 handling: https://github.com/mastodon/mastodon/issues/8615 https://genomic.social/users/bensb/statuses/109347612990393791
│       ├── 2022-11-15 11:39 Edent: Cor! That @katebevan is good for engagement! Look at all those conversations she's kicked off! https://mastodon.social/users/Edent/statuses/109347627634008550
│       │   ├── 2022-11-15 11:58 Edent: Indeed, how could they be?That means that ID of a reply is different depending on where you see it.So the ID of this post is:mastodon. social /@ edent/ 123456But when you see it on your server, it might appear as:your. server /@ edent/ 987654The #MastodonAPI copes with this really well. But it is a mite confusing to get one's head around. https://mastodon.social/users/Edent/statuses/109347703064222520
│       │   │   ├── 2022-11-15 12:02 erincandescent: @Edent the numeric IDs are not part of the protocol - it's all URL based. Pleroma uses UUIDs for example https://queer.af/users/erincandescent/statuses/109347716173491502
│       │   │   │   └── 2022-11-15 12:06 Edent: @erincandescent oh! That's interesting. Thanks. https://mastodon.social/users/Edent/statuses/109347734283971306

Once you have a tree, you can format the contents however you like.

Grab the code

You can download the code for my Mastodon API tools from CodeBerg. Enjoy!

About a decade ago, I wrote about how the Twitter archive works and where it is deficient. Things have got better, but there are still annoying limitations.

For example, Hannah Kolbeck - founder of the Alt Text Reminder Bot recently pointed out that there's no alt text in the archives.

Here's a snippet of Twitter's JSON for an image I posted:

"media" : [
   {
      "expanded_url" : "https://twitter.com/edent/status/1579574033720705025/photo/1",
      "indices" : [
        "66",
        "89"
      ],
      "url" : "https://t.co/J1hr0ZfbTl",
      "media_url" : "http://pbs.twimg.com/media/FevGM32XEAA0FX2.jpg",
      "id_str" : "1579574018776174592",
      "id" : "1579574018776174592",
      "media_url_https" : "https://pbs.twimg.com/media/FevGM32XEAA0FX2.jpg",
      "sizes" : {
           "small" : {
                "w" : "680",
                "h" : "510",
                "resize" : "fit"
              },
              "medium" : {
                "w" : "1200",
                "h" : "900",
                "resize" : "fit"
              },
              "thumb" : {
                "w" : "150",
                "h" : "150",
                "resize" : "crop"
              },
              "large" : {
                "w" : "1236",
                "h" : "927",
                "resize" : "fit"
              }
       },
       "type" : "photo",
       "display_url" : "pic.twitter.com/J1hr0ZfbTl"
     }
],

Lots of different media sizing options, but no room for accessibility.

By comparison, the Mastodon social network gives you the alt text. Here's a snippet of Mastodon's JSON for the same image which was cross-posted:

"attachment": [
   {
     "type": "Document",
     "mediaType": "image/jpeg",
     "url": "/media_attachments/files/109/145/933/102/890/212/original/84ae501e39f45091.jpg",
     "name": "A sign for priority seating. The pregnant person's face has been replaced by 😍. The person holding a baby has a face of 😫. The elderly person with a cane has 🥴.",
     "blurhash": "UhKdk{0LRit6-:t6WCWC-oxaRmWBozt7xaa|",
     "width": 1236,
     "height": 927
   }
],

Mastodon is a friendlier alternative to Twitter and - mostly - gets accessibility right. There's still some work to do

You can fix Twitter's missing alt text using Hannah's Alt Text Archive Tool. That'll get you a JSON file full of your alt text, which you can use to recreate your archive.

Look, it's obvious that Alan doesn't give a flying fuck about accessibility, so I don't expect this to change any time soon.

Instead, people should do what they did when MySpace went to shit; move to a different platform.

Join Mastodon today!