When you click on a link on my website which takes you to another website, your browser sends a Referer⁰. This says to the other site "Hey, I came here using a link on shkspr.mobi". This is useful because it lets a site owner know who is linking to them. I love seeing which weird and wonderful sites have linked to my content.

It is also something of a privacy nightmare as it lets sites see who is clicking and from where they're clicking. So Mastodon sets a noreferrer¹ attribute on all links. This tells the browser not to send the Referer.

This means sites no longer know who is sending them traffic.

That's either a good thing from a privacy perspective or a disaster from a marketing perspective. Or a little bit of both.

Here's a related issue. When a user posts a link to your website on Mastodon, the server checks your page to see if there are any oEmbed tags for a rich link preview. But, at the moment, it doesn't check your website's robots.txt file - which lets it know whether it is allowed to scrape your content.

In the case of something like Twitter or Facebook, this is fine. If a million users post a link, the centralised social network checks the link once and caches the result.

With - potentially - thousands of distributed Mastodon sites, this presents a problem. If a popular account posts a link, their instance fetches a rich preview. Then every instance which has users following them also requests that URL. Essentially, this is a DDoS attack.

I can fix you

So here's my thoughts on how to fix this.

When a user posts a link to Mastodon, their instance should send a WebMention to the site hosting the link. This informs the website that someone has shared their content. Perhaps a user could adjust their privacy settings to allow or deny this.

The instance would check the site's robots.txt and, if allowed, scrape the site to see if there were any Open Graph Protocol metadata elements on it.

That metadata should be included in the post as it is shared across the network.

For example, a status could look like this:

{
  "id": "123",
  "created_at": "2022-03-16T14:44:31.580Z",
  "in_reply_to_id": null,
  "in_reply_to_account_id": null,
  "visibility": "public",
  "language": "en",
  "uri": "https://mastodon.social/users/Edent/statuses/123",
  "content": "<p>Check out https://example.com/</p>",
  "ogp_allowed": true,
  "ogp": {
      "og:title": "My amazing site",
      "og:image:url": "https://cdn.mastodon.social/cache/example.com/preview.jpg",
      "og:description": "A long description. Perhaps the first paragraph of the text."
      ...
   }
   ...
}

When a post is boosted across the network, the instances can see that there is rich metadata associated with the link. If there is an image associate with the post, that will be loaded from the cache on the original Mastodon instance - avoiding overloading the website.

Now, there is a flaw in this idea. A malicious Mastodon server could serve up a fake OGP image and description. So a link to McDonald's might display a fake image promoting Burger King.

To protect against this, a receiving instance could randomly or periodically check the OGP metadata that they receive. If it has been changed, they can update it.

Perhaps a diagram would help?

What other people say about the problem

David Gerard

@davidgerard@circumstances.run

yes, you should put a cache in front of a blog. nginx and wp-supercache do well. but.

mastodon's auto-DDOS feature is still obnoxious. and in a social network, technically designed in obnoxiousness is incompetent.

i realise it'd need extension of activitypub, but is anyone working on sending prerendered cards with the URL? just to save 1000 servers hammering the URL to generate their own cards locally.

2022-11-28, 14:44 7 boosts 23 favorites

Feedback?

Is this a problem? Does this present a viable solution? Have I missed something obvious? Please leave a comment and let me know 😃

1. Launch a product to great fanfare
2. Spend a few years hyping it as ✨the future✨
3. Stop answering emails and pull requests
4. If you're lucky, announce that the product is abandoned but, more likely, just forget about it.

Open Graph Protocol (OGP) is one of those products. The value-proposition is simple.

• It's hard for computers to pick out the main headline, image, and other data from a complex web page.
• Therefore, let's encourage websites to include metadata which tells our services what they should look at!

OGP works pretty well! When you share a link on Facebook, or Twitter, or Telegram - those services load the website in the background, look for OGP metadata, and display a friendly snippet.

Facebook Meta were the driving force behind OGP - and have now left it to fester.

• The website - https://ogp.me/ - still works.
• But the Facebook OGP Discussion Group is now full of spam.
• The Developer Mailing List is broken.
• The Google Documentation links to a dead Google+ page.
• And the GitHub Page has been archived.

Is OGP finished?

And, that might be fine. Facebook Meta are a small company with limited resources. They can't afford to fund standards work indefinitely. And, anyway, OGP is complete, right? It has all the tags that anyone could ever possibly want. Why does it need any improving?

Well, that's not the case. We know, for example, that Twitter have created their own proprietary OGP-like meta tags. Similarly, Pinterest have their own as well. And even Google are going their own way with Rich Snippets.

This is annoying for developers. Now we have to write multiple different bits of metadata if we want our links to be supported on all platforms.

Standards work is never "finished". Developers want to add new features. Users want to interact with new forms of content.

Tomorrow someone is going to invent a way to share smells over the Internet. How does that get represented in an Open Graph Protocol compliant manner?

<meta property="twitter:olfactory" content="C₃H₆S"> or <meta property="facebook:nose" content="InChIKey/MWOOGOJBHIARFG-UHFFFAOYSA-N"> or <meta property="og:smell" content="pumpkin spice"> or...

We know from bitter experience that having several mutually incompatible ways to implement something is a nightmare for developers and provides a poor user-experience.

So we create standards bodies. They're not perfect, but a group of interested folks can do the hard work to try and satisfy oppositional stakeholders.

This is my plea to Facebook Meta. If you're no longer interested in improving OGP, OK. You do you. But hand it over to people who want to keep this going. Maybe it's the W3C, or IndieWeb, or Schema.org or someone. Hell, I'm not busy, I'll take it on.

Remember, if you love something, let it go.