Every site that I sign up for asks me to upload an avatar to represent myself. Whenever I change my photo, I have to log in to a hundred sites and change it there⁰.

Perhaps they could all use Gravatar - but that's a centralised service¹ and doesn't work with wildcard email addresses. Libravatar also relies on email addresses and requires implementers to set up new DNS entries.

So I'm proposing .well-known/avatar. Here's how it works (for now). I'd like your feedback before going further².

I sign up to a service and use the email address whatever@shkspr.mobi.

The service looks up my avatar using a well-known path. For example, request https://shkspr.mobi/.well-known/avatar?resource=acct:whatever@shkspr.mobi and you'll get back this JSON:

{
    "subject": "acct:whatever@shkspr.mobi",
    "links": [
        {
              "rel": "http:\/\/webfinger.net\/rel\/avatar",
             "type": "image\/webp",
             "href": "https:\/\/shkspr.mobi\/.well-known\/avatar\/avatar-1024.webp",
            "sizes": "1024x1024"
        },
        {
              "rel": "http:\/\/webfinger.net\/rel\/avatar",
             "type": "image\/jpeg",
             "href": "https:\/\/shkspr.mobi\/.well-known\/avatar\/avatar-512.jpg",
            "sizes": "512x512"
        }
    ]
}

That's a slightly enhanced https://webfinger.net/rel/#avatar which adds a sizes parameter. The service can then pick the appropriate MIME and size.

Alternatively, you can request the same URl but with a header of Accept: image/gif and receive the default sized avatar in that specific format.

Try it by running:

curl -H "Accept: image/avif" https://shkspr.mobi/.well-known/avatar/ --output "test.avif"

You should receive an auto-converted version of my avatar.

Some Thoughts

Please add your thoughts to the comments box. Here's some feedback I've received so far.

Perhaps this is too complicated? What's wrong with just serving up an image when the URl is requested? That would make it easier for static sites.

Simon Josefsson

@jas@fosstodon.org

@Edent Thinking about this, while I like content negotiation as a clever hack, I wonder if maybe it isn’t too clever. The nice thing with WKD is that you can deploy it with any normal static HTTP file without any special magic. Maybe the protocol could be dumbed down to simply rely on WKD-style URLs? I’m not sure how to configure my web server (Apache) for your avatar well known URL with negotiation magic.

2025-10-23, 16:50 0 boosts 1 favorites

What about a size parameter?

Chip

@chip@talking.dev

@Edent It'd be nice if the query could limit the size of the avatar being returned. If only there were `Accept-Max-Size`, but maybe a query param? I wouldn't want my performance taking a dive if Alice has a 35M avatar that my client starts downloading. If my client had requested with `max_size=3072` I'd rather not see the avatar than degrade performance/pull excess data

2025-10-23, 15:02 0 boosts 1 favorites

Will anyone actually use it?

Andreas Gohr

@splitbrain@fedi.splitbrain.org

@Edent good luck with getting the hundreds of services to implement it. I mean it. it would be awesome and you might be well connected enough to make it happen.

2025-10-23, 15:03

What about hashing the email?

David Bushell 🎮

@db@social.lol

@Edent would using a hash of the email address in its place improve privacy? 🤔

2025-10-25, 11:52 0 boosts 0 favorites

You've already given the service your email address, and your domain already knows your account name - so there's no privacy leak here. Obviously, a service shouldn't hotlink to your avatar image.

How about DNS?

I like it. Is there an argument that service / endpoint should be specifiable at the DNS level?As others in your comments pointed out, if your site is currently just static, some users might prefer to run an entirely separate dedicated avatar service.

— Emily Shepherd (@emi.ly) 2025-10-25T11:57:43.456Z

Personally, I think that's a bit complicated, but I'm happy to be convinced.

Is this restricted to email?

No! For example, if you know my GitHub username then you should be able to get the avatar from https://github.com/.well-known/avatar?resource=acct:edent

How can a service tell if the avatar has been updated?

Perhaps a hash, timestamp, or something else?

Can requests for multiple accounts be sent at once?

I'm not sure how / if WebFinger handles this. I suppose there ought to be some limit to avoid overwhelming a server.

Proposal

I think the default should be to return an image.

If an accept of image/… is requested, the server should try to return an image in that format.

If an accept of application/json or similar is requested, the server should return a JSON document listing the available avatars.

I don't think a ?size= GET parameter is necessary; services can resize once they've downloaded, or use the JSON document to get the right size.

A limited amount of alt text could be added using the title attribute in the JSON.

Before I start writing up anything formal - I'd love your constructive criticism on this.

So I want to host my own Git instance for my personal projects. I'm experimenting with https://git.edent.tel/

It isn't quite self-hosted; I'm paying PikaPod €2/month to deal with the hassle of hosting and updating the software. I get to point my domain name at it which means I can always change the underlying service if I want. For example, it uses Gitea and I might want to switch to Forgejo later.

So far, it works. But there are a few significant drawbacks.

Network Effects

A large service like GitHub has network effects which are incredible. It feels like 90%+ of all developers have an account there. That means if someone wants to leave a comment or send a PR there is no barrier to entry. That's huge! There are a bunch of popular FOSS projects which make me sign up for yet another service when all I want to do is send a simple bug report which I find deeply annoying.

Luckily, Gitea has built in support for various OAuth providers. So I've set up single-sign-on with Gits Hub and Lab.

I asked people how easy it was to use - most people were able to use it, although a few people wanted a local-only account.

But is is still a bit of a faff. Even a little bit of hassle turns people away.

Forking isn't Federated

Suppose you want to make a Pull Request or just take a copy of the code. At the moment, you have to create a fork on my server. There's no way to easily fork something to your GitHub or personal server.

You can git clone the repo to your local machine, and you can manually move it elsewhere, but there's no way to send a PR from your repo to mine.

There's also no way for users to find other forks. Perhaps the upcoming ForgeFed proposals will fix things - but it doesn't exist yet.

As pointed out in "Let's Make Sure Github Doesn't Become the only Option" - most of the tooling of git hosting platforms isn't adequate for modern needs.

Discoverability

The easiest way to find code at the moment is to search GitHub. Moving my stuff to a different site means it will only be discovered by a general search engine.

I want people to find and use my code. If that's hard, they won't. I can point existing users to the repo - but it still cuts down on discovery.

Admin Hassles

Although PikaPods takes care of all the hosting administration, there's still the faff of setting up all of Gitea's options.

If I get hit by an automated spam attack, it'll be up to me to clean it up.

I'm not sure if I have the time, patience, or expertise to correctly and safely configure everything. Time spent administrating is time not spent coding.

Sponsorship

I get a little bit of money when people sponsor me on GitHub. There's no "sponsor" option on Gitea and, even if there was, the network effects of GitHub are substantial. Getting people to enter their credit card info into a random site isn't as convenient as clicking a button in GitHub.

Now What?

My most popular Github repo has around 140 contributors. I genuinely don't think I could attract that many people to OAuth onto my personal git hosting service.

Gitea seems to have a mixed reputation. But it's the only one offered by PikaPods.

There are interesting discussions about how to replace GitHub but they're only in the early stages.

Although €2/mo isn't a huge amount, I've gotten used to having free services on GitHub / GitLab / CodeBerg.

So this, I think, is my plan:

1. Leave my popular / sponsored repos on GitHub
2. Move my smaller repos to https://git.edent.tel/
3. Create new repos in there as well

I'm also going to look for a hosted Forgejo instance which lets me use my own subdomain - hopefully at a cheaper or comparable price. If you have any recommendations - please let me know!

It's expensive to self-host, there are moderation problems, and the difficulty level is too high for most people.

But, in my opinion, I think he misunderstands something about self-hosting because, as a term, it is both misleading and unhelpful. When people say "Defund The Police" what they mean is "Move funds away from miliary style policing and give it to trained mental health professionals" - what people hear is "Abolish the police and let anarchy reign".

The ability to "Self Host" doesn't just mean "run this on a Raspberry Pi in your cupboard and be responsible for constant maintenance". Yes, you can do that if you're a masochist, but it isn't restricted to that.

To me, "Self-Hosting" means "I am in control of where I host something". I currently pay a company to host this blog. It has previously been hosted on Blogger, WordPress, my own VPS, and a variety of other services. Tomorrow I could decide to host it with a big company, or I could run it from my phone. I get to choose. That's what "Self-Hosting" is - a choice in where to host.

Similarly, Mastodon allows me self-host my account. I can have my content on one of the big servers and let them do moderation, storage, and maintenance for me - or I can move my account anywhere I choose. To a server in my cupboard and back again.

Email is similar. I know people who've gone from CompuServe, to HoTMaiL, to Gmail, to their own domain, then to OutLook. Their address-book moves with them. Forwarding rules ensure incoming email is routed correctly. They can choose to actively moderate spam, or outsource it. They can pay a company to host, keep backups in their basement, or watch adverts in return for services.

I agree with nearly everything Mathew says in his post. It is absurdly privileged to think that running your own services is something normal people want to do and are capable of doing. Strong regulation helps everyone, people want simplicity, and ecosystems can be fragile.

But witness all the people moving over from Twitter to new networks. Do they care where their data is hosted and how it is maintained? No! But they want to move their social graph with them. And when BlueSky and Mastodon collapse, people will want to move again.

In the UK, I have the ability to move my phone number between hundreds of providers. If I'm particularly techy, I can even run my own infrastructure and route the number there. People love the fact that they can leave crappy service providers and move somewhere cheaper or with with better customer service or whatever it is they value. I think that's a form of self-hosting; I get to choose who provides my services.

Similarly, I believe people have a desire for "self-hosting" which is difficult for them to articulate. They want to move their data around - be it old photos, a social graph, or a username. Most of them don't really care about the underlying technology (and why should they?) but they do care about continuity of service and being able to escape crappy service providers.

So, that's my reckons. Self-Hosting means you can choose where to host, and I think most people can find value in that.

What do you think?

For those who don't know, OSM (OpenStreetMap) is like the Wikipedia of maps. Anyone can go in and edit the map. This isn't a corporate-controlled space where your local knowledge is irrelevant compared to the desire for profit. You can literally go and correct any mistakes that you find, add recently built roads, remove abandoned buildings, and provide useful local information.

Editing the full map is... complicated. For simple edits like changing the times of a postal collection, there are simple forms you can fill in. There's also an aerial view so you can drag and drop misplaced locations. But for anything more complicated than that, you'll need to spend some time understanding the interface. There's a friendly community who are happy to check or correct your submissions.

I'll be honest, I don't use the web editor much. Instead, I use the Android app StreetComplete. It's like an endless stream of sidequests. As you travel through the world, it will ask if a shop is still open, or if the highway is lit, or how many steps there are on a bridge, or whether a playground is suitable for all children, or if restaurants serve vegetarian food, or if a bus-stop has a bench, or... the list is almost endless!

I use it when I'm walking around somewhere new, or on holiday, or waiting for a bus. I used it so much that, for a short while, I became the #1 mapper in New Zealand!

So get stuck in! Make mapping more equitable and more accurate.

In it, they make the convincing case that phones are perfectly capable of hosting websites and - if we want more people to escape the walled-gardens - this could be a good way to get people back into self-hosting.

I loved hosting a small site on my Nokia N95 back in the day, and I'd be overjoyed if modern phones allowed this. But there are a few pitfalls.

Connectivity is the main one. If you're sat at home on WiFi all day, or only ever in an area with good cellular reception, then this is great. But the second you get on an underground train or climb a mountain your website goes offline. Perhaps it might be nice to introduce a little randomness into our sites' availability. But most people want their site to work even if their phone is in a deadspot.

Cost is also a factor. If you're on an unlimited data plan, then you're fine. If you're on a cheap PAYG plan with only a few GB then your website becomes a liability. If one of your links becomes suddenly popular, you might burn through your monthly limit in a few minutes.

Battery life is the next big problem. Again, if you're sat in an office all day with your phone plugged in, you're fine. But if you're out and about, running a server and delivering data is power-hungry. Do you want your phone to die because your cat photo went viral?

Security also becomes a factor. Web servers are constantly under attack. Will your private data be compromised if your phone's webserver hasn't received the latest patch? And if an attacker gets onto your phone, can they then infect the rest of your home network?

All that being said... I still like the idea! I have a few old Android devices which I use as IP Cameras. They stream pictures and videos across my local network - and I can control them over a web browser when I'm at home. But would I want to expose them to the whole Internet? No!

With the price of SBCs coming down, I think it makes more sense to have a "website in a box" device. A little Raspberry Pi which is only running enough of a server to power WordPress or similar. Hardened against attack, and easy to manage and edit from a cheap mobile phone. Perhaps give it battery backup and a SIM card if you want redundancy. Boom! Instant personal webserver.

Yes, our phones should be under our control - and I certainly support anyone who wants to run a server from their device - but it seems to me that there are several hurdles which make it impractical.

In one sense, this is sad. A set of useful public services are being cut off from their audience. My friend, Bill Thompson, described this as "unnecessary disruption" I, on the other hand, think that creative destruction is sometimes a necessity.

You can listen to more of the discussion on the Gareth and BillCast about 19 minutes in.

Look, first off, it makes sense for organisations to put warning messages where their audience is likely to see it. No one sensible begrudges that. Twitter is a channel just as legitimate as radio, TV, or loudhailers in the street.

The problem comes when that's the only channel.

Twitter, somewhat accidentally, set itself up as a replacement for the public sphere. Now it has made it clear exactly what the cost of "free speech" is - alert bots are abandoning it.

Almost all the bots I mentioned above have websites. You can go check them, if you remember - but that isn't the same as having their warnings mixed in with your daily scrolling.

Some of those bots also have RSS feeds. But the RSS model doesn't lend itself to your peer group reposting urgent content into your feed.

And none of those websites and feeds are easily discoverable if you're in a new city.

So, how could this be solved with ActivityPub / Mastodon / Fediverse?

On the one hand, it's pretty easy for any organisation to set up a decentralised service with their alerts. Visiting https:// mass_transit.ak will show you all of Arstotzka's transport bots. You could subscribe to @flood-warnings@alerts.gov.ak for Arstotzka's official warnings.

On Twitter, it's easy to search "Train Delays London" and see everything. Hopefully including one of the accounts which gives you official alerts.

But, one of the interesting problems with a decentralised service is that search is (deliberately⁰) difficult. Unless someone you're already following happens to share an alert, it can be very difficult to find relevant posts and accounts.

There is something of a solution which, I think, could be helpful. ActivityPub defines a way to add location to a post.

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Note",
  "name": "Arstotzka' Flood Service:",
  "content": "Heavy rains! Expect flooding and major disruption.",
    "location": {
       "name": "Arstotzka Central District,
       "type": "Place",
       "longitude": 12.34,
       "latitude": 56.78,
    }
}

At the moment, there's no way to search for geo-tagged posts - but it doesn't have to be that way. It is relatively simple to use the Haversine formula to select database entries which are within a geographic area.

But, the key problem is too much decentralisation. It goes against the credo of redecentralization to have a single server called official-alerts.com - having a single destination to search is convenient, but provides for a single point of failure.

A country might have alerts.ak which has multiple accounts, or might have single-use instances like flood.ak and fire.ak and traffic.ak. They might be federated to, for example, official-alert-service.eu.

At which point, I get stuck.

How does a user discover these accounts?

There are three options, as I see it:

1. Advertising. The organisation promotes these accounts via other channels.
2. Serendipity. A user sees that another user has shared one of these accounts.
3. Better discovery tools. A user's instance could recommend accounts to follow based on geography or other factors.

Thoughts?

A few months ago, we landed in a foreign country, opened the same Uber app as we used back home, and booked a cab. It just worked. I didn't need to register for a different version. I didn't need to create a new account. I didn't need to add a new credit card. That's the sort of seamless experience which can only come from a centralised service.

But, hey, we're all moving to a ReDeCentralised Federated Future. Nae king! Nae quin! Nae laird! Nae master! We willna' be fooled again! So let's think about how a decentralised Uber would work.

Broadcast

In the days before mobile phones, we could have used something like CB radio. Broadcast a message to everyone in a few miles saying you want to travel from point A to point B.

Cabbies, or their agents, could reply to you with their availability and prices.

But that puts a lot of work on you to manage all the replies. And you have no idea of the driver rating. And you'll be broadcasting your location to all and sundry.

And, sadly, modern handsets have no concept of a broadcasting in a P2P manner.

Perhaps a little centralisation?

In most parts of the world, taxis are regulated. You don't want rapists, thieves, and murderers driving you home late at night. You want a predictable fare calculated by an honest meter.

So let's say you have an app which knows your location, and send your request to the taxi-regulation-boards which cover your area. They can then broadcast out the request to all drivers, they confirm the fare, and then send it back to you.

I guess that could work. Think of all the taxi-regulators as little ActivityPub servers, and the cab drivers getting a stream of requests on their federated timeline. Your phone could receive "accepts" from multiple drivers and you could pick the one you like.

But how would your phone know which local taxi services are available? You would need a centralised list of them!

FOAF and Signed Claims

One of the supposed advantages of Uber is that it allows participants to rate each other. Passengers don't want to get in a car with an arsehole. Drivers don't want to pick up people who litter.

How would that work without a central entity? Could there be a solution with Verified Credentials? Have all participants write cryptographically signed ratings to a *spits* Blockchain?

Reputation Systems are hard.

User Needs

Of course, before sketching out such a system, it probably helps to sketch out some vague thoughts on what users actually want from a taxi system - and how decentralisation would help.

Firstly, who are the users?

As a driver, I don't want to have multiple phones and apps stuck to my dashboard.

As a user, I only want to use a reputable driver. But who determines reputation? What options do I have if something goes wrong?

Basically - what are the advantages of decentralisation in the taxi space? Users want competition on price and service - which they don't get if Uber are a monopoly. Users also don't want to deal with dangerous drivers - which is easier to enforce if there is only a single platform.

Drivers don't want their livelihood ruined by unjust reviews - how could that be enforced with a decentralised platform?

We haven't even got to payments yet!

Basically, it's not clear to me that is a compelling user need for decentralisation. Don't get me wrong, I like it as a concept. But I see it as exchanging one set of wicked problems for another.

Does this already exist?

Well, ish. There are various different attempts to build federated delivery services and the like.

But I am very interested in being proved wrong! Is there a decentralised takeaway service I should investigate? Are there federated hairdressers? Does anyone offer an open protocol for domestic cleaners?

Anyway, that means there isn't one Mastodon website. There are many. There is only one Twitter. There is only one Facebook. There is only one Instagram. If you want to interact with Twitter/FB/Insta then you have to do it on those websites, or via the official apps.

Mastodon is decentralised. I am on Mastodon.Social, and you are on Mastodon.me.uk. Those are different servers. But we can still see each others' messages.

All of those different servers have multiple users. Some of the big servers have hundreds of thousands of accounts, some of the smaller ones only a few dozen. We're all apart, even when we're together. This is the joy - and the sorrow - of decentralisation.

What's the optimal size of a server? Bigger ones might be slower, or they may benefit from efficiencies of scale. Smaller ones might be more nimble, but may not have the resources to keep the server going.

Let's take a look at PixelFed - a federated version of Flickr / Instagram. There are about 280 different PixelFed instances you could join. Here's a graph showing the number of users on the top 25 instances:

Pretty classic "long-tail" there. A dominant main server. A credible second place. A distant third. And then all the rest quickly descend under 100 users.

I want to be very clear - I don't think that's necessarily a bad thing. It is entirely possible to have a sustainable server which only has one user.

For example, the Raspberry Pi foundation has an instance at aspberrypi.social. It only has a single account you can follow - @Raspberry_Pi@raspberrypi.social.

Is that the future?

What happens if everyone has their own instance? Would managing spam and blocklists on the Fediverse be as difficult as managing them for email?

Inside of you there are two extremes

Consider email. Almost everyone is on GMail, O365, or one of a limited number of big email providers. Even if you're a digital extremist and have your own domain name, you almost certainly don't run your own email server. It's just too complicated and no one wants a full-time unpaid job keeping up with the admin of configuring, patching, and defending their mail.

Similarly, I doubt anyone wants to spend much more than a couple of hours per year setting up and maintaining their federated social network app.

Those of us vain / paranoid enough to want to be @MyName@my-domain.xyz will almost certainly just rent computing power and space from one of the big Mastodon providers.

What about big businesses?

I love that idea. But businesses don't run servers any more. If you're a small business, you just outsource everything to Google or Microsoft. If you're a big business... the same!

It's easier and cheaper to pay dedicated professionals to run specialist appliances than it is to bring them in-house.

Predictions

Making predictions is a fool's game. So here are my predictions for the future of Federated Services:

• Die-hard geeks like me will still run our family's Fediverse apps from a Raspberry Pi tucked in a cupboard.
• Small sellers will offer Fediverse packages just the same way as they do one-click WordPress installs. But they'll mostly all be running on AWS anyway.
• Twitter - or someone like them - will start offering "Fediverse-as-a-Service". You can be @Bob@Mastodon.Your_Domain.com - but it'll be Twitter under the hood. Just like GMail for domains.
• People will have multiple instances they need to be on. Work and personal, for example. Just as some of us have a family email which is different from the friends one. But they'll all come through to a single app.
• Facebook - or someone like them - will gradually add "enhancements" to their service which will cause it not to work other Federated systems. The first time this happens it will be excused as a mistake. But, just like how Google killed off XMPP, it will eventually become a silo. Not many people will notice, because the majority of their friends are all on the same instance.
• Someone will invent a server which runs on your Android phone. It would revolutionise decentralisation and federation. But it chews up battery and Apple bans the iOS version for no good reason. So it never takes off except with a few people prepared to root their devices and carry an extra battery.
• A nation-state will insist that every citizen and resident must have an account on the national Mastodon. Perhaps in order to listen to the thoughts of Dear Leader™. Perhaps for some sinister monitoring purpose. If you want to talk to your buddies in that region, your server may have to Federate with something running old, outdated, or hostile software.
• Geeks like me will rage that this all could been avoided if everyone bought their own Raspberry Pi and learned half-a-dozen simple Linux commands.

Perhaps that's pessimistic of me? I think the pendulum will swing a few more times at least.

There might be a dozen different phone networks in your country. Yes, they Federate with each other, but they all happen to be running close-to-identical networking gear made by Nokia or Huawei. Yes, you can put your SIM in any device, but chances are it is either Android or iOS.

We'll be stuck with the illusion of Federation.

I've had to manually change it on a dozen different Slacks, a bunch of social networks, a few forums, all my email accounts, and I'm still not done.

I just want to change my photo once. Because I'm vain and lazy.

For a nerd like me, the solution is obvious:

• My latest avatar image has a permanent web address - https://edent.tel/avatar.
• When I register for a service, it should ask me for my homepage address and automatically detect the URl for my avatar.
• Periodically, it should check for an update.

What about Gravatar?

The Globally Recognised Avatar project from WordPress is supposed to be a solution to this problem. But it doesn't work, for two main reasons.

1. Not everywhere uses it.
2. Works on a per-email basis.

Nothing we can do about (1), but I find (2) is annoying. I use a different email address for each website I use. Which means I have dozens of Gravatars!

Because Gravatar uses MD5 hashes, there's no way around this. It's also a (minor) privacy concern.

What about Libravatar?

The Libravatar project is basically an open source version of Gravatar. It also uses hashes for email addresses. And, sadly, very few sites use it.

What about Webfinger?

The documentation for Webfinger is comically absent.

What about Microformats?

Standards like microformats2 let you add an image to your profile.

Similarly, Schema.org an add an image to your personal metadata.

Social Networking Sites

This sort of exists. Services like Avatars.io let you use you Twitter & Facebook avatars as a URl - for example avatars.io/twitter/edent.

Sadly, the service isn't maintained any more, has broken images for Instagram, and doesn't include newer services like GitHub.

Regain Control

There are two fundamental mistakes we're making.

• An email address is not an identity.
• A 3rd party service is not an identity.

Given that Gravatar is promoted by WordPress - the largest website provider on the planet - and it still isn't universally accepted, I don't think there's any hope for smaller services.

So, I guess what I need is an app which can log in to all my accounts and automatically change the avatar whenever I want.

Or is there some other practical action I can take?

Most days I use DropBox. But it's just FTP, but a bit easier to use and automate.

I waste a lot of time on Slack. When I explain it to old-school nerds, I say it's IRC - but developed by someone who gives a damn about user experience.

Most people in the world don't have access to WWW. Instead, they use Facebook which gives them a much simpler way to post photos and share their thoughts. It doesn't ask them to hand-edit an .htaccess file.

I don't know anyone who uses Listserve. It turns out that Telegram is faster, more convenient, and doesn't require esoteric commands.

Indeed, why bother with Email? You don't need to learn how to configure SMTP when you have WhatsApp.

What other, classic, decentralised Internet tools have been turned from open protocols to closed and proprietary services?

A large part of this is our fault. And, by us, I mean gatekeeping nerds. We developed tools which were unforgiving. We had no interest in the "soft" skills of empathy. We were too socially-awkward to speak to real users. We were insular and we liked it! Worse than that - we revelled in it.

LOL! ROFL! LMAO!

And then Apple eats everyone's lunch by relentlessly focussing on being user friendly. Good for them. But it means handing over control to a single organisation.

I don't claim to be any good at user experience - far from it. But I despair at some of the redecentralised efforts I see springing up. They are technically brilliant, and follow the open-source philosophy of scratching one's own itch. And they all - without fail - are terrible to use.

Redecentralisation won't happen because of us nerds. It must happen despite us. Despite our ingenuity and despite our self-infatuation. It must be inclusive, and put user-needs at its very heart.

MPEG-DASH is a simple method of streaming videos which doesn't require any specialised server software. You convert a high resolution video into a series of smaller resolution videos. You chop each of the videos up into several chunks. As the video plays, your browser then decides which chunk of the video to load next depend on the bandwidth available to it. Easy!

Aim

From the command line:

• Upload a video to YouTube
• Download transcoded video in several formats suitable for MPEG DASH Streaming
• Generate DASH manifest
• Generate the HTML5 markup

Uploading

There are a couple of YouTube video uploaders. These are useful if you've got a video on a server, or don't want to waste a browser window.

• porjo/youtubeuploader has binaries available for Linux/Mac/Windows.
• tokland/shoogle a Python interface for all of Google's APIs.

I'm using Porjo's client. After following the authentication instructions, the output is something like:

./youtubeuploader_linux_amd64 -filename testing.mp4

Uploading file 'testing.mp4'...
Progress:     1.37 Mbps, 9852099 / 9852099 (100.000%) ETA          0s
Upload successful! Video ID: 6NV5heuaVuc

Downloading

The only program worth using is youtube-dl.

We can see every video format that YouTube has created using:

youtube-dl -F 6NV5heuaVuc

That will return something like:

It may take several minutes for YouTube to generate all the video formats - the length of time will depend on the length and quality of the video.

We want to download all the DASH videos and the DASH audios. We also want the subtitles - even if they are autogenerated. Finally, we want the thumbnail image for the HTML5 poster.

The youtube-dl command to download all the DASH audio is:

youtube-dl -f 'all[vcodec=none]' -o '%(title)s-%(format)s.%(ext)s' 6NV5heuaVuc

To get all the DASH videos:

youtube-dl -f 'all[acodec=none]' -o '%(title)s-%(format)s.%(ext)s' 6NV5heuaVuc

This will create a separate MP4 video for each resolution, and separate audio file for each bitrate.

To generate a single thumbnail, run:

youtube-dl --write-thumbnail --skip-download 6NV5heuaVuc

To download the subtitles:

youtube-dl --write-auto-sub --sub-format vtt --write-sub 6NV5heuaVuc

Convert to DASH

For this step, we need the open source Bento4 toolkit.

The video should already have "fragments" - the chunks which make up the stream - you can check this with:

./mp4info "Video-133 - 426x240 (DASH video).mp4"

In the output, you should see:

Movie:
  duration:   0 ms
  time scale: 90000
  fragments:  yes

The audio downloaded from YouTube will not have fragments, so you'll need to run this command on every audio file you've downloaded:

./mp4fragment "Video-140 - audio only (DASH audio).m4a" high.m4a

Combine and create manifest

Let's take those audio and video files and make ourselves a manifest!

./mp4dash -o output-video 133.mp4 135.mp4 136.mp4 137.mp4 160.mp4 high.m4a low.m4a

This creates the folder output-video and creates folders for the chunked audio and video.

It will also produce the MPD - a manifest file which references all of the chunks.

We're nearly ready to play back.

Play DASH Video in the browser

As far as I can tell, no browser natively supports MPEG-DASH. You will need to use the latest version of dash.js to make everything work smoothly.

Add this piece of JavaScript to your page:

<script src="https://cdn.dashjs.org/latest/dash.all.min.js"></script>

Then reference the MPD in a video element:

<video width="1080" data-dashjs-player src="stream.mpd" controls poster="example.jpg">
    <track kind="subtitles" src="en.vtt" srclang="en" label="English">
</video>

Demo

Here is an adaptive stream of one of my videos - hosted natively on this site, with no server-side streaming.

Problems?

Fallback video

Some browsers can't play back DASH, even with the JavaScript - so you can put a fallback video in the HTML.

<video width="1080" data-dashjs-player controls >
    <source src="stream.mpd"   type="application/dash+xml">
    <source src="fallback.mp4" type="video/mp4">
</video>

HLS

Because Apple are special, they don't support DASH on its phones. So you'll need to use their proprietary HLS streaming solution. You can use Bento4 to create HLS in exactly the same way as you create DASH.

You may also need to use hls.js JavaScript to get things to work smoothly.

Subtitles

You can use DASH native subtitles, or HTML5 subtitles.

I found that the HTML5 subtitles downloaded by youtube-dl contain some strange CSS which didn't work well with the DASH video. You may need to test to see how it works for you.

All in one command?

Hahahaha! Not yet! If you know of a good way to do this - let me know!

Most of our domestic energy use is in the evening. So, when I'm out at work I can schedule the tumble dryer, robot vacuum cleaner, and WiFi rice-cooker to consume energy when the sun is shining. The rest is sold back into the grid for my neighbours to use.

Wouldn't it be great to capture that energy and use it to power my lights and games console in the evening? Yes! Enter the Maslow - a 2kWh wall mounted battery designed to capture solar power and put it to use when it is most needed.

The Maslow gobbles up the surplus power and stores it for later use.

I received a subsidised Maslow battery as part of Project ERIC.

Project ERIC (Energy Resources for Integrated Communities) is an initiative bringing solar PV power and smart energy storage to up to 100 homes in Rose Hill, East Oxford. Project ERIC is led by Moixa Technology and Bioregional and is part-funded by Innovate UK.

Installation

It's a relatively svelte box - smaller than a normal domestic boiler - and fits neatly onto my interior wall.

Here it is before installation and without any batteries.

As you can see, the batteries are designed to be replaceable. If they die unexpectedly, or technology leaps ahead, you should be able to keep your existing installation and just swap out the cells for something newer.

A couple of screws - and a bit of swearing - and it was fitted.

Installation took around 3 hours. The battery needs its own RCD in your consumer unit. A few holes had to be drilled in order to get all the wiring hidden away.

The fans on the side run when the batteries are charging or discharging. They're a little louder than a typical PC - and will be easily drowned out by a boiling kettle or rumbling tumble-dryer. You probably wouldn't want this installed in a bedroom or living room - but it's fine for a kitchen or loft.

They're needed for cooling these chunky capacitors.

Down in the bottom of the unit, you'll find the fairly unremarkable control circuitry.

Wait... wait a moment! Is that...? OMG! This thing is RASPBERRY PI POWERED!

Wooo! I think this is the 5th Pi I have controlling something in my home ?

There's a WiFi antenna which runs to the outside of the unit. You can also connect directly to the Pi's Ethernet port if you don't have a strong WiFi signal.

Hidden behind a panel is a full range of blinkenlights.

A mini-USB port is present for debugging purposes, along with some diagnostic LEDs. Apparently there's also Zigbee for controlling smart appliances in the future.

Network and API

As part of the Project ERIC trial, I've agreed to have my battery's energy usage monitored.

The Maslow collects details of usage every minute, these data are then combined to provide an overview of what our village is consuming and producing:

I also get access to a personalised dashboard - which will hopefully show more details soon!

There's nothing really to control on the battery - it just sits there doing its job. There is an API, more details in a future blog post!

Cost

Because I'm on a trial of this hardware, the total cost of the Maslow battery and installation was £600 (including VAT@ 20%).

What do you get for that?

A typical mobile phone battery holds around 0.015kWh. My electric car has a battery capacity of 19kWh. My house's idle usage overnight (lights, chargers, servers, devices on standby) is around 100 Watts. So a typical night (2200 - 0600) will use around 1kWh.

My electricity costs me 13.08p / kWh - sign up to Ovo Energy and we both get a £25 Amazon voucher.

So, assuming the battery is fully charged once during the day and then fully discharged overnight, how long before the device pays for itself?

£600 / (£0.1308 * 2) ~= 2,300 days.

2,300 / 365 = 6 years 4 months.

At the unsubsidised price of £2,000 the repayment period is around 21 years. That's essentially the guaranteed lifetime of most solar-power systems. It also assumes that energy prices stay static.

Of course, the solar panels don't always generate enough - let alone enough surplus - to fill the battery:

Edent's Solar Panels

@Edent_Solar
Automated

Pretty grim. @edent's solar panels generated ~1.42kWh, earning £0.25
FAQ: shkspr.mobi/blog/solar-gra…
#edentSolar pic.x.com/92wikj4nuy

---

There's no doubt the Maslow is an expensive "lifestyle" product. It's for saving the planet rather than fattening the wallet. Built in to the cost of a typical home it could be a real game-changer in how energy is used in this country.

As an aside, the UK Government pays me for every kWh the solar panels generate. They don't measure the actual amount I export. So the Maslow doesn't reduce the Feed In Tariff (FIT) payments.

Features

OK, other than offsetting my grid usage, what does this big ol' box do?

• Powering DC lighting. I decided not to go for this option. I already have energy efficient bulbs, so the extra cost for replacement didn't make sense for my needs.
• DC Power Circuitry. I've not yet installed this, but it should let me charge USB devices without needing to do a costly AC/DC conversion.
• Works during a power-cut. For safety reasons, the Maslow won't power normal household sockets during a power-cut, but it will allow you to use the aforementioned USB sockets and LED lighting. Handy in an emergency
• WiFi for monitoring usages. Which of course means…
• An API available for HACKING!

Alternatives

There are number of alternatives to this sort of technology.

• Divert to immersion heater. I currently have a Solar iBoost which heats up my water and reduces my gas usage.
• Build it yourself! If you have lots of spare batteries and are a qualified electrician.
• Car storage. At the moment, I've been using my electric car to store surplus electricity - but I'm not always at home during the day. I can't "reverse the polarity" and feed back from the car into the house.
• Tesla Powerwall - the elephant in the room!

Tesla battery has the cachet of the Tesla brand - but hasn't yet been released to the UK public. At around US$3,000 for a 7kWh model, the cost/power ratio is pretty good - but it is significantly larger in size to the Maslow.

The Tesla is only generally available in the USA, so it will be very interesting to see what their price is for the UK market. The Maslow is designed for typical UK use, which may be different from Tesla's focus - UK houses don't tend to have air conditioning, for example.

Final Thoughts

I've only just had the batteries installed - but it's obvious that the Maslow provides a vital way to reduce peak grid usage from domestic properties. Even if it is a little costly.

I'm delighted to be part of such an innovative trial - I'd love to see every home in the UK generating and storing their own energy.

So, here's a quick tutorial on how to download all your videos - and metadata - from YouTube.

The Official Way

Google offers a "takeout" service which will allow you to package up all your YouTube videos for export.

It creates a multi-gigabyte archive - which isn't particularly suitable for hosting elsewhere. Once the archive is created, you have to download it in 2GB chunks. The archives are only available for 7 days - so if you're on a normal speed Internet connection, you might not be able to grab everything.

If you do manage to download everything - you'll find another problem. The files are enormous because you're downloading the originals - not the web-optimised versions.

So, how can we download high-quality, low-filesize copies of the videos suitable for HTML5 use?

P-p-p-p-pick Up Some Python

We'll be using the excellent YouTube-DL - make sure you have the most recent version installed.

The Google Takeout from above contains a file called uploads.json - it has a list of every video you've uploaded and some associated metadata:

...
,{
  "contentDetails" : {
    "videoId" : "2gIM9MzfaC8"
  },
  "etag" : "\"mPrpS7Nrk6Ggi_P7VJ8-KsEOiIw/7fgl8GbhgRwxNU3aCz9jzUB_65M\"",
  "id" : "UUAEmywW2HASHP0MohSNIN0qHLpdeAxkQv",
  "kind" : "youtube#playlistItem",
  "snippet" : {
    "channelId" : "UCyC5lCspQ5sXZ9L3ZdEEF2Q",
    "channelTitle" : "Terence Eden",
    "description" : "Look at me walk to work!",
    "playlistId" : "UUyC5lCspQ5sXZ9L3ZdEEF2Q",
    "position" : 168,
    "publishedAt" : "2010-12-02T10:03:15.000Z",
    "resourceId" : {
      "kind" : "youtube#video",
      "videoId" : "2gIM9MzfaC8"
    },
    "thumbnails" : {
      "default" : {
        "height" : 90,
        "url" : "https://i.ytimg.com/vi/2gIM9MzfaC8/default.jpg",
        "width" : 120
      },
      "high" : {
        "height" : 360,
        "url" : "https://i.ytimg.com/vi/2gIM9MzfaC8/hqdefault.jpg",
        "width" : 480
      },
      "medium" : {
        "height" : 180,
        "url" : "https://i.ytimg.com/vi/2gIM9MzfaC8/mqdefault.jpg",
        "width" : 320
      }
    },
    "title" : "Walking To Work In The Snow"
  },
  "status" : {
    "privacyStatus" : "public"
  }
}, {
...

So, we want to go through that JSON, download some web-friendly versions of the media, and save them.

This Python downloads a high resolution MP4 video and AAC audio - it then mixes them together. It will also download a slightly lower resolution WEBM file. It then grabs a screenshot and any subtitles which are present. Finally, a little HTML5 snippet is written.

from __future__ import unicode_literals
from datetime import datetime

import youtube_dl
import json
import urllib
import os

#    Read the JSON
with open('uploads.json') as data_file:
    data = json.load(data_file)

#    Itterate through the JSON
for video in data:
    videoId     = video["contentDetails"]["videoId"]
    description = video["snippet"]["description"]
    publishedAt = video["snippet"]["publishedAt"]
    title       = video["snippet"]["title"]
    status      = video["status"]["privacyStatus"]

    #    Create a date object based on the video's timestamp
    date_object = datetime.strptime(publishedAt,"%Y-%m-%dT%H:%M:%S.000Z")

    #    Create a filepath and filename
    #    /YYYYMMDD-HHMM/YYYYMMDD-HHMM_My Video_abc123
    filepath    = date_object.strftime('%Y%m%d-%H%M')
    filename    = date_object.strftime('%Y%m%d') + "_" + title + "_" + videoId

    #    YouTube Options
    ydl_opts = {
        #    The highest quality MP4 should have the best resolution.
        #    Best quality AAC audio - because that's the codec which will fit in an MP4
        #    A lower quality WEBM for HTML5 streaming
        'format': 'bestvideo[ext=mp4]+bestaudio[acodec=aac],webm',
        'audioformat' :'aac',
        'merge_output_format' : 'mp4',
        #    Don't create multiple copies of things
        'nooverwrites' : 'true',
        #    Some videos have subtitles
        'writeautomaticsub' : 'true',
        'writethumbnail' : 'true',
        #    Make sure the filenames don't contain weird characters
        'restrictfilenames' : 'true',
        #    Add the correct extention for each filename
        'outtmpl' : filepath + '/' + filename + '.%(ext)s',
        #    If there is a problem, try again
        'retries' : '5',
        # 'verbose' : 'true',
    }

    #    Download the files
    with youtube_dl.YoutubeDL(ydl_opts) as ydl:
        if status == "private":
            #    YouTube-DL won't download private videos
            print "Skipping private video " + videoId
        else:
            print "Downloading " + videoId + " to " + filename
            ydl.download(['http://www.youtube.com/watch?v='+videoId])
            #    Write an HTML5 snippet
            html  = "<video   poster=\""+filename+".jpg\" controls >\n"
            html += "    <source src=\""+filename+".mp4\"    type=\"video/mp4; codecs=mp4a.40.2, avc1.42001E\">\n"
            html += "    <source src=\""+filename+".webm\"   type=\"video/webm; codecs=vorbis, vp8.0\">\n"
            html += "    <track  src=\""+filename+".en.vtt\" kind=\"subtitles\" srclang=\"en\" label=\"English\">\n"
            html += "</video>"
            #    Write the HTML
            with open(os.path.join(filepath, "video.html"), 'w') as html_file:
                html_file.write(html)

The files it downloads are significantly smaller than the original uploads - with no noticeable loss of quality. The combined size of the MP4 and WEBM are around half the size of the original files.

There is one major bug - occasionally the script will crap out with:

youtube_dl.utils.DownloadError: ERROR: content too short (expected 153471376 bytes and served 88079712)

This is a persistent error with YouTube-DL.

The script can be run again - it's smart enough to avoid re-downloading the videos.

And there you have it - a quick way to grab everything you've uploaded. It's missing a few things - view counts and comments, mostly - but it's good enough for re-hosting elsewhere.

This is what it looks like

This is what happens when you put that dash of HTML into a web-page:

I think there's a problem with the subtitle track - but it is not a finalised standard yet.

Google knows me. I've been using Google since long before they were fashionable. I have a Gmail account (in my name), YouTube (also in my name), an Android developer account (name and bank details), Play account (name, credit card, and PayPal), and I've successfully reported security bugs to them. Google, I would suggest, has a pretty good idea of who I am.

Which is why I was somewhat confused to receive this email a few days ago:

My initial thoughts were... ‮!ti kcuf ‭

Who uses Google Plus? I don't! Earlier this year I wrote a mildly popular blog-post about how to make your G+ profile private. Frankly, I think Google+ is a massive mistake and I actively resent the rare occasions when I have to use it. Having my G+ blocked? No great loss.

Until I went to YouTube...

Arse! Because of Google's relentless desire to force the shit-sandwich of G+ down everyone's throats, they made me link my YouTube and G+ pages. What utter ‮.sniatsknaw‭

I use YouTube as part of my professional portfolio. Clients often ask me to share on YouTube a video of what I'm building them. I also monetise some of my videos - Google pays that money straight into my registered business account WHICH HAS THE SAME BLOODY NAME ON IT AS EVERY-SODDING-THING ELSE!

Right. Ok. Deep breath. What magic incantation do I have to do in order to appease the Google gods?

Yeah... I'm that popular that someone wants to make a "fan page" for me. If you are one of my hordes of rabid fans - I'd rather you fed me horse-crap than created a G+ page.

Fine... Here we go!

I've never been quite so glad to live in the industrialised world with enough funds to buy a Government issued ID! I was also feeling pretty smug that I'd had the wherewithal to save a photo of my passport for just such an occasion.

Of course, lots of people have a copy of my passport. Every job interview I've been to has taken a copy. Every hotel I've stayed in wants my details. Countless bars, bank-tellers, travel agents, and HR staff have had their grubby paws on my passport. If someone was pretending to me - how hard would it be to obtain a copy of my passport?

Or, why bother - I'm pretty sure that the monkey-fiddlers at Google wouldn't know a fake passport if it bit them on the meatus. But screw it, let's upload it and wait.

And wait.

And wait.

It was at this point I began to get worried for all my other Google-linked services.

• G+ was down, so I couldn't share my location with my wife.
• No YouTube - which meant grovelling to clients.
• Analytics seemed fine. Oh, yes, Google has validated several of my websites which have my name in the WHOIS. If I'm impersonating myself, it's a long con.
• All my apps were still available in the app store. If someone was impersonating me - they were free to infect Android phones with whatever piss they liked.
• Gmail - thank the maker that they didn't destroy my Gmail! Ok - everyone hates email, but it is kind of a necessary evil.
• Hangouts - luckily I've abandoned Google's pathetic chat offering for something more stable and secure.
• Wallet - weirdly, Google was quite content to let me send and receive money through them. Fraudtastic!
• Who knows what else?! The ADHD-addled toddlers there have so many half-finished ideas lying about, that it's possible I.... OOOH! SOMETHING SHINY! LET'S "SUNSET" THE BORING STUFF!!!!!!

While I waited, I realised that I'd been given a bit of a wake-up call. There's no due-process with multinational corporations. One lives and dies entirely at their whim. The feeling of helplessness when faced with an uncaring monolith is profoundly depressing.

A day later, and G+ had decided that my account was legitimate.

It's not much of an apology - and I particularly like the way that they continue to imply that I did something wrong and should stick to their terms & conditions.

But, sadly, that didn't solve all the problems created by the vomit-munchers in Mountain View.

Terence Eden is on Mastodon

@edent

Replying to @edentWell, my @GooglePlus account has been reinstated but my @YouTube channel is still blocked.
Guess I've been radicalised into #redecentralize!

---

Terence Eden is on Mastodon

@edent

Replying to @edentFan-fucking-tastic! My @GooglePlus was suspended - now *all* my @YouTube videos have been set to private. pic.x.com/v7tm4uc2qm

---

Several hundred YouTube videos set to private. What a gloriously twuntish thing to do. Of course, rolling back a change is a bit too hard for the flange-ticklers at Google. Move Fast And Break Customers' Things!

YouTube does offer a bulk editing tool - but I couldn't use it. Around half my videos were already set to private or unlisted - mostly the ones I share either with clients or close friends. So I had to painstakingly go through every single one to validate what I wanted to do with it - all the while cursing every Google employee.

click cockwombles click cloaca drippings click litter-bugs click melon-farmers click toad-lickers click.

Finally, all was fixed. I was out of Google purgatory.

Oh shove a duck up my arse and serve me for Christmas dinner! What fresh hell is this?

An utterly ancient app - which I probably should have removed years ago - is all of a sudden in violation? Of what? Literally all the app does is play that dramatic "Dah-dah-daaaaaaaah" sound. That's it! It doesn't impersonate anyone. I don't even think it works on modern phones. I'd be jiggered if anyone downloaded it in the last couple of years. What kind of nincompoop would design a system which would mark that as...

Oh, just forget it. I'm done.

I've no way of knowing what has caused these problems. It could be an innocent mistake, an internal or external party with a vendetta, something I've done... But there's no way for me to find out. Google doesn't do customer services - that's part of its business plan. Got a problem? Feel free to kindly piss right off.

That's no way to live. Constantly in fear that one day the most powerful information gatekeeper on the planet can decide you are persona non grata.

It's time to start moving away from Google. I know it won't be easy, but it's now obvious it is the right thing to do. If I'd been away on holiday, or couldn't afford ID, or wasn't acquainted with several Google employees - this could have been an absolute disaster.

It's not going to plain sailing either - running services is hard. But if it goes wrong, I will only have myself to blame - not some faceless giblet-flicker a thousand miles away.

Google have broken my trust - and they aren't getting it back. Wankers.