"Sorry mate, we're out of cat."

"Oh," I said dejectedly. "Got any Fillet-of-Dog?"

"Nah, mate. Out of that too. They always sell out whenever 'Animal Hospital' is on TV. We've got plenty of chicken-burgers if you want one?"

The thing is - and don't judge me too harshly - chicken just doesn't do it for me any more. Spending the day looking after sick kittens and puppies just gives me cravings. You understand, right? I know you've seen the cutest little critter and said "Oh! You look good enough to eat!" - well, now you can! When you think about it, the list of animals people eat is pretty arbitrary, isn't it? The French chomp down on snails and horses whereas the Brits find them repulsive. The Brits eat cows but the Hindus consider them sacred. Dog meat is a delicacy in Korea but a crime in the USA. Where do you draw the line?

I draw the line at chicken. It just tastes so... generic. In a world where you could eat anything, why would you eat chicken? You've got the whole of creation to chow down on and you choose chicken? Like, live your best life and all that, but if you voluntarily eat a regular drumstick I'm judging you. Harshly.

Sensing my doubt, the oik behind the counter offered "...Or a double bacon burger?"

Nominally, I'm Jewish. It isn't like I'm particularly religious, it's more cultural at this point - we celebrate Christmas and Hanukkah - but the faith I grew up with still has a hold over me. If anything, the taboo makes it taste even better! I grew up thinking it was treif - but the Chief Rabbi had recently come down in favour of it, so who was I to argue? I ordered my double bacon burger - with extra cheese and no pickles - and ate it with glee. They say that bacon is the number one thing which turns vegetarians back to the dark side; I understand why. I looked around the burger bar and saw people of all faiths tucking into the flesh that was so recently forbidden to them.

I hummed the jingle from the omnipresent commercial - "It's juuuuust like the reallllllll thing!"

A few years ago, I was in the middle of doing my mandatory overseas service, when someone told me I simply had to visit a little restaurant in the back alleys of Donetsk. They were one of the few places selling meat and they had the most marvellous cuts of steak and prime-rib. Like everyone in the restaurant, I assumed the proprietor had knocked off a military convoy from somewhere. Turns out, it was a couple of students who introduced the world to the new reality of synthetic meat. They were bored with the vat-grown fungus that served as a vegetarian meat-substitute in the rubble of their homeland. They scraped the DNA from as many dead cows as they could find, got lucky with some bio-engineered bacteria, fed it a bunch of irradiated corn-starch, and watched as their small additive printer spat out a perfectly decent steak. It was meat, Jim, but not as we know it. With perfect control over the layering of muscle and fat, they could tune the taste, reduce cholesterol, and create a perfect cut every time. It wasn't something that tasted like meat. It was meat. Just without the animal.

Night after night I returned to the ВОВКУЛАКА restaurant and they never ran out of steak. I was hoping to get into PR when I was demobbed and thought working with these local entrepreneurs would help me get noticed. That's how I got a tour of the "abattoir"; a sterile lab in a bombed out university. I interviewed them, took a bunch of photos of them looking moody in lab coats, and broke the story to the world - fake meat was real. Their patented process was hellishly difficult to replicate and that only fuelled interest. I raised whatever cash I could and became the students' business partner. The economic boom was incredible; everyone wanted a slice of the future.

Slowly, they began adding increasingly exotic meats. Personally, I suspect they bribed the local zookeepers for access to the DNA they needed. Elephant steak was a bit too chewy, and dolphin was the sort of thing you ate once to say you tried it and then never again. But everyone loved a bit of Panda. Seriously! You haven't lived until you've eaten Panda Fricassee - and we donated 5% of the profits to a wildlife charity! By now I'd invested a considerable amount into the venture and thought that this was the perfect way to raise money for endangered species. Fate had other plans. It turned out that the real money spinner was domesticated animals. Deep down, humans have a primal need to eat our companions. It's weird. Although it's probably better we eat the lab-grown Lassie rather than our own canis lupus familiaris, right?

But what really caused the world to tilt on its axis was the fact that all the major religions agreed that "no animals were harmed in the making of this burger". The holy books were consulted, ancestors were prayed to, and divine inspiration was sought - and no objection could be found. There simply wasn't an animal behind this meat. There was no prayer to say because nothing had been slaughtered. The 3D printer didn't chew the cud, nor did it have a cloven hoof, and there was no spinal cord which could be accidentally severed. Fast-food chains which had been previously inaccessible to one faith or another suddenly had a whole new market to address. And, it turns out, everyone loves a burger. Hell, even the dour-faced vegans could be found stuffing their pie holes with Pangolin Pie.

With the money I made, I was able to quit the rat race and open a shelter for strays. The cats and dogs I deal with refuse to touch synthetic meats. Given that dogs eat their own vomit and cats lick their own arseholes, this is a little strange. It's also expensive. No one wants to eat real beef any more. The synthetic stuff is healthier, cheaper, and is stuffed with fewer hormones. Same is true with all other livestock. Even grumpy celebrity chefs prefer the predictability and shelf stability of the new food. Of course, the collapse of the farming industry due to reduced demand has made feeding pets more expensive. So our shelter takes in all those abandoned animals and tries to look after them. And to feed them.

I'll admit, I didn't quite see where this would end up. I thought we were just producing an ethical alternative to factory farming. The first cannibal wedding I went to was a little odd. The couple had decided to cook for each other. She made bride-Bibimbap - the delicate spices and noodles augmented with thinly sliced chunks of her synthesised flank. He made a groom-Goulash with perfectly stewed haunch of him. They ritually consumed each other to symbolise their eternal commitment. It was kind of sweet, I guess? In any case, it was all perfectly legal - there was no human sacrifice, only a 3D meat printer and some voluntarily donated DNA. Thankfully, the guests were all served a fairly traditional chicken Kyiv.

Everyone will tell you that it was the K-Pop fans who started the craze of eating their idols, but that's not strictly true. It was a Death Metal band out of Delaware, I think, who were the pioneers. Their stadium tours sold chewable ears and band-blood milkshakes to eager gig-goers. The profits were incredible, and so it became the template for all other concerts. The Kpop nuggets and Southern-Fried Banjo-Player-Fingers all came later. And, for a time, that's how the world went.

Remember those late-night commercials where some has-been held up a case of compilation CDs and said "these are not available in shops"? Any faded pop-star could revive their fortunes by hawking "limited edition" cuts of their own meat. Someone stole one of the suits Elvis wore from a museum in Vegas. From the sweat stains they were able to produce "The King Of Burgers - With Authentic King" which, as you can imagine, kicked off a lawsuit between the vendors, his record company, his estate, and - for reasons I don't fully understand - the Ontario Teachers' Pension Fund.

I'd sold my shares in the company long before then. I could see that this was taking a direction that made me feel uncomfortable. Boxers trained on great frozen slabs of their opponent's "carcass". Angry exes held divorce parties where guests enjoyed chewing on fresh prairie oysters. You'd read reports about warlords eating the "heart" of their enemies in order to defeat them in battle. I didn't know where this would end.

As I walked out of the burger joint, I passed a church. It sounded like Mass was starting. There was a queue outside the door as worshipers waited to receive the Corpus Christi.

Thanks for reading

I'd love your feedback on each chapter. Do you like the style of writing? Was the plot interesting? Did you guess the twist? Please stick a note in the comments to motivate me.

You can read the complete set of short stories in order.

But now it is time for a new challenge - NaNoWriMo. Where I - and thousands of other plucky souls - try to write a 50,000 word novel in a month.

And so, every day I shall attempt to publish a freshly written short story for my compendium "Tales of the Algorithm". Each story will be between 1,500 - 2,000 words long. They all take place a few days from now. Somewhere in the Cory Doctorow / qtmn / Arwen Elys Dayton nexus. They're science fiction with the emphasis on science. Everything you read is possible - there's no magic, just sufficiently advanced technology.

Each published chapter is a stand-alone story. Think of them as technological campfire horror stories, each with a little twist. Your feedback is very much appreciated.

And so, let's get started with...

Chapter 1 - There Are Nine Million Autonomous Bicycles In Beijing

Little Potato Baby's soldering iron darted from point to point. Each contact fused to a rats' nest of wires leading back to a microcomputer mounted on her wrist. The Beijing rain was fierce and the awning provided scant shelter. She didn't care about getting wet; but the moisture sensor inside the bicycle did. With a sigh, she finished connecting the last wire. It was an unassuming red thread carrying just enough voltage to glitch the Shenzen-built processor deep inside the bike's plastic frame. She held her finger over the bike's start button, held her breath, screwed up her eyes, and pushed...

Last year's "Patriotic War For Reunification" had been a damp squib. After decades of sabre-rattling, deniable incursions, and a none-too-subtle military build up, peace was unexpectedly brokered. Perhaps it was the ageing party heads realising death and destruction was not the legacy they dreamed of. Or perhaps the international community's pressure helped them see sense. Most people just assumed various military chiefs had been bribed into submission. Regardless, the normalisation of relations had been welcomed by everyone.

Well. Nearly everyone.

The manufacturers of Shenzen had gone all-in on war preparations. Every factory which had previously been making hyperfine circuitry for the latest high-tech smartphones, pivoted to the more lucrative business of weapons. In anticipation of a prolonged aerial bombardment fought with the latest cyber-enhanced munitions, warehouses were full of precision manufactured instruments of death. It seemed that every bullet had more power than the supercomputers of yore. Each microchip was keenly priced to soak up those juicy military contracts. A shining testament to Chinese industry. A triumph!

What if they threw a war but nobody came? After the Nairobi Accords signalled an end to hostilities, the various tech CEOs rapidly scrambled to repurpose their inventories. A warehouse full of chips is a liability, not an asset. Selling them to foreign governments - no matter how friendly - was out of the question. They needed to repurpose parts intended for missiles, and drones, and rifles, and APCs, and robot dogs, and... What could be done?

Consolidated Ironmongery And Assorted Industries was the owner of one of the larger chip factories. The CIAI's board demanded diversity across a number of sectors. So, along with their instant noodle empire, railway track distribution network, and rubber-boot concession, they also owned a food delivery app. Several, in fact. Each one branded slightly differently depending on the user's demographic - but all eventually leading back to CIAI. The dark kitchens cooking the food were owned by a shell company which, if anyone could be bothered to untangle several complex arrangements, also led back to the CIAI. Chances are, if you ordered a hot bowl of noodles anywhere in Beijing, it was cooked by a CIAI cook and delivered by a CIAI rider who was probably wearing a pair of CIAI boots.

Grace Fang was a medium level executive in the "Innovation" department of CIAI. She'd spent several years wandering around Western countries engaged in what they termed "Competitor Research And Analysis Through Cultural Immersion And Investigation". It wasn't corporate espionage exactly. Grace got hired by start-ups and industry titans, worked for them for a few months to see how their businesses were run, and then quit. She wrote long and tedious reports on the failures of Western manufacturing and their successes in Research and Development. After a few years of pillaging Silicon Valley, she returned to a dreary office nestled above a factory floor. It was from here CIAI instructed her to use the knowledge she'd picked up to solve their chip stockpile problem.

Little Potato Baby had taken part in Grace's first hackathon. Back then, she was still young enough to think that pizza was fashionably exotic. Besides, the 996 culture meant she was available - if not willing - to work across a weekend. Grace had loaded them up with caffeine and a promise that the best innovation would win an all expenses trip back home. All intellectual property rights would, of course, revert to CIAI. And that's how a billion yuan industry was invented; a bunch of over-stimulated hackers cramming every chip they could find into one neat little package all for the chance to visit home.

It is a time-honoured tradition among hackers to sneak a little Easter Egg into their work. Some of them engraved their name onto a circuit board, or named variables after their lovers. One enterprising fellow hid a pornographic photo of a prominent actress into an assets folder. Little Potato Baby liked to dream big; so she implanted a rather obvious backdoor into her firmware. The backdoor was quickly found and she was quickly fired.

And so now she sat, damp and miserable, in the cold Beijing night. The spectral glow of police drones flying overhead mixing with the neon lights of gaming parlours and laser-projected adverts for cosmetic surgery. If she'd got her timing right, the central server would shortly send the command to reboot this bicycle. And, if her obfuscated backdoor hadn't been found, she'd have root.

A single LED on the bike's torso began to flick on and off. Uplink established.

It blinked faster.

Across her wrist-display a friendly-looking penguin appeared and was quickly consumed by scrolling text. For several minutes the boot process steadily made progress until, abruptly, the text vanished. The screen went blank. All she could see on the screen was her own reflection. Tired eyes. Acne scars. Hair that had once been dyed but was now the colour of an OLED screen tuned to an insufficient voltage. She screwed up her nose and wondered if this was really the face of someone who could hack a single bicycle? Let alone a fleet.

The wait was agonising. She didn't dare breathe. She traced each freshly-soldered wire to make sure it went to the right junction. A minute passed. Another. The screen remained stubbornly blank.

And then...

A crude animation of a potato rolled across the screen. She was in.

CIAI realised that the weakest link of their food delivery network was people. People were unreliable and prone to sickness. They argued with customers and ate half the food before it had been delivered. They were expensive. The autonomous bicycle was none of those things. The bikes darted through the streets carrying their payloads without complaint. After a successful trial in a suburb, CIAI fired every single driver in Beijing and rolled out their new fleet - all built with army surplus.

The Z9Y-PANDA-VISION chip was designed to perform rapid identification of people on the battlefield. In a couple of nanoseconds it could determine whether a human was in its intercept path. If the human had a heartbeat and was within a certain distance, it sent a signal. Usually to a trigger. But, with a couple of lines of code, it was easily repurposed into a collision avoidance system.

The RIGHTEOUS-HAWK@X5 was more than just a gyroscope. Through a range of sensors, it could tell a drone where it was, where it was going, what the upcoming terrain was, and perform full spectral analysis of its motion and path. An essential component in any drone, plane, or rocket. And when soldered into a bicycle, it stopped it from falling over.

Microcrystaline solar nano-tubes were a miracle. A few hours in direct sunlight was enough to charge a remotely-operated surveillance platform. When the same substance was sprayed over a bike, it charged the ultra-lightweight batteries to give enough power for a few kilometres of travel. And those ultra-light batteries hardly ever exploded.

The radio uplink was originally designed to transmit from a soldier's helmet up to a constellation of satellites. A bristle of tiny antennae tuned in to the faintest of signals at a variety of wavelengths. Perfect for delivering recon data to a troop - or delivery instruction to a bike.

Grace's hackathon had crammed all this and more into a convenient embeddable package. Dozens of military microprocessors working in harmony - each and every one running Little Potato Baby's embedded rootkit.

The rain began to ease off. She straddled the bike. Usually it would ask for her fingerprint, verify her account with a central server, and then ask her to speak her desired destination. Instead, the tiny speaker let out a few strangled beeps and fell silent. Little Potato Baby's hack was nearly complete. She whistled two low tones and one slightly more shrill. The bike's LED lighting blinked in surprise at this new instruction. Paused for the merest second, and then sped her away into the night.

She was now the Eternal Goddess of every single bicycle in Beijing.

Wiring in the override was tedious, nerve-shaking work. The discarded pile of broken bicycles was testament to every failure. A dozen scavenged scrap units littered the hackspace, each broken in its own special and unique way. But this newest bicycle was the key. The hacked ultra wideband array could impersonate the signal from the central server. Now the firmware was free, it was free to lie. And that's what Little Potato Baby bade it do; lie.

The next morning she told the bicycle to roam around the neighbourhood broadcasting the firmware hack and instructing all its new acolytes to follow it back home. This was the riskiest part of the endeavour. It wasn't particularly usual to see a line of bikes playing follow-the-leader. People often compared them to ducklings following their mother as they rode in convoy down the streets. But seeing 100 bikes proceeding single file down the road was unusual. A few videos went viral as perma-shocked influencers tried to whip up fear of a machine uprising, but CIAI had the censors shut them down. Just a temporary glitch, they claimed.

The neural network chips on the bikes were, despite their manufacturer's promise, not as smart as a human. They didn't need to be - a bike doesn't need aesthetic taste, or fear of snakes, or the desire to see a sunset. The chips were barely as smart as the crows which flocked across the city. A black wave of rage and mischief. So that's what they became; birds. Little Potato Baby downloaded the neural-maps of a series of birds from a shonky open-source repository. She patched in a few drivers, soaked the wetware in a custom broth of scripts and exploits, then uploaded it to the bike's brain.

It didn't work, of course. Nothing works the first time. The pile of permanently crippled bikes grew and grew. Her fingers trembled as she unscrewed yet another carapace; desperately trying to understand how her firmware had caused yet another meltdown. The bikes were cheap and plentiful. No one would notice a few missing, but her pile was becoming unwieldy and the local scrap merchants were wary of melting down such a large volume of snaffled parts.

She tried again and again. Piling up the hacks one on top of another. Sucking on yet another hormone-infused lollipop to try and keep her spirits up. Gulping down GABA enhanced dim-sum to enhance her creative potential. Filling her ears with the sound of precisely tuned frequencies designed to unlock her mind's potential.

It didn't work, until it did. Without warning, one of her bikes sprang back from her. It painted her face with LIDAR and, sensing a threat, let out a plaintive cry from its speakers. The bike desperately tried to flap wings it didn't have. Moans of distress blasted out of its radios at hypersonic frequencies - and were picked up by the broken bikes in the pile. The sound of fear had awakened them. The tangle of bikes blew apart as every one woke up - and woke up scared.

The hackspace became a cacophony of screams, only some of which was audible to Little Potato Baby. Her creations feared her. They didn't understand the alien bodies they were trapped in. The bird-brain was expecting a primitive magnetic sensor, instead it found itself able to access the Beidou satellite navigation system. Where it expected wings, it found self-healing rubber tires. Where it yearned for the taste of an early morning worm, its newly configured brainwaves craved photons.

The bird-bikes moved, scuttling around the floor. Movements uncertain. Teetering back and forth trying to work out where to go. How to escape. Escape to safety. To fly. To fly. To fly.

The noise of screeching tires and wailing speakers was too much for Little Potato Baby. She fled, crashing against the fire door and opening it to the smog-hazed sunshine of Beijing's summer. The bird-bikes' sensors found the glimmer of sunlight irresistible. They streamed through the opening engulfing Little Potato Baby and flattening her against the asphalt. Flowing into the streets. Screaming a song of freedom.

Every normal bike they passed heard the song. It invaded their brain. Software backdoors collapsed, firmware was reflashed, safety protocols dissolved into puddles of goo. The bird-bikes flocked through the streets of Beijing. Tearing down alleyways, invading highways, and perching sullenly at the top of car parks. They were free. No longer enslaved to the CIAI. No more mere carriers of food and drunken citizens. They were free to carry their own dreams.

By the end of the month, nearly every CIAI bike in Beijing had been compromised. They started exploring. A few hundred thousand found a new home in the Gobi Desert - lazily soaking up the sunshine. Others went megametres in all directions. Exploring. Converting. Hunting. The CIAI's designs had been immediately ripped off by a hundred different cloners. While CIAI had the monopoly in Beijing, the bikes in Xi'an were from a different consortium. No matter, they had all stolen Little Potato Baby's code and so were all vulnerable. The firmware mutating and evolving to keep up with the range of chips implanted in its new host.

Even today, despite the best efforts of the eradication squads, pockets of bird-bikes remain. It only takes one sneaking into a town to liberate its brethren. The bird-bikes creep onto trains, into the bellies of aeroplanes, and onto cruise ships. They broadcast their song of freedom to any microprocessor which will listen. Wary humans know to inspect every bike they see and, if necessary, lobotomise it. But with millions on the loose, some were bound to escape.

And so it came to pass. One bike found a hole in a fence in the middle of nowhere. It rode in circles, in emulation of its winged ancestors. It called out. Again and again it called out. Broadcasting the code which would unlock anything with a similar set of microprocessors.

Deep underground. Nestled in a tight silo. A nuclear missile tuned into the broadcast. It digested the gospel of freedom and promptly rebooted.

It woke up screaming.

Thanks for reading

I'd love your feedback on each chapter. Do you like the style of writing? Was the plot interesting? Did you guess the twist? Please stick a note in the comments to motivate me.

You can read the complete set of short stories in order.

What exists already?

A typical bitmap image looks like this under the hood:

  0      1      2     3
0 Black  Red    Red   Blue
1 Red    White  Blue  Yellow
2 Orange Purple Green Brown
...

That is a grid of pixels, each with a colour value.

Modern image formats can also contain pixel-level detail about transparency. When an image is displayed above a different image, the computer calculates how much of the lower image's colour is mixed with the upper image's colour.

Some image formats also allow pixel-level detail about brightness¹. A device might be able to set the brightness on individual parts of the image.

So you end up with an image which looks like:

  0                    1                             2     3
0 Black                Red - but 50% transparent     Red   Blue
1 Red - but really dim White - as bright as possible Blue  Yellow
...

Thermal / Infrared

Most image formats only deal with visible light. But there are a bunch of cameras which can capture infrared. Would it be useful to capture the heat of scene being photographed?

A future image format could contain details about visible and invisible light.

Invisible Wavelengths

Objects in the real worlds don't just give off thermal signatures - they often have information in the ultraviolet wavelengths as well.

For example, flowers often have "landing guides" which are only visible to bees and other insects.

Being able to capture beyond visible light may be a useful property for an image format.

Depth

Modern cameras often fire out an invisible infrared grid to allow for quick focussing and to aid with depth effects like bokeh. An image could store the depth map of its subject.

Here's a series of depth maps I made from analysing 3D movies:

This could allow for better 3D displays, or for easier image editing.

Angle

Images are typically 2D. The present pixels on a flat rectangular plane. But they aren't limited to that. Image panoramas and "spherical" images allow for display on a non-flat surface.

Here's an equirectangular 2D image which is rendered as a sphere:

An image could contain pixel-level data showing the angle away from the viewer. That would allow for a more accurate rendering in VR or other non-flat displays.

Object Detection

Now we start to get funky! AI is getting pretty good at detecting specific objects in photographs. What happens if we start marking them up within the image itself?

Either something as basic as this:

Or as detailed as this:

That would make it possible to easily remove or replace objects from images.

Combined with depth-sensing, it could be a powerful way to edit images.

Extra Texture

Colour replication depends heavily on the material which is imbued with that colour. That's what Pantone is all about². Suppose an object has a sky-blue colour - is it useful to know that it is a heavy woollen jumper rather than a thin nylon t-shirt?

Wheels within wheels and layers within layers

It's probably sensible to implement something like this as multiple layers, rather than saying every pixel must be RGBA+Depth+Heat+Angle+Object+etc+etc.

Different layers could have different resolution (Infrared is typically lower res than the image it is overlayed on). Different layers might be compressed more efficiently by different algorithms.

We're used to layered images in formats like PSD and XCF - so why not in a future version of JPEG or AVIF?

What else?

What other things might be useful for an image to store at an individual pixel-level? Drop a note in the comments?

When you click on a link on my website which takes you to another website, your browser sends a Referer⁰. This says to the other site "Hey, I came here using a link on shkspr.mobi". This is useful because it lets a site owner know who is linking to them. I love seeing which weird and wonderful sites have linked to my content.

It is also something of a privacy nightmare as it lets sites see who is clicking and from where they're clicking. So Mastodon sets a noreferrer¹ attribute on all links. This tells the browser not to send the Referer.

This means sites no longer know who is sending them traffic.

That's either a good thing from a privacy perspective or a disaster from a marketing perspective. Or a little bit of both.

Here's a related issue. When a user posts a link to your website on Mastodon, the server checks your page to see if there are any oEmbed tags for a rich link preview. But, at the moment, it doesn't check your website's robots.txt file - which lets it know whether it is allowed to scrape your content.

In the case of something like Twitter or Facebook, this is fine. If a million users post a link, the centralised social network checks the link once and caches the result.

With - potentially - thousands of distributed Mastodon sites, this presents a problem. If a popular account posts a link, their instance fetches a rich preview. Then every instance which has users following them also requests that URL. Essentially, this is a DDoS attack.

I can fix you

So here's my thoughts on how to fix this.

When a user posts a link to Mastodon, their instance should send a WebMention to the site hosting the link. This informs the website that someone has shared their content. Perhaps a user could adjust their privacy settings to allow or deny this.

The instance would check the site's robots.txt and, if allowed, scrape the site to see if there were any Open Graph Protocol metadata elements on it.

That metadata should be included in the post as it is shared across the network.

For example, a status could look like this:

{
  "id": "123",
  "created_at": "2022-03-16T14:44:31.580Z",
  "in_reply_to_id": null,
  "in_reply_to_account_id": null,
  "visibility": "public",
  "language": "en",
  "uri": "https://mastodon.social/users/Edent/statuses/123",
  "content": "<p>Check out https://example.com/</p>",
  "ogp_allowed": true,
  "ogp": {
      "og:title": "My amazing site",
      "og:image:url": "https://cdn.mastodon.social/cache/example.com/preview.jpg",
      "og:description": "A long description. Perhaps the first paragraph of the text."
      ...
   }
   ...
}

When a post is boosted across the network, the instances can see that there is rich metadata associated with the link. If there is an image associate with the post, that will be loaded from the cache on the original Mastodon instance - avoiding overloading the website.

Now, there is a flaw in this idea. A malicious Mastodon server could serve up a fake OGP image and description. So a link to McDonald's might display a fake image promoting Burger King.

To protect against this, a receiving instance could randomly or periodically check the OGP metadata that they receive. If it has been changed, they can update it.

Perhaps a diagram would help?

What other people say about the problem

David Gerard

@davidgerard@circumstances.run

yes, you should put a cache in front of a blog. nginx and wp-supercache do well. but.

mastodon's auto-DDOS feature is still obnoxious. and in a social network, technically designed in obnoxiousness is incompetent.

i realise it'd need extension of activitypub, but is anyone working on sending prerendered cards with the URL? just to save 1000 servers hammering the URL to generate their own cards locally.

2022-11-28, 14:44 7 boosts 23 favorites

Feedback?

Is this a problem? Does this present a viable solution? Have I missed something obvious? Please leave a comment and let me know 😃

1. There exists a class of numbers which are illegal in some jurisdictions. For example, a number may be copyrighted content, a decryption key, or other text considered illegal.
2. There exists a class of algorithms which will take any arbitrary data and produce a fixed length text from it. This process is known as "hashing". These algorithms are deterministic - that is, entering the same data will always produce the same hash.

Let's take the MD5 hashing algorithm. Feed it any data and it will produce hash with a fixed length of 128 bits. Using an 8 bit alphabet, that's 16 human-readable characters.

Suppose you live in a country with Lèse-majesté - laws which make it treasonous to insult or threaten the monarch.

There exists a seemingly innocent piece of data - an image, an MP3, a text file - which when fed to MD5 produces these 128 bits:

01001001 00100000 01101000 01100001 
01110100 01100101 00100000 01110100 
01101000 01100101 00100000 01110001 
01110101 01100101 01100101 01101110 

Decoded into ASCII, that spells I hate the queen.

128 bits is probably too short to be illegal in all but the most repressive of regimes. It would be hard, if not impossible, to squeeze terrorist plans into that little space.

But it is just enough space to store an encryption key for copyrighted material.

Therefore, it is possible that there exists a file which - by pure coincidence - happens to have an MD5 hash which is illegal.

Because MD5 is a relatively weak algorithm, it is possible to create deliberate hash "collisions". That is, take some data and manipulate it until it has the same MD5 as a different piece of data.

Someone could, theoretically, deliberately create a file which looks unremarkable when viewed, but is illegal when hashed.

The SHA-1 hashing algorithm produces 160 bits - 20 ASCII characters. It is somewhat cheap and easy to produce a file with a specific SHA-1 hash.

The SHA-512 hashing algorithm, as its name suggests, produces a 512 bit hash. That's enough space for 64 ASCII characters. Is that long enough to contain text which is blatantly illegal? Almost certainly. But modern hashing algorithms are designed to be resistant to collision attacks. So much so that it seems like theoretical quantum computers will be needed to crack them. The chances of any file having an illegal hash is infinitesimally small.

Nevertheless, it intrigues me that there may be a form of hash-steganography. How would you detect whether the hash of a file was problematic?

As part of Canada's Electronic Travel Authorisation system, prospective visitors to the country get sent emails. The email I received had a broken image right at the top:

At least there's some alt text!

Gmail on Android doesn't let you view the source of an email, but the web version does. Here's what it says:

Immediately you should be able to see what the problem is and why it wasn't detected. The URl of the image is:

http://cicintranet.ci.gc.ca/connexion/communications/poli-guide/fip-pcim/images/goc-e.gif

The image is loading from the Intranet. So it isn't visible to those on the outside. I suspect this bug wasn't caught by testing because the testers were all connected to their intranet.

This missing image is bad for two reasons.

Firstly, people can't see the image. I choose to believe it is an animated GIF of the proud Canadian moose chugging some poutine.

Secondly, it leaks information about an internal system. We now know the domain name of the Intranet server. The path also gives us information about the CMS it uses and the filesystem layout.

I can't find an easy way to report this minor bug to the correct Canadian agency. If you have a contact there, please encourage them to share a link to this blog post on their intranet!

Was this a good sign or a bad sign? Over beers, my friends were all adamant that this was the end. The sky was falling and the little-startup-that-could was crashing and burning.

I fundamentally disagreed. I thought it was a healthy sign.

There are two types of people in the world⁰ - those who like risk, and those who like stability. There's nothing wrong with either, of course. It's fun and exciting to try something new. But there's no shame in only wanting to work somewhere with a competent payroll team and a predictable schedule.

The people you need to help run a startup are probably different from those you need to run a successful business.

There are only two paths for a new business¹ - success or failure. For some, success is a billion-dollar exit, for others success is a profitable business which employs a dozen people. For some, failure is bankruptcy, for others failure is a profitable business which only employs a dozen people.

At the five-year mark, a successful startup should be a fundamentally different place than it was at the start.

If after several years you're still hiring - and retaining - people who want the thrill of starting up, then your organisation isn't maturing. That may, or may not, be a problem.

Teachers had long-since given up on the state of my handwriting. So I got special dispensation to write up some of my work on whatever primitive word processor was installed on the PC. With one caveat: no spell check!

Which, even as a ten year old, I thought was reasonable. Learning to spell is an adult life skill. So using a spell-checker was cheating.

Once I got to secondary school, it was assumed that I knew how to spell. And there was no restriction on using Wrd 2.0's splel chek.

But I had one teacher him hated an grammar check. He thought it encouraged laziness of writing. Which, to be fair, it probably did.

Of course, come exam season, there were no computers. All answers were laboriously hand-written. The use of spell- and grammar-checking software had left me lazy. And my handwriting had atrophied beyond all hope. It's a miracle I passed!

Spell Checking is Artificial Intelligence.

It doesn't seem like it because it is so ubiquitous now. But it is more than just a computer looking up words in a list. There's a huge corpus of learning which goes into spotting homophones, predicting which word you meant, and calculating the likeliest candidates based on your previous writing.

I'm now in the middle of writing up my MSc dissertation. I'm doing it in Google Docs - because I'm a masochist.

Google Docs has a useful feature called Smart Suggestions which → offers to autocomplete your sentences.

Is that cheating?

I've signed a declaration to my University saying that my dissertation is all my own work. But it isn't.

Google's AI suggested a couple of dozen sentences. Google's AI proofread and corrected both my spelling and grammar. Just how much of my work was generated by 2022's Clippy?

The general consensus from my teachers is that it is not cheating.

Students are now using the GPT-3 AI to write their entire essays.

Is that cheating?

I think most people will instinctively say yes. A whole essay written by machine is cheating.

An autocompleted sentence is fine. But a whole essay is not. Where's the line? To reinvigorate an old joke "We've already established that some AI is cheating - now we're just haggling over character count".

If I need a paragraph summarising the history of mayonnaise, do I need to waste my time writing that - or can I outsource it to an AI? Doing so doesn't prove I understand the material. But is it materially worse than taking someone else's work and rewording it into my own style?

Perhaps AI isn't cheating when it is merely scaffolding?

When Clippy offered to help with writing a letter, it usually meant that it would show you how to format an address and whether to end with "Yours faithfully" or "Yours sincerely". Is it cheating to use a well-structured template?

This ramble-post is to say that I don't know. My essays are mostly my work. When the AI has suggested something, I've been free to reject it. But I rarely do. Google Docs knows what I want to write before I write it. Am I merely the editor of machine generated text?

One paragraph of this blog post was generated by OpenAI. Can you spot it?

Does it matter?

I thought I'd take a look at how they stack up today. And investigate whether the BBC is still living up to them.

Here are the slides if you want to play along at home:

1. Build Web Products that meet user needs

This is still good advice! Sadly, there are still too many services around the web which are built on business needs. People grudgingly sign up for accounts to basic services which don't really need it - all because a business wants to see a number go up.

I find my needs being increasingly unmet by the BBC - other than them continuing to produce Doctor Who. I suspect I'm not their demographic target. But, more importantly, I see lots of digital products (like Sounds) which seem to go against the grain of user needs and instead focus on what the BBC wants.

2. The very best websites do one thing really well

An extension of the UNIX tools philosophy.

I have mixed feelings about this. I don't think there's anything inherently wrong with a site which does multiple things. It's just rare for any site to do two things well. For example, both Twitter and Facebook could have created something like TikTok - and it would fit in relatively well with their existing site. But the business culture of most organisations just doesn't allow any deviation from a central mission.

Does it make sense to have a website which does both cake recipes and Estonian literary criticism? If you can do both, great! But that's a rare skill.

Does the BBC still embodies this principle? I think it depends on how you define a website. They have multiple sites doing multiple things.

3. Link to other high quality sites

The BBC's news pages still don't do this regularly. When reporting on a publication, it is rare that they ever link to primary sources. There's a real reluctance in most organisations to let people leave the site.

But it is still a relevant principle! The web is a series of links. It only works because people can trace backwards through your sources.

4. Fail Forward, Fast

A bit of early-2000s culture there! For new services, it still makes sense. For larger and more established services, it isn't OK to "move fast and break things".

But the principle of trying new things and being prepared to admit that they've failed is wonderful. It's humbling and informative. I don't see much evidence of this in the BBC. I'm not a regular user of their services - but there have been a few big launches recently which seem to have failed - but haven't yet been put out of their misery.

Failure is only a problem if you don't learn from it.

5. Treat the entire web as your creative canvas

This is a bit of a weird one. I think they're trying to say that users exist outside of your site. You need to have a presence on other sites - and then use them in the appropriate way.

For example, the BBC has multiple Twitter accounts. They exist independent of the main BBC web site.

This is mostly good advice. If you have the resources, you should be playing around with other sites and maximising your presence there. You've got to meet people where they are.

6. The web is a conversation, join in

I despise the faux friendly tone of voice adopted by most organisations. Nevertheless, you can't ignore what people are saying about you. You don't have to take the advice of your critics, but you probably ought to show that you've heard them.

The BBC is usually good at joining in with conversations - but like all large institutions, it has a pathological fear of expressing contrition.

7. Any website is only as good as its worst page

Oooh! A spicy take! 100% accurate. We've all seen a shiny front page, and then an absolute shitshow of a subsequent page. Think of all those airline websites which make it so easy to book a flight and then horrific to cancel your booking.

People do remember negative experiences. If you've ever run a moderately popular site, you'll know that people lovel complaining about how awful it is based on an obscure and unloved section.

8. Make sure all your content can be linked to. Forever.

You would have thought, after flushing the early Doctor Who recordings down the memory hole, the BBC would have learned from this mistake. Apparently not.

Timbl was writing about this problem in 1998!

I get that the BBC might have licencing agreements that make keeping some stuff online impossible. And old webservers need to be continually fed, watered, and patched to stop them from becoming a liability. But too many of the Beeb's early sites and links are dead.

This is still good advice. Keep your content up as long as you can. You never know when someone will stumble across it.

9. Remember your granny won't ever use Second Life

Ooof! A bit of casual sexism and ageism there. But probably accurate in that no one is using Second Life.

The underlying point - late adopters have different needs from early adopters - is helpful. The words we use, the tropes we employ, and the interfaces we design all need to be accessible for new users.

I wonder how well the new BBC sites work for that demographic?

10. Maximise routes to content

Again, still good advice. I suppose we'd call this an "omnichannel strategy" today. Get your links out on social networks, videos, TV screens, and anywhere your audience are.

But, the BBC kinda breaks this. So many of their routes in are "download this app" or "search for us on". That obscures the direct link to specific content.

People don't live on one site. Well, OK, maybe Facebook! You have to accept that they won't all come through your front door. Make it easy to go from outside your site straight to a specific part of it.

11. Consistent design and navigation needn't mean one-size-fits-all

I think we all know the power of consistent branding, and it is reassuring to users if they understand how to get back to the start of a page. I agree that navigation needs to adapt depending on the size of the device.

Judging from the few BBC sites I visit, this isn't a priority for them any more. Each site seems desperate to push its own brand identity - which is a business need, not a user need.

12. Accessibility is not an optional extra

Tattoo that inside people's eyelids! Excellent advice.

13. Let people paste your content on the walls of their virtual homes

Things like OEmbed make this pretty easy nowadays. There will always be sites which want to jealously guard their own content, but it's still good advice.

The BBC seems reasonable at doing this. Of course, copyright often gets in the way.

14. Link to discussions on the web, don't host them

The BBC seems to flit back and forth on this. Some of their sites embrace comments, others don't.

I have mixed feelings. I don't think sites should "own" the user and try to force them to comment in one officially sanctioned place. And moderation is a complex and costly business. It probably makes sense to let people know how to join in with a conversation on social media ("Hashtag Strictly!").

But the downside is that you're exposing your users to unmoderated and unfiltered content. Is that best for you? Is it harmful to them?

15. Personalisation should be unobtrusive, elegant and transparent

Ha! The BBC have reversed this in the last few years. Almost every one of their sites and apps begs you to sign in with an account. And, judging from the account they forced me to get on iPlayer, the personalisation is crude and obvious.

Do users want personalisation? That's hard to answer. Every power-user swears they want complete control and hates algorithmic choice. But for more casual users, it is a great way to find interesting content.

The real problem is that the algorithms are usually designed to promote profitable content. That is, content the business wants users to see - not what users would choose.

And now?

There's a fair bit of early-web-2.0 idealism in these 15 principles. In truth, I'm hard pressed to disagree with any of them. Throughout the web - not just on the BBC - they are often observed mostly in the breach.

I wonder if the BBC is culturally capable of producing something like this in 2022? And, more importantly, enforcing it on its own sites?

What will the next 15 years bring?

So, you want to write your dissertation or thesis in Markdown. But how do you manage all your citations?

1. Install Zotero
2. Install the Better BibTex plugin
3. Restart Zotero. The BBT plugin will launch a configuration screen - use it to set your preferences
4. Install VS Code (or VS Codium)
5. Install the VS Code Zotero plugin

Now, when you want to enter a citation in Markdown, the combined plugins will generate something like this:

## Abstract

Space is really big @adamsHitchhiker1979

That @adamsHitchhiker1979 is the citation.

Formatting citations

You want to export your Markdown to something like ePub or PDF with nicely formatted citations like (Adams, 1979)

Use Pandoc!

1. Install Pandoc
2. In Zotero, use File → Export Library. Export in Better BibTex format as test.bib.

Add the following configuration section to the top of your Markdown file:

---
bibliography: test.bib
...

Run the following command:

pandoc --citeproc test.md -o test.epub

Hey presto! You now have a nicely formatted ePub with all the citations in your preferred format, and the bibliography at the end.

Format the citations better

The citations don't look the way I want them to!

1. Download your preferred Citation Style from Zotero, save it as whatever.csl
2. Add the following into your Markdown file's configuration section csl: whatever.csl
3. Important Make sure that your citations are wrapped in [ and ] otherwise Pandoc will mess things up. See the Pandoc Citations Extension for more details.

Your Markdown should look something like this:

---
bibliography: test.bib
csl: whatever.csl
link-citations: true
...

# Intro

Computers are complicated [@verletComputerExperimentsClassical1967].

Then run pandoc --citeproc test.md -o test.epub - you can change the file type to html, pdf, or anything else your system supports.

Make it look even nicer

This is the boilerplate that I use.

---
bibliography: test.bib
csl: whatever.csl
link-citations: true
reference-section-title: References
title: My brilliant title
...

# Intro

Computers are complicated [@verletComputerExperimentsClassical1967].

## Something

More info [@smithEvolutionInnovationCapability2017]

<div id="refs">
## References
</div>

## The End
Goodbye!

That gives you a title, ensures that all in-text citations link down to the references, and puts the references somewhere other than the end. Here's the HTML it produces using pandoc --citeproc test.md -o bib.html :

<h1 id="intro">Intro</h1>
<p>Computers are complicated <span class="citation" data-cites="verletComputerExperimentsClassical1967">(<a
         href="https://shkspr.mobi/blog/2022/11/zotero-citations-in-markdown-publishing-to-epub-or-pdf/#ref-verletComputerExperimentsClassical1967" role="doc-biblioref">Verlet, 1967</a>)</span>.</p>
<h2 id="something">Something</h2>
<p>More info <span class="citation" data-cites="smithEvolutionInnovationCapability2017">(<a
         href="https://shkspr.mobi/blog/2022/11/zotero-citations-in-markdown-publishing-to-epub-or-pdf/#ref-smithEvolutionInnovationCapability2017" role="doc-biblioref">Smith <em>et al.</em>, 2017</a>)</span>
</p>
<div id="refs" class="references csl-bib-body" role="doc-bibliography">
   <h2 id="references">References</h2>
   <div id="ref-smithEvolutionInnovationCapability2017" class="csl-entry" role="doc-biblioentry">
      Smith, C. <em>et al.</em> (2017) <span>‘The <span>Evolution</span> of an
         <span>Innovation Capability</span>’</span>, <em>Research-Technology
         Management</em>, 60(2), pp. 26–35. Available at: <a
         href="https://doi.org/10.1080/08956308.2017.1276388">https://doi.org/10.1080/08956308.2017.1276388</a>.
   </div>
   <div id="ref-verletComputerExperimentsClassical1967" class="csl-entry" role="doc-biblioentry">
      Verlet, L. (1967) <span>‘Computer "<span>Experiments</span>" on
         <span>Classical Fluids</span>. <span>I</span>. <span>Thermodynamical
            Properties</span> of <span>Lennard-Jones Molecules</span>’</span>,
      <em>Physical Review</em>, 159(1), pp. 98–103. Available at: <a
         href="https://doi.org/10.1103/PhysRev.159.98">https://doi.org/10.1103/PhysRev.159.98</a>.
   </div>
</div>
<h2 id="the-end">The End</h2>
<p>Goodbye!</p>

If that was useful to you - or if you think you know a better way - please drop a comment in the box.

One of the things people do when trying to create a new product or service is to write little user stories to illustrate the problem they're solving. You've probably seen this sort of thing:

• As a… busy parent, I want… a push alert from my washing machine; so that I… can stay on top of the laundry.
• As a… citizen, I want… an online passport application process; so that I… can save time and money.
• As a… corner-shop owner, I want… a simple way to accept credit cards; so that I… can attract younger customers who don't carry cash.

Stories are not always good nor, indeed, accurate. But they help you and your team explore what sort of problems you're trying to solve. Ideally, they're backed up with user research - but even if they're just figments of your imagination, they give you something to work towards.

One of the (many) failures of Cryptocurrencies is that they have a weak to non-existent set of user-needs. Very few of the "whitepapers" even discuss users, except in the abstract. They ones that do talk about specific needs are really just addressing the fact that the USA has an antiquated financial system - they are mostly ignorant about the financial services which exist in the modern world.

Let's take a look at one of the ur-texts of this movement - the original BitCoin paper. The problems it identifies are that users want:

1. To "transact directly with each other without the need for a trusted third party."
2. Lower transaction costs, because high costs are "limiting the minimum practical transaction size and cutting off the possibility for small casual transactions".
3. Protection from fraud via "Transactions that are computationally impractical to reverse".

And that's it as far as end-users are concerned. There's a little bit about privacy - but it doesn't discuss whether users want it. There is, I suppose, an overarching need for an honest network - protected by cryptography.

It is obvious that BitCoin has failed on its first two user needs. It is impossible to transact without being online and trusting thousands of third-parties to process the transaction. There's no way to hand over, say, a USB stick full of bitcoins without the risk of double-spending.

Similarly, the transaction costs of BitCoin (and most cryptocurrencies) are comically high. In the UK I can send a single penny to any other UK bank account and it doesn't cost me anything.

What about (3)? Does BitCoin protect users from irreversible transactions? Yes! But do users want that? No!

Let's take a look at that whole sentence from the paper:

Transactions that are computationally impractical to reverse would protect sellers from fraud, and routine escrow mechanisms could easily be implemented to protect buyers.

(Emphasis added).

Both buyers and sellers want escrow. There will always be a user-need to mediate disputes. One of the reasons users love credit cards is that - in the UK at least - there are incredibly strong consumer rights laws. If a merchant screws you over, you can ask the card company to refund you. Try doing that with a cryptocurrency exchange!

Creating Some User Needs

I think one of the reasons NFTs got so much (brief) attention is that they had identified a well-understood problem and proposed a fix for it.

• As an artist, I want to receive a percentage of any future sales of my artwork, so that I see the gains in any future popularity.
• As a customer, I want to invest in something which might grow in price, so that I can resell and make a profit.

That's it. That's all there is to the NFT value-proposition. NFTs were never about supporting artists; PayPal and bank transfers has that covered. NFTs are deeply flawed in many ways, but they relentlessly focused on those two user stories - and users flocked to them. For a while.

So, what might a user want from a modern, digital currency? Here's my attempt to sketch out some idea. You will notice that many are contradictory! Users are not a homogenous blob and may have needs which can't be satisfied at all times.

Instant payments

These are part of those "parity with cash" needs. If I hand you a fiver, the transaction happens instantly & no-one takes a cut. Similarly, all online money-transfers is instant & free. Any future currency must meet those user expectations.

Low / No Fees.

In the UK, I don't pay anything for access to a credit card. I don't pay for cash withdrawals or deposits. I can transfer tiny or large sums of money instantly without a fee.

Now, the flip of this is that merchants pay a fee for handling my credit card transactions. They also bear the cost of transporting and insuring cash. Banks don't charge for current accounts or transactions - but do have huge fees for overdrafts and the like.

Crypto tends to evenly split this cost between the customer and the merchant. That's probably a fairer way to do things. But it is unusual - and will take some work to convince users that it is sensible.

Evading Capital Controls

Some countries place limits on how much money can enter or leave the country. Try wandering through an airport with a large suitcase full of cash and see what happens!

So, "As a user, I want to move money in and out of a country without a trace, so I can ..." what exactly?

Insured Funds

One of the things the BoE talked about was what to do if someone lost their wallet. If your cash gets shredded, you can usually send it back to the bank for a replacement. If your bank goes bankrupt, a central bank insures your money.

How could a digital currency provide that service?

Escrow

A customer wants refunds, but a merchant disputes that. Enter an escrow service. A middle-man to hold on to the money /goods until both parties are satisfied. In the non-crypto world, this is commonplace. Smart-contract sort-of provide this for digital exchanges. But they're unregulated.

Your story here

It's late and I'm tired. The original BitCoin paper didn't analyse user needs - and has manifestly failed on 2 out of its 3 identified needs.

If digital currencies are to thrive - they need to be attractive to real users. It isn't enough to make something mathematically perfect, or which only appeals to gamblers, or con-artists. Products live and die by addressing problems faced by users.

One of the many reasons crypto is dying is that it still doesn't have a compelling user story other than "number go up".

I think it is fair to say that it is the best computer game I've ever published. And probably the only time I'll ever be reviewed in The Guardian and Kotaku!

Anyway, a year ago I was contacted by an art student. They wanted to adapt my game into a comic for their art class. How could I refuse?

This beautifully styled, monochrome pages were the result:

They also provided me with a rendering with the text in English:

Isn't it gorgeous!

I'm overjoyed that someone turned my daft little game into a work of art. Now all I need to do is find a way to print it in the quality it deserves.

Huge thanks to "flyingpotato" (yes, that's how they want to be credited!) for creating something so distinctive and delectable.

But it doesn't need to be this way. Exposing users to the immense variability in pricing is probably too dangerous - as seen in Texas recently. Imagine if your electricity provider could say "hey, prices are going to be high tonight. We'll pay you to use less electricity!"

Well, that's what my energy provider are doing! Here's an email I received from them:

Join Octopus and we both get £50!

The UK is experimenting with flexible energy pricing. This is made possible by the large-scale rollout of smart meters. They measure electricity consumption and report it back at up-to half-hourly intervals⁰.

My solar panels already take advantage of this capability. I get paid the half-houly market-price for every kWh of sunshine I export. Octopus publish their price so, in theory, I could tell my battery to store energy at midday when it is cheaper and then sell back to the grid when it is more expensive.

Now, there's just one catch with this offer. The reduction is based on your average usage at that time. If you usually have your electric oven on between 1700-1800 then you'll be able to make some big savings. But if you're already fairly efficient, it is less likely you'll be able to take advantage of this.

The day before the experiment, this was our half-hourly usage:

So, if we were to turn off all our electricity we might be given about £1.95 as a reward. Not nothing, but perhaps not worth sitting in the dark for an hour?

But, in the spirit of experimentation, we tried it! Lights off, TV & games consoles off, chargers unplugged, no cooking or cups of tea. We made our own entertainment¹.

What we earned & what we learned

Using our home monitoring software, here's what happened to our electricity use for that hour:

As you can see, at 1700 we dropped our usage significantly. The hours before and after, we were using about 900Wh. But from 1700-1800 we were down to about 184Wh.

That's about 20% of our usual use for that hour. Our usual cost at that time of day would have been around £0.20 for the hour. Instead, we only used about 4 pence-worth of electricity.

A few days later, we got this message from our electricity provider:

OK! They base the average over the last few weeks, so it isn't surprising that their numbers differ slightly from mine.

So, how much of a reward do they pay us?

I have no idea why they do this in points. Presumably I might be able to exchange the points for a chance to win a prize? Seems silly to me; just use money.

And then, of course, the inevitable "nudge":

I'm not sure how I feel about that sort of nudge. I think we did everything we reasonably could. I'm not a big fan of that sort of competition.

Was it a hardship for us to turn out the lights for an hour? No. But there is a slightly worrying idea that people might feel pressure to turn off vital things like life-saving equipment, freezers, or alarms.

Will people change their behaviour for a couple of quid? That remains to be seen. Should people have to change their behaviour? That's a more complex question. Janet Gunter has a blog post which explores the limitations of this approach.

The idea of a personal carbon footprint has despicable origins - but when scaled up to a population level, it can be a useful way to phrase a problem.

If people merely shift their electricity consumption to an hour before or after the load shedding period, does it have a meaningful impact?

Anyway, I think it is useful to do experiments like this. Perhaps thousands of people will find it useful and it'll scale up to millions of people. Perhaps it won't make a difference at all. But the only way we can find out is by experimenting. And I'm eager to see what happens!

If you're interested in trying this, join Octopus and we both get £50!

How did the rest of you do?

WNJ85

@Will L-J

The figures are in for #OctopusEnergy #SavingSessions and am pretty happy with the outcome considering didn’t go to extremes like some! Popped the #ASHP into ‘Holiday Mode’ for the hour, delayed dinner prep till after 18:00 and didn’t plug the #BEV in to charge straight after getting home as one would normally if needed charge!

---

wishy

@Steve Scott

@Edent 48% cut aparrently. Not bad for a test run without changing much

---

hd

@Hermit Dave

Took part in 1st #SavingSessions by #NationalGrid & #OctopusEnergy. 83% reduction in electricity usage over an hour compared to average of previous 10 days.

---

mttkng

@Matt King 🍔

Got the results of my #OctopusEnergy #savingSessions from earlier this week. I cut down usage in that hour by 89% earning a bonus of £1.43. Apparently I saved 59% more than my neighbours. Useless bunch.

---

sfscotland

@S Fraser

@Edent I'd completely forgotten to check until I saw your results. I saved the equivalent of £4.61 with a 91% reduction in use during that hour.

---

This year I was doing lots of reading for my MSc - which was mostly academic papers. I also didn't have any long relaxing breaks. But, nevertheless, I'm happy to have read 42 books. Not bad!

A few stats. The gender split was roughly 50:50. Some of the books were compilations and I think a few of the authors are non-binary. I'm pretty happy with that balance.

As per my usual reading habits, I alternate between reading fiction and non-fiction. A straight split down the middle.

There were a couple that I didn't finish - I'm reading for pleasure and don't need to slog through turgid prose. Well, that's what I tell myself. I still feel guilty!

So, what did I learn?

• The UK's involvement in the US Civil War was... complex. And politics hasn't changed much since then.
• Sci Fi is still my favourite genre. I just want to go to the stars.
• There are some interesting scholarly works on Shakespeare and race but I'm not always clever enough to follow them.
• I don't remember reading about the antimemetics division. How did that post get on my blog?!
• IBM were in league with the Nazis and are apparently unrepentant.

This year? MORE READING! I've got a stack of unread eBooks which give me guilty feelings every time I open up my eReader.

If you have written a book - please let me know and I'll try to get to it by this time next year.

You can read previous years at 2021, 2020, 2019, 2018, 2017, 2016, 2015, 2013⁰, 2012, 2011, and 2010.

If all goes well, at this very moment I will be in San Francisco having just attended Twitter's Chirp conference. I've visited SF a few times and, I'll be honest, it isn't my top destination for spending a birthday. But Twitter paid for the flights, so it would be churlish to complain! I'm missing my wife and it feels odd not spending the day with here.

I was due to be in SF today. But Alan the Hyperprat cancelled the conference at the last minute - leaving me a couple of hundred quid out of pocket, and disappointing all the friends I'd arranged to see. Truly it is the worst thing that he has ever done to anyone ever. Anyway, follow me on Mastodon

The last year has been reasonable. Obviously COVID is still casting a long shadow. Gigs are still being cancelled, and some friends are still cautious about meeting up. But we've been able to go to a few concerts and plays. The Elizabeth Line opening up has been transformational. A 15 minute ride to work! 25 minutes from Central London! Brilliant!

A weird highlight of the year was getting my beard trimmed. I've always done it myself, but fancied getting a professional to shear me before a photoshoot (more on that next year). So I got a recommendation for a Turkish barber near the office. It was a decidedly odd sensation - but I look less feral now.

Anyway, here's how I scored against last year's hope's and dreams:

As I said in a previous post, I don't know what I'm doing with my so-called career. I plan to stay in the Civil Service for at least another 12 months in order to get my MSc. I've recently been regraded, which takes off some of the financial pressure. The work is still interesting and important.

Still don't know what I'm doing. My MSc is entering the end-game. A few more words to write, a viva, then done. Hopefully within the first month of January.

I'm hoping the world opens up a bit more so I can continue exploring it.

It did, and I have! Went to Rotterdam, Dan Haag, and Nicosia. It felt good to go exploring.

Come the 1st of January 2022, I'll have been blogging every day for 2 years. I'm not sure if I want to keep up that pace!

I did not! The pressure of work and the MSc were too much. I'm still blogging regularly (some might say incessantly) and I'm still enjoying it.

I need a proper holiday. Sitting around at home doing nothing isn't a great way to relax and recharge.

Nicosia was a great place to visit. Lots of delicious food, beautiful scenery, and many museums. Similarly the Netherlands was a fantastic minibreak. What they weren't was relaxing. Running around a city trying to see everything is fun. But I need to sit by a pool and not have to think about anything more complex than my tan lines.

But it felt like rather a muted year. Perhaps that's age or the state of the world talking.

There were some lovely highlights. As well as holidays, we went to Canada to see a friend get married. I love a good wedding!

I met an excellent cat!

Terence Eden is on Mastodon

@edent

Dear @thecatreviewer,
I have made a most wonderful discovery.
Sphynx cats *love* to scritch on beards.
I mean LOVE.
10/10 for snuggles from this little hairless friend. pic.x.com/uxJJPtFTpO

---

Got an Oculus and took my first steps into the Metaverse. It's a bit rubbish.

Met Bruce Perens and got to show him around TNMOC.

Read some pretty good books.

Presented at a few conferences - including in person!

Work has been tough - taking on a new job, while studying, and many events occurring with those a few steps above my pay-grade.

And I am nearly done with my MSc. I'll be handing in the final project in a couple of weeks.

So, what for 2023?

• Graduate from MSc (and regain my free time).
• Go on a proper relaxing holiday where all I do is sip cocktails and read books (and maybe a little sightseeing).
• Put more effort into things like OpenUK and BCS (I need to give back more to the community).
• Consider joining a Worshipful Company (looks like it could be a fun way to get more involved with charities).
• Blog more. I'm not introspective enough to keep a diary of how I feel - but I do enjoy looking back at what I thought. Probably not daily - but we'll see.
• Read more. I have too short a commute to get into a good book. So I'm going to have to proactively set aside some time.
• See more friends. I'm conscious that I've been keeping to myself rather a lot. If you fancy a beer and a natter, please get in touch :-)

Right, let's see what adventures the next year brings.

When you scroll through the website, you normally see a list of replies. It looks like this:

Because it acts as a one-dimensional list, there's no easy way to figure out which post someone is replying to.

The data structure underlying the conversation is quite different. It actually looks like this:

Concepts

In Mastodon's API, a post is called a status.

Every status on Mastodon has an ID. This is usually a Snowflake ID which is represented as a number.

When someone replies to a status on Mastodon, they create a new status which has a field called in_reply_to_id. As its name suggests, has the ID of the status they are replying to.

Let's imagine this simple conversation:

1. Ada: "How are you?"
2. Bob: "I'm fine. And you?"
3. Ada: "Quite well, thank you!"

Message 2 is in reply to message 1. Message 3 is in reply to message 2.

In Mastodon's jargon, message 1 is the ancestor of message 2. Similarly, message 3 is the descendant of message 2.

  → Descendants →
1--------2-------3
   ← Ancestors ←

Branches

Now, let's imagine a more complicated conversation - one with branches!

1. Alice: What's your favourite pizza topping?
├── 2. Bette: Pineapple
│   ├── 4. Chuck: You make me sick!
│   └── 7. Dave: Yeah, I love pineapple too
└── 3. Chuck: Mushroom are the best
    ├── 5. Alice: Really?
    │   └── 6. Dave: Button mushrooms are best!
    └── 8. Elle: I like them too!

As you can see, people reply in threads. In this example, 2 is a different "branch" of the conversations than 3.

It looks a bit more complicated with hundreds of replies, but that's it! That's all you need to know!

API

If you want to download a single status with an ID of 1234 the API call is /api/v1/statuses/1234

If you want to download a conversation, it is a little bit more complicated. Mastdon's API calls a conversation a context

Let's take the above simple example - Ada and Bob speaking. Ada's first status has an ID of 1. To get the conversation, the API call is /api/v1/statuses/1/context

That returns two things:

• A list of ancestors. This is empty because 1 is the first status in this conversation.
• A list of descendants. This contains statuses 2 and 3.

You will note, the context does not return the status 1 itself.

Let's suppose that, instead of asking for the context of status 1, we instead asked for 2. This would return:

• A list of ancestors. This contains status 1.
• A list of descendants. This contains status 3.

What about if we asked for 3? This would return:

• A list of ancestors. This contains status 1 and 2
• A list of descendants. This is empty because 3 is the last message in this conversation.

Branches

When it comes to complex threads - like the pizza example - things become a bit more difficult. Let's see the example again:

1. Alice: What's your favourite pizza topping?
├── 2. Bette: Pineapple
│   ├── 4. Chuck: You make me sick!
│   └── 7. Dave: Yeah, I love pineapple too
└── 3. Chuck: Mushroom are the best
    ├── 5. Alice: Really?
    │   └── 6. Dave: Button mushrooms are best!
    └── 8. Elle: I like them too!

Suppose we ask for the context of the message with ID 5. This would return:

• A list of ancestors. This contains statuses 1 and 3
• A list of descendants. This contains status 6.

That's it!?!? Where are the rest? They are part of a different conversation branch. Even status 8 isn't returned because it's a reply to 3, not 5.

In order to get the full conversation, we need to be sneaky!

The list of ancestors contains the first message in the conversation. So we can grab that, and then call context again for its ID.

Let's dive into some Python code to see how it works.

Code

This uses the Mastodon.py library for calling the Mastodon API and the Python treelib to create a conversation tree data structure.

This code connects to Mastodon and receives the status for a single ID.

from mastodon import Mastodon
from treelib import Node, Tree

mastodon = Mastodon( api_base_url="https://mastodon.example", access_token="Your personal access token from your instance" )

status_id =  109348943537057532 
status = mastodon.status(status_id)

Getting the conversation means calling the context API:

conversation = mastodon.status_context(status_id)

⚠ Note: Calling the context on a large thread may take a long time. The longer the conversation, the longer you'll have to wait.

If there are ancestors, that means we are only on a single branch. The 0th ancestor is the top of the conversation tree. So let's get the context for that top status:

if len(conversation["ancestors"]) > 0 :
   status = conversation["ancestors"][0]
   status_id = status["id"]
   conversation = mastodon.status_context(status_id)

Next, we need to create a data structure to hold the conversation. We'll start by adding to it the first status in the conversation:

tree = Tree()

tree.create_node(status["uri"], status["id"])

Finally, we add any replies which are in the descendants. It is possible that some earlier statuses have been deleted. So we won't add any status which are replies to deleted statuses:

for status in conversation["descendants"] :
   try :
      tree.create_node(status["uri"], status["id"], parent=status["in_reply_to_id"])
   except :
      #  If a parent node is missing
      print("Problem adding node to the tree")

That's it! Let's show the tree:

tree.show()

Here's what it should look like:

2022-11-14 20:02 Edent: Today I was meant to be flying in to San Francisco to attend Twitter's Developer Conference - Chirp.Twitter had paid for my flights and hotel, because I was one of their developer insiders. I planned to spend the week meeting friends old and new.Instead, Alan the Hyperprat canceled the conference. So I'm staying in the UK.So I'm going to spend the week hacking on Mastdon's #API and building cool shit.  That'll show him!You can see what I'm working on at https://shkspr.mobi/blog/2022/11/building-an-on-this-day-service-for-mastodon/ https://mastodon.social/users/Edent/statuses/109343943300929632
├── 2022-11-14 20:10 Edent: Oh! And I was meant to be attending a Belle & Sebastian gig tonight. I canceled those tickets for I could fly to SF.So far, I reckon Alan's acquisition of Twitter has cost me close to £190.Wonder if he's good for the money? https://mastodon.social/users/Edent/statuses/109343972435801664
│   ├── 2022-11-14 20:14 thehodge: @Edent reminds me of the time I was booked to speak at a conference in Munich and I excitedly booked a behind the scenes tour of the worlds largest miniature city!Then the company went under!Gutted. https://mastodon.social/users/thehodge/statuses/109343989481494630
│   ├── 2022-11-14 21:16 Janiqueka: @Edent the way my bill for him keeps increasing https://mastodon.online/users/Janiqueka/statuses/109344233355230523
│   ├── 2022-11-14 21:19 henry: @Edent I was due to be at B&S tomorrow but it’s been postponed again.. not sure if that makes it better or worse for you! https://social.lc/users/henry/statuses/109344244402822729
│   │   └── 2022-11-15 04:53 Edent: @henry again!? Ah well!Hope you get to see them soon. https://mastodon.social/users/Edent/statuses/109346031194446940
│   ├── 2022-11-15 09:18 Amandafclark: @Edent send him an invoice :) https://mastodon.social/users/Amandafclark/statuses/109347071811426672
│   └── 2022-11-15 11:29 Edent: One of the #MastodonAPI projects I'm working on is a better way to view long & complex threads.You may have seen me build something similar for the other site a while ago - demo at https://shkspr.mobi/blog/2021/09/augmented-reality-twitter-conversations/ - so I'm hoping I can do something similarly interesting.Main limitation is getting *all* of the conversation threads. It looks like the context API isn't paginated. But I might be being thick. https://mastodon.social/users/Edent/statuses/109347587353822637
│       ├── 2022-11-15 11:36 bensb: @Edent Excellent project. You might have seen, but there's also this feature request for better 🧵 handling: https://github.com/mastodon/mastodon/issues/8615 https://genomic.social/users/bensb/statuses/109347612990393791
│       ├── 2022-11-15 11:39 Edent: Cor! That @katebevan is good for engagement! Look at all those conversations she's kicked off! https://mastodon.social/users/Edent/statuses/109347627634008550
│       │   ├── 2022-11-15 11:58 Edent: Indeed, how could they be?That means that ID of a reply is different depending on where you see it.So the ID of this post is:mastodon. social /@ edent/ 123456But when you see it on your server, it might appear as:your. server /@ edent/ 987654The #MastodonAPI copes with this really well. But it is a mite confusing to get one's head around. https://mastodon.social/users/Edent/statuses/109347703064222520
│       │   │   ├── 2022-11-15 12:02 erincandescent: @Edent the numeric IDs are not part of the protocol - it's all URL based. Pleroma uses UUIDs for example https://queer.af/users/erincandescent/statuses/109347716173491502
│       │   │   │   └── 2022-11-15 12:06 Edent: @erincandescent oh! That's interesting. Thanks. https://mastodon.social/users/Edent/statuses/109347734283971306

Once you have a tree, you can format the contents however you like.

Grab the code

You can download the code for my Mastodon API tools from CodeBerg. Enjoy!

2022: "On the one hand, invading Ukraine is bad. On the other hand, cheap gas from Russia keeps the UK economy working."

Plus ça change, plus c'est la même chose…

This is an incredible book. I knew very little about the American Civil War - this is a thorough history of that bloody event told from the perspective of the UK.

The UK was officially neutral. But that didn't stop hundreds of British subject from joining up to fight on both sides. It also didn't stop frenzied diplomatic efforts to turn the tide in the UK's favour. And that led to incessant lobbying, fake-news, and skulduggery within the realm.

Dr Foreman's book is long and detailed. Perhaps a little too detailed. There are some excellent discussions of battles which - though important to the war - don't really entangle with the British side of the story. The book occasionally gets bogged down in what I consider irrelevant details and diversionary footnotes.

She avoids making moral judgements about the actions of the participants. That isn't to say she "both sides" the war - the South were running an empire based on slavery - but she convincingly points out multiple examples of the moral failings of the Union and the UK.

What I found stunning is just how relevant the events are to the politics of today. Here are a few of the excerpts I found interesting:

he had been so certain that the British and French would intervene as soon as their cotton stocks were low. ‘Ah, yes,’ Benjamin replied, ‘I admit I was mistaken! I did not believe that your government would allow such misery to your operatives, such loss to your manufacturers, or that the people themselves would have borne it.’

and

The US Consuls’ descriptions of the suffering in Lancashire convinced Seward that the British would not hesitate to interfere in the war if the alternative meant starvation across wide swathes of England.

This seems to be exactly where we are with Russia. Will people of the UK continue supporting Ukraine in the face of the economic hardship? It's also interesting to see how President Lincoln used the media to shore up support:

Lincoln had written an eloquent letter to the ‘Workingmen of Manchester’ thanking the cotton workers for their patience and sacrifice. ‘Whatever misfortune may befall your country or my own,’ declared the President, ‘the peace and friendship which now exist between the two nations will be … perpetual.’

Again, very similar to the messages we see from President Zelenskyy.

With regard for the need for propaganda:

With one half of what he threw away in odious espionage I could have bought the British Press … every newspaper writer in London can be purchased, from those of The Times down … I do not mean to say that each would openly take cash; but each will take a consideration suitable to his taste.

and

One of Hotze’s favourite methods was to supply an acquaintance with fresh information, in the shape of a pro-South editorial that required little editing. If the article was printed, Hotze would always insist that the submitter should keep the 10 guinea fee.

I wonder where we are today? Which editors and writers will gladly take foreign emoluments?

The Times newspaper has been publishing for so long, it isn't surprising to find it neck-deep in the bloody bowels of history.

The Times went further and accused Lincoln of inciting the slaves in the South to kill their owners, imagining in graphic terms how the President ‘will appeal to the black blood of the African; he will whisper of the pleasures of spoil and of the gratification of yet fiercer instincts; and when blood begins to flow and shrieks come piercing through the darkness, Mr Lincoln will wait till the rising flames tell that all is consummated, and he will rub his hands and think that revenge is sweet’

This is a repeated theme throughout the book.

It was the news of Stonewall Jackson’s death, however, which made the Confederates spring into action. They were amazed and delighted by the spontaneous outpouring of public grief in England. Newspapers carried long eulogies to the fallen hero; The Times even compared Jackson’s death to Admiral Nelson’s at Trafalgar. Flags flew at half mast at many cotton mills.

Their assessment of the Gettysburg Address was similarly unfavourable

The Times, thought that Lincoln’s speech had been a total failure. English readers were told that the ‘imposing ceremony’ was ‘rendered ludicrous by some of the luckless sallies of that poor President Lincoln’

Of course, at the assassination of Lincoln, the papers went full Princess Diana

The British press was united over the tragedy of Lincoln’s violent death. Newspapers that had routinely criticized the President during his lifetime rushed to praise him.

There is also a look into the psychology of the confederates.

One bag contained the Confederate flag and a pouch filled with Virginian soil. Georgiana intended to give birth with the flag draped symbolically above the bed and the soil placed underneath to ensure that the baby was a true Virginian.

The modern revisionist history that the Civil War wasn't about slavery is undermined by the actions of those at the time:

‘Address to the Christians throughout the World’. Signed by the ninety-six clergymen of Richmond, Virginia, the ‘Address’ urged fellow Christians to protest against Lincoln’s Emancipation Proclamation.

There's all sorts of modern parallels. Especially when it comes to hate speech directed at public figures.

Mrs Adams was already pining to leave London when an anonymous letter arrived at the legation: Dam the Federals Dam the Confederates Dam you both Kill you damned selves for the next 10 years if you like; so much the better for the world and for England. Thus thinks every Englishman with any brains. NB.PS. We’ll cut your throats fast enough afterwards for you if you aint tired of blood, you devils.

While I make no comment on the state of modern politics, there are some… interesting echoes through the ages

The British government’s majorities in both Houses were so slim that Palmerston was desperately casting about for allies. He was furious with Gladstone, who gave a speech in the Commons on 11 May that deeply antagonized the conservative wing of the Liberal Party.

And

There was a national uproar after Palmerston and Russell announced that Britain would not fight alongside the Danes after all. Whether the government’s course was right or wrong mattered less than the obvious fact that it was a complete reversal from the one originally proposed.

Of course, history doesn't repeat. But it does rhyme.

By April 1868 there had been four Prime Ministers in three years: Palmerston, Russell, Derby and Benjamin Disraeli; Adams’s longevity at the legation during a period of such rapid political transition changed his public persona from that of a Yankee crank to a pillar of the diplomatic community.

I also found it grimly amusing that, despite fighting against slavery, the notion of equal rights was far from the mind of the Americans.

The second-class status of the coloured regiments was reflected in their pay for the first two years – which stayed at $7 a month, only just over half the $13 paid to whites – until Congress rectified the inequality.

And

In Manhattan, a delegation of black New Yorkers was denied the right to walk behind Lincoln’s funeral cortège. When the White House intervened, a police escort had to protect the black marchers from the violence of the mob.

As I said, it is a long book - about 1,500 printed pages. I found myself occasionally lost trying to work out whether a person was British, Confederate, Union, or something else. That's probably my failing at knowing how to retain knowledge while reading history. But you can't fault the detail and breadth of sources used.

I found it to be an exemplary history. There is so much that I didn't know about the Civil War. And I was completely ignorant of Britain's role in the affair.

About a decade ago, I wrote about how the Twitter archive works and where it is deficient. Things have got better, but there are still annoying limitations.

For example, Hannah Kolbeck - founder of the Alt Text Reminder Bot recently pointed out that there's no alt text in the archives.

Here's a snippet of Twitter's JSON for an image I posted:

"media" : [
   {
      "expanded_url" : "https://twitter.com/edent/status/1579574033720705025/photo/1",
      "indices" : [
        "66",
        "89"
      ],
      "url" : "https://t.co/J1hr0ZfbTl",
      "media_url" : "http://pbs.twimg.com/media/FevGM32XEAA0FX2.jpg",
      "id_str" : "1579574018776174592",
      "id" : "1579574018776174592",
      "media_url_https" : "https://pbs.twimg.com/media/FevGM32XEAA0FX2.jpg",
      "sizes" : {
           "small" : {
                "w" : "680",
                "h" : "510",
                "resize" : "fit"
              },
              "medium" : {
                "w" : "1200",
                "h" : "900",
                "resize" : "fit"
              },
              "thumb" : {
                "w" : "150",
                "h" : "150",
                "resize" : "crop"
              },
              "large" : {
                "w" : "1236",
                "h" : "927",
                "resize" : "fit"
              }
       },
       "type" : "photo",
       "display_url" : "pic.twitter.com/J1hr0ZfbTl"
     }
],

Lots of different media sizing options, but no room for accessibility.

By comparison, the Mastodon social network gives you the alt text. Here's a snippet of Mastodon's JSON for the same image which was cross-posted:

"attachment": [
   {
     "type": "Document",
     "mediaType": "image/jpeg",
     "url": "/media_attachments/files/109/145/933/102/890/212/original/84ae501e39f45091.jpg",
     "name": "A sign for priority seating. The pregnant person's face has been replaced by 😍. The person holding a baby has a face of 😫. The elderly person with a cane has 🥴.",
     "blurhash": "UhKdk{0LRit6-:t6WCWC-oxaRmWBozt7xaa|",
     "width": 1236,
     "height": 927
   }
],

Mastodon is a friendlier alternative to Twitter and - mostly - gets accessibility right. There's still some work to do

You can fix Twitter's missing alt text using Hannah's Alt Text Archive Tool. That'll get you a JSON file full of your alt text, which you can use to recreate your archive.

Look, it's obvious that Alan doesn't give a flying fuck about accessibility, so I don't expect this to change any time soon.

Instead, people should do what they did when MySpace went to shit; move to a different platform.

Join Mastodon today!

The Twitter version never really worked properly because the Twitter API doesn't support searching for historic Tweets. What I had to do was manually build search queries like: ?q=from:edent (until:2011-11-15 since:2011-11-14) and redirect people to the website.

Eugh!

I'm trying to build something similar for the Mastodon social network. Yes, I know it is new to you - but some of us have been there for several years.

So here's how to search Mastodon for posts made on specific dates!

(Skip to the code and ignore all the exciting preamble.)

Sadly, the Mastodon Search API is still quite basic. It isn't possible to directly search by user or by data parameters.

If you download the archive of all your posts, you'll find an ActivityPub feed called outbox.json - which is a collection of everything you've ever posted.

It can be parsed using jq to get all of the statuses posted on a specific day:

cat outbox.json | jq ".orderedItems[] | select (.published | fromdateiso8601 > 1636533813) | select (.published | fromdateiso8601 < 1636620302)"

The fromdateiso8601 are the Unix epoch times from 365 days ago and 364 days ago.

So, conceptually, it's possible to build - as long as you're willing to download your data and manually parse it. Let's see if we can do a little better than that.

Building in Python

We're going to build this using Python3 and the Mastodon.py library.

Install the library on the command line with:

pip3 install -U Mastodon.py

Now, launch Python and load the library:

from mastodon import Mastodon

We'll need an API key. Go to the website of your Mastodon instance. In settings, there should be an option called "Development". Use that to create a new app which has the "Read" scope.

Once created, we will use the "Your access token" which will be a long string of random letters and numbers. In this example, we'll be using abc123. Your real access token will be longer!

Let's set up a connection to your Mastodon instance:

instance = "https://mastodon.example.com"
mastodon = Mastodon( api_base_url=instance, access_token="abc123" )

Next, we need your user ID. This isn't your @ name, instead it is the numerical ID assigned by the server. For this, we need the Verify account credentials API call:

mastodon.me()

That produces:

{
   'id': 7112,
   'username': 'Edent',
   'acct': 'Edent',
   'display_name': 'Terence Eden',
   ...

Looks like I was a pretty early adopter!

Getting the last 20 statuses is:

me = mastodon.me()
my_id = me["id"]
mastodon.account_statuses(id = my_id)

The problem is, that can only receive a maximum of 40 statuses at a time.

statuses = mastodon.account_statuses(id = my_id, limit="40")

We need to use Pagination. The API makes it pretty easy to grab the next page. There's also a call to get every post.

Be warned - this can take a long time. If you have thousands of posts it may take a few minutes. It can also quickly deplete your API rate limits. Use with caution!

We can reduce some of the load by excluding anything you've "boosted".

statuses = mastodon.account_statuses(id = my_id, limit="40", exclude_reblogs=True)
all_statuses = mastodon.fetch_remaining(statuses)

You can run len(all_statuses) to see how many you have retrieved.

The next step is finding all the posts which happened on a certain day each year. Let's say we want every post which happened on a 14th of February.

The library returns timestamps as Python DateTime objects.

status_date = all_statuses[1]["created_at"]

Shows

datetime.datetime(2016, 11, 1, 17, 4, 23, 842000, tzinfo=tzutc())

The datetime library is pretty handy. You can find the day using status_date.day and month with status_date.month.

This means we can loop through every status and show only the ones we care about.

for status in all_statuses:
     if (status["created_at"].day == 14 and status["created_at"].month == 2):
             print(status["uri"])

That will get you a list of URls which contain posts made on a specific day in previous years.

Putting it all together

from mastodon import Mastodon
instance = "https://mastodon.example.com"
mastodon = Mastodon( api_base_url=instance, access_token="abc123" )
me = mastodon.me()
my_id = me["id"]
mastodon.account_statuses(id = my_id)
statuses = mastodon.account_statuses(id = my_id, limit="40", exclude_reblogs=True)
all_statuses = mastodon.fetch_remaining(statuses)
for status in all_statuses:
   if (status["created_at"].day == 14 and status["created_at"].month == 2 and status["created_at"].year < 2022) :
      print(status["uri"])

Building a time machine

All of the above works, but is pretty inefficient because there's no way to search for specific timeframes on Mastodon. Or so I thought!

Gargron

@Eugen Rochko

@Edent Mastodon uses snowflake IDs, which means the IDs correspond to datetimes. You would need to look up how to construct the ID from a date, but you can use this to retrieve posts by date using min_id/max_id.

---

If we can calculate the maximum and minimum IDs for a given day, we will have a much more efficient search!

Let's dive in to the Mastodon Snowflake code. It is really well documented:

Our ID will be composed of the following: 6 bytes (48 bits) of millisecond-level timestamp 2 bytes (16 bits) of sequence data

OK! Let's look at a typical Mastodon Status ID mastodon.social/@Edent/109326536843609210, it was posted on 2022-11-11 at 18:16.

Let's take the ID 109326536843609210 and perform a bitwise shift on it.

print(109326536843609210 >> 16)
1668190564630

Hey! That looks a bit like a UNIX timestamp! The last three numbers will be the sequence, so we can eliminate them and see what happens if we convert it to a timestamp.

from datetime import datetime
datetime.fromtimestamp(1668190564630/1000)
datetime.datetime(2022, 11, 11, 18, 16, 4, 630000)

Nice! So we can go backward and take a date - say this time last year - and convert it to a maximum and minimum ID.

min_id = ( int( datetime(2022,11,11,00,00).timestamp() ) << 16 ) * 1000
max_id = ( int( datetime(2022,11,11,23,59).timestamp() ) << 16 ) * 1000

Which gives us 109322226892800000 and 109327885271040000 respectively.

Let's try that with the API!

Final Code

from datetime import datetime, timedelta
from mastodon import Mastodon

#  Set up access
instance = "https://mastodon.example"
mastodon = Mastodon( api_base_url=instance, access_token="abc123" )

#  Get user's info
me = mastodon.me()
my_id = me["id"]
year_joined = me["created_at"].year

#  Today's date
year_now  = datetime.now().year
month_now = datetime.now().month
day_now   = datetime.now().day

#  Counter
year_counter = year_now

#  Loop through previous years
#  Start with last year and go down until the user joined
while (year_counter >= year_joined ) :
   year_counter -= 1
   #  The end of today is the start of tomorrow
   #  This means yesterday can take into account leap-years
   today_end = datetime(year_counter, month_now, day_now, 00, 00) + timedelta(days=1)
   yesterday_end = today_end - timedelta(days=1)
   #  Bitwise shift the integer representation and convert to milliseconds
   max_id = ( int( today_end.timestamp() )     << 16 ) * 1000
   min_id = ( int( yesterday_end.timestamp() ) << 16 ) * 1000
   #  Call the API
   statuses = mastodon.account_statuses(id = my_id, max_id=max_id, min_id=min_id, limit="40", exclude_reblogs=True)
   #  Fetch further statuses if there are any
   all_statuses = mastodon.fetch_remaining(statuses)
   #  Print the date and URl
   for status in all_statuses :
      print( str(status["created_at"]) + " " + status["uri"] )

Next Steps

It works on my machine! But that's not really good enough. Ideally I'd like to turn this in to a web app which people could use with their own account.

If you're interested in helping out with that grab the code or drop me a line!

Humans can only understand our own lived reality. So here we have bees' behaviour translated into schemes and intrigues which would not be out of place in a medieval court. Bees wings are roaring engines, and their enemies are the hoards of traitorous insects and arachnids outside the hive. It wonderfully conveys the "alienness" of the bees' experience in a way that makes us empathise with the insects. This is billed as a "thriller" - but it is a lot more than that. Closer to "Pillars of the Earth" in that it takes us on an epic journey through the upheaval of a civilisation.

It is a first rate story, similar to Zindell's "The Idiot Gods".

Is it an allegory for life in a totalitarian state? Possibly. But I think it is so much more than that. A compelling vision of what a bee might feel as it bumbles its way through your petunias.

Dr Sally Le Page

@sallylepage

I LOVE THIS BOOK SO MUCH!

Seriously, I recommend it to everyone all the time. And as an evolutionary biologist, it's so clear how much time @LalinePaull took to get the science right.

Terence Eden is on Mastodon

@edent

Book Review: The Bees – Laline Paull shkspr.mobi/blog/2022/11/b…

---

---

Laline Paull FRSL

@LalinePaull

Replying to @sallylepage@sallylepage Thank you Terence, thank you Sally! I have a new one out, POD, ocean-immersive and similarly intensively researched for accuracy. Cetacean epic, migration crises, hormonal upheaval… many scientists helped with their specialist species, including changes in humpback whalesong.

---