This opens up verification to small organisations, individuals, and anyone who wants to prove who they are. Brilliant!

Verification means that your @username will change to @Your.Website.com - this means that everyone can see your BlueSky account is owned by that specific website. When you change your name, you keep all your followers and posts.

Here are some organisations and people at risk of impersonation who have already done this:

• UK Newspaper https://bsky.app/profile/theguardian.com
• Trade Union https://bsky.app/profile/utaw.tech
• Labour MPs https://bsky.app/profile/sarahowen.org.uk
• Small Publisher https://bsky.app/profile/canongate.co.uk
• Fun Website https://bsky.app/profile/openbenches.org

There is an easy way to get verified and a hard way. Let's do the easy way!

1) Sign Up For BlueSky

Sign up and register a username. This can be anything you want. For example, I registered edent.bsky.social

2) Change Your User ID

Follow these steps:

1. Visit https://bsky.app/settings
2. Scroll down and select "Change Handle"
3. Click "I have my own domain"
4. Select "No DNS Panel". The screen should look like this:
   • 
5. Type in the domain name you want to verify
6. Click "Copy File Contents"

Keep this web page open.

3) Copy and Save Your DID

On your clipboard, you will have a bit of text which looks like this did:plc:dip7ueksh627fxacagfrdyz2

Save it in a text file called atproto-did

It is very important that the file doesn't end with .txt - it must be called atproto-did and nothing else.

The file should only contain the text you copied. Nothing else.

4) Upload The File To Your Website

This is the only technical bit of the process. You need the ability to upload a file to your website. I don't know whether you use FTP, a control panel, or email things to the person who manages your site.

You need to save the atproto-did file in a folder called /.well-known/

If that folder doesn't exist, create it. The folder name must be typed exactly like that, with the dot at the start.

You can check it has worked by visiting YourWebsite.com/.well-known/atproto-did

If you can see your DID, it worked!

5) Change Your Username

Go back to the "Change Handle" web page you opened in Step 2.

Click "Verify Text File" and then "Update".

6) That's It!

Feel free to share this guide with people and organisations who want to get verified on BSky.

Leave a comment if you found it useful or want me to clarify something.

The IntenseDebate comment system is slowly dying. It hasn't received any updates from Automattic for years. Recently it stopped being able to let users submit new comments.

So I've switched to Commentics which is a self-hosted PHP / MySQL comment system. It's lightweight, pretty good at respecting users' privacy, and very customisable. But it doesn't let you easily import comments. Here's how I fixed that.

Export From IntenseDebate

Go to your site's dashboard and click export. They'll email you a link when the process has finished. It's an XML file which looks something like this:

<?xml version="1.0" encoding="UTF-8"?>
<output>
    <blogpost>
        <url>https%3A%2F%2Fopenbenches.org%2Fbench%2F123</url>
        <title>The Page Title</title>
        <guid>https://openbenches.org/bench/123</guid>
        <comments>
            <comment id='123456798' parentid='0'>
                <isAnon>0</isAnon>
                <name><![CDATA[Terence Eden]]></name>
                <email>terence.eden@example.com</email>
                <url>https://example.com</url>
                <ip>198.51.100.123</ip>
                <text><![CDATA[You can read more about the song at Wikipedia <a href="https://en.wikipedia.org/wiki/Pokarekare_Ana" target="_blank">https://en.wikipedia.org/wiki/Pokarekare_Ana</a>  ]]></text>
                <date>2017-08-01 14:51:55</date>
                <gmt>2017-08-01 14:51:55</gmt>
                <score>1</score>
            </comment>
        </comments>
    </blogpost>
    ...

Understand the Commentics Table Structure

Once you've installed and configured Commentics, you will be able to replace the database with your old comments. To do that, you'll need to convert your exported XML file into three CSV files.

there are 3 tables you need to understand.

Users Table

This is the easiest one to understand. The columns are:

id, avatar_id, avatar_pending_id, avatar_selected, avatar_login, name, email, moderate, token, to_all, to_admin, to_reply, to_approve, format, ip_address, date_modified, date_added

Hopefully they're self-explanatory. The moderate is always set to default. The token can be any random string. A typical user will look like this:

2, 0, 0, , , Terence Eden, terence.eden@example.com, default, abc123, 1, 1, 1, 1, html, 127.0.0.1, 2017-08-01 14:51:55, 2017-08-01 14:51:55

Note The user's URL is not part of this table! That confused me at first. It is part of the comment table.

Pages Table

Every comment is associated with a page. Therefore every page needs a table.

id, site_id, identifier, reference, url, moderate, is_form_enabled, date_modified, date_added

Again, pretty straightforward. A typical page looks like:

123, 1, openbenches.org/bench/123, OpenBenches - Bench 123, https://openbenches.org/bench/123, default, 1, 2023-07-16 21:29:52, 2023-07-16 21:29:52

The id is a unique number. I've set it to be the same as my page's actual ID - but it doesn't need to be.

Comments Table

This is a slightly cumbersome table:

id, user_id, page_id, website, town, state_id, country_id, rating, reply_to, headline, original_comment, comment, reply, ip_address, is_approved, notes, is_admin, is_sent, sent_to, likes, dislikes, reports, is_sticky, is_locked, is_verified, number_edits, session_id, date_modified, date_added

You can ignore most of them - unless you really want to record someone's home town - and a typical user looks like this:

4, 17, 123, , , , , , , , Wow! That's a great bench. And I imagine the view must be special too. , Wow! That's a great bench. And I imagine the view must be special too., , 127.0.0.1, 1, Moderating all comments., 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, abc123, 2017-08-10 17:56:50, 2017-08-10 17:56:50

The id is unique per comment. The user_id is matched to the id on the users table. And the page_id is the id on the pages table. notes appears to be hardcoded to Moderating all comments.

Horrible evil no-good Python

Here's a Python script I made after a few beers. It ingests the XML file and spits out 3 CSV files.

import csv
import random
import string

import xml.etree.ElementTree as ET
tree = ET.parse('IntenseDebate.xml')
root = tree.getroot()

comment_header = ["id","user_id","page_id","website","town","state_id","country_id","rating","reply_to","headline","original_comment","comment","reply","ip_address","is_approved","notes","is_admin","is_sent","sent_to","likes","dislikes","reports","is_sticky","is_locked","is_verified","number_edits","session_id","date_modified","date_added"]

i = 1

with open('comments_to_import.csv', 'w') as file:
    writer = csv.writer(file)
    # writer.writerow(comment_header)   
    for child in root:
        for children in child:
            if (children.tag == "guid") :
                guid = children.text
                page_id = int(''.join(filter(str.isdigit, guid)))
            if (children.tag == "comments") :
                for comments in children :
                    commentID = comments.attrib["id"]
                    for commentData in comments :
                        if (commentData.tag == "name") :
                            username = commentData.text
                        if (commentData.tag == "text") :
                            commentHTML = commentData.text
                        if (commentData.tag == "url") :
                            userurl = commentData.text
                        if (commentData.tag == "date") :
                            commentDate = commentData.text
                            i = i+1
                            digits  = random.choices(string.digits, k=10)
                            letters = random.choices(string.ascii_lowercase, k=10)
                            sample  = random.sample(digits + letters, 20)
                            session_id  = "".join(sample)
                            row = [i,i,page_id,userurl,"","","","","","",commentHTML,commentHTML,"","127.0.0.1","1","Moderating all comments.","0","1","0","0","0","0","0","0","0","0",session_id,commentDate,commentDate]
                            writer.writerow(row)

pages_header = ["id","site_id","identifier","reference","url","moderate","is_form_enabled","date_modified","date_added"]
with open('pages_to_import.csv', 'w') as file:
    writer = csv.writer(file)
    # writer.writerow(pages_header)
    for i in range(30000):
        row = [i,"1","openbenches.org/bench/" + str(i),"OpenBenches - Bench " + str(i),"https://openbenches.org/bench/" + str(i),"default","1","2023-07-16 21:29:52","2023-07-16 21:29:52"]
        writer.writerow(row)

users_header = ["id","avatar_id","avatar_pending_id","avatar_selected","avatar_login","name","email","moderate","token","to_all","to_admin","to_reply","to_approve","format","ip_address","date_modified","date_added"]
i = 1
with open('users_to_import.csv', 'w') as file:
    writer = csv.writer(file)
    # writer.writerow(users_header)
    for child in root:
        for children in child:
            if (children.tag == "guid") :
                guid = children.text
                page_id = int(''.join(filter(str.isdigit, guid)))
            if (children.tag == "comments") :
                for comments in children :
                    commentID = comments.attrib["id"]
                    for commentData in comments :
                        if (commentData.tag == "name") :
                            username = commentData.text
                        if (commentData.tag == "url") :
                            userurl = commentData.text
                        if (commentData.tag == "date") :
                            commentDate = commentData.text
                            i = i+1
                            digits  = random.choices(string.digits, k=10)
                            letters = random.choices(string.ascii_lowercase, k=10)
                            sample  = random.sample(digits + letters, 20)
                            session_id  = "".join(sample)
                            row = [i,"0","0","","",username,"","default",session_id,"1","1","1","1","html","127.0.0.1",commentDate,commentDate]
                            writer.writerow(row)

Note my pages all follow a numeric sequence /1, /2 etc, hence the loop to quickly regenerate them. Your pages may be different.

Importing

You will need to use PHPmyAdmin or a similar database manager. TRUNCATE the tables for pages, users, and comments. Then import the CSV files into each one.

If everything works, you will have all your old comments imported into your new system.

Enjoy!

But - fuck me! - the documentation is atrocious! Well, that's a lie. There is no documentation. It has the "Chat to us on Discord" anti-pattern that infects so many otherwise great projects.

So I'm left to figure out how to make the Watchy's haptics work.

The example watchfaces have a file called settings.h which contains .vibrateOClock = true. Set that and you'll get a little buzz on the hour, every hour.

But how do you make it buzz when you've hit your step-count goal? Or when the time is 13:37? Or whenever you want?

There's no documentation. And searching the code for "vibrate" or "vibe" or "vibration" yielded nothing. But I accidentally typo'd my search and entered "vib" - and that did the trick.

In config.h there's this line: #define VIB_MOTOR_PIN 13.

So, pin 13 is wired in to the vibration motor. We can add that line to our own config file. But how do we turn the buzzer on and off?

With a bit of help from Discord, it became clear:

pinMode(VIB_MOTOR_PIN, OUTPUT);     // Set vibration motor pin as an output.
digitalWrite(VIB_MOTOR_PIN, true);  // Enable vibration motor
delay(150);                         // Vibe for 150 ms.
digitalWrite(VIB_MOTOR_PIN, false); // Disable vibration motor.
delay(500);                         // Wait half a second
digitalWrite(VIB_MOTOR_PIN, true);  // Enable vibration motor
delay(250);                         // Vibe for quarter of a second
digitalWrite(VIB_MOTOR_PIN, false); // Stop vibrating

That's the same as the HTML5 Vibrate API - you can't control the intensity, but you can program your own vibration patterns.

I just don't understand why hardware vendors are so crap at writing documentation. Don't they like people using their products?

When you scroll through the website, you normally see a list of replies. It looks like this:

Because it acts as a one-dimensional list, there's no easy way to figure out which post someone is replying to.

The data structure underlying the conversation is quite different. It actually looks like this:

Concepts

In Mastodon's API, a post is called a status.

Every status on Mastodon has an ID. This is usually a Snowflake ID which is represented as a number.

When someone replies to a status on Mastodon, they create a new status which has a field called in_reply_to_id. As its name suggests, has the ID of the status they are replying to.

Let's imagine this simple conversation:

1. Ada: "How are you?"
2. Bob: "I'm fine. And you?"
3. Ada: "Quite well, thank you!"

Message 2 is in reply to message 1. Message 3 is in reply to message 2.

In Mastodon's jargon, message 1 is the ancestor of message 2. Similarly, message 3 is the descendant of message 2.

  → Descendants →
1--------2-------3
   ← Ancestors ←

Branches

Now, let's imagine a more complicated conversation - one with branches!

1. Alice: What's your favourite pizza topping?
├── 2. Bette: Pineapple
│   ├── 4. Chuck: You make me sick!
│   └── 7. Dave: Yeah, I love pineapple too
└── 3. Chuck: Mushroom are the best
    ├── 5. Alice: Really?
    │   └── 6. Dave: Button mushrooms are best!
    └── 8. Elle: I like them too!

As you can see, people reply in threads. In this example, 2 is a different "branch" of the conversations than 3.

It looks a bit more complicated with hundreds of replies, but that's it! That's all you need to know!

API

If you want to download a single status with an ID of 1234 the API call is /api/v1/statuses/1234

If you want to download a conversation, it is a little bit more complicated. Mastdon's API calls a conversation a context

Let's take the above simple example - Ada and Bob speaking. Ada's first status has an ID of 1. To get the conversation, the API call is /api/v1/statuses/1/context

That returns two things:

• A list of ancestors. This is empty because 1 is the first status in this conversation.
• A list of descendants. This contains statuses 2 and 3.

You will note, the context does not return the status 1 itself.

Let's suppose that, instead of asking for the context of status 1, we instead asked for 2. This would return:

• A list of ancestors. This contains status 1.
• A list of descendants. This contains status 3.

What about if we asked for 3? This would return:

• A list of ancestors. This contains status 1 and 2
• A list of descendants. This is empty because 3 is the last message in this conversation.

Branches

When it comes to complex threads - like the pizza example - things become a bit more difficult. Let's see the example again:

1. Alice: What's your favourite pizza topping?
├── 2. Bette: Pineapple
│   ├── 4. Chuck: You make me sick!
│   └── 7. Dave: Yeah, I love pineapple too
└── 3. Chuck: Mushroom are the best
    ├── 5. Alice: Really?
    │   └── 6. Dave: Button mushrooms are best!
    └── 8. Elle: I like them too!

Suppose we ask for the context of the message with ID 5. This would return:

• A list of ancestors. This contains statuses 1 and 3
• A list of descendants. This contains status 6.

That's it!?!? Where are the rest? They are part of a different conversation branch. Even status 8 isn't returned because it's a reply to 3, not 5.

In order to get the full conversation, we need to be sneaky!

The list of ancestors contains the first message in the conversation. So we can grab that, and then call context again for its ID.

Let's dive into some Python code to see how it works.

Code

This uses the Mastodon.py library for calling the Mastodon API and the Python treelib to create a conversation tree data structure.

This code connects to Mastodon and receives the status for a single ID.

from mastodon import Mastodon
from treelib import Node, Tree

mastodon = Mastodon( api_base_url="https://mastodon.example", access_token="Your personal access token from your instance" )

status_id =  109348943537057532 
status = mastodon.status(status_id)

Getting the conversation means calling the context API:

conversation = mastodon.status_context(status_id)

⚠ Note: Calling the context on a large thread may take a long time. The longer the conversation, the longer you'll have to wait.

If there are ancestors, that means we are only on a single branch. The 0th ancestor is the top of the conversation tree. So let's get the context for that top status:

if len(conversation["ancestors"]) > 0 :
   status = conversation["ancestors"][0]
   status_id = status["id"]
   conversation = mastodon.status_context(status_id)

Next, we need to create a data structure to hold the conversation. We'll start by adding to it the first status in the conversation:

tree = Tree()

tree.create_node(status["uri"], status["id"])

Finally, we add any replies which are in the descendants. It is possible that some earlier statuses have been deleted. So we won't add any status which are replies to deleted statuses:

for status in conversation["descendants"] :
   try :
      tree.create_node(status["uri"], status["id"], parent=status["in_reply_to_id"])
   except :
      #  If a parent node is missing
      print("Problem adding node to the tree")

That's it! Let's show the tree:

tree.show()

Here's what it should look like:

2022-11-14 20:02 Edent: Today I was meant to be flying in to San Francisco to attend Twitter's Developer Conference - Chirp.Twitter had paid for my flights and hotel, because I was one of their developer insiders. I planned to spend the week meeting friends old and new.Instead, Alan the Hyperprat canceled the conference. So I'm staying in the UK.So I'm going to spend the week hacking on Mastdon's #API and building cool shit.  That'll show him!You can see what I'm working on at https://shkspr.mobi/blog/2022/11/building-an-on-this-day-service-for-mastodon/ https://mastodon.social/users/Edent/statuses/109343943300929632
├── 2022-11-14 20:10 Edent: Oh! And I was meant to be attending a Belle & Sebastian gig tonight. I canceled those tickets for I could fly to SF.So far, I reckon Alan's acquisition of Twitter has cost me close to £190.Wonder if he's good for the money? https://mastodon.social/users/Edent/statuses/109343972435801664
│   ├── 2022-11-14 20:14 thehodge: @Edent reminds me of the time I was booked to speak at a conference in Munich and I excitedly booked a behind the scenes tour of the worlds largest miniature city!Then the company went under!Gutted. https://mastodon.social/users/thehodge/statuses/109343989481494630
│   ├── 2022-11-14 21:16 Janiqueka: @Edent the way my bill for him keeps increasing https://mastodon.online/users/Janiqueka/statuses/109344233355230523
│   ├── 2022-11-14 21:19 henry: @Edent I was due to be at B&S tomorrow but it’s been postponed again.. not sure if that makes it better or worse for you! https://social.lc/users/henry/statuses/109344244402822729
│   │   └── 2022-11-15 04:53 Edent: @henry again!? Ah well!Hope you get to see them soon. https://mastodon.social/users/Edent/statuses/109346031194446940
│   ├── 2022-11-15 09:18 Amandafclark: @Edent send him an invoice :) https://mastodon.social/users/Amandafclark/statuses/109347071811426672
│   └── 2022-11-15 11:29 Edent: One of the #MastodonAPI projects I'm working on is a better way to view long & complex threads.You may have seen me build something similar for the other site a while ago - demo at https://shkspr.mobi/blog/2021/09/augmented-reality-twitter-conversations/ - so I'm hoping I can do something similarly interesting.Main limitation is getting *all* of the conversation threads. It looks like the context API isn't paginated. But I might be being thick. https://mastodon.social/users/Edent/statuses/109347587353822637
│       ├── 2022-11-15 11:36 bensb: @Edent Excellent project. You might have seen, but there's also this feature request for better 🧵 handling: https://github.com/mastodon/mastodon/issues/8615 https://genomic.social/users/bensb/statuses/109347612990393791
│       ├── 2022-11-15 11:39 Edent: Cor! That @katebevan is good for engagement! Look at all those conversations she's kicked off! https://mastodon.social/users/Edent/statuses/109347627634008550
│       │   ├── 2022-11-15 11:58 Edent: Indeed, how could they be?That means that ID of a reply is different depending on where you see it.So the ID of this post is:mastodon. social /@ edent/ 123456But when you see it on your server, it might appear as:your. server /@ edent/ 987654The #MastodonAPI copes with this really well. But it is a mite confusing to get one's head around. https://mastodon.social/users/Edent/statuses/109347703064222520
│       │   │   ├── 2022-11-15 12:02 erincandescent: @Edent the numeric IDs are not part of the protocol - it's all URL based. Pleroma uses UUIDs for example https://queer.af/users/erincandescent/statuses/109347716173491502
│       │   │   │   └── 2022-11-15 12:06 Edent: @erincandescent oh! That's interesting. Thanks. https://mastodon.social/users/Edent/statuses/109347734283971306

Once you have a tree, you can format the contents however you like.

Grab the code

You can download the code for my Mastodon API tools from CodeBerg. Enjoy!

The Twitter version never really worked properly because the Twitter API doesn't support searching for historic Tweets. What I had to do was manually build search queries like: ?q=from:edent (until:2011-11-15 since:2011-11-14) and redirect people to the website.

Eugh!

I'm trying to build something similar for the Mastodon social network. Yes, I know it is new to you - but some of us have been there for several years.

So here's how to search Mastodon for posts made on specific dates!

(Skip to the code and ignore all the exciting preamble.)

Sadly, the Mastodon Search API is still quite basic. It isn't possible to directly search by user or by data parameters.

If you download the archive of all your posts, you'll find an ActivityPub feed called outbox.json - which is a collection of everything you've ever posted.

It can be parsed using jq to get all of the statuses posted on a specific day:

cat outbox.json | jq ".orderedItems[] | select (.published | fromdateiso8601 > 1636533813) | select (.published | fromdateiso8601 < 1636620302)"

The fromdateiso8601 are the Unix epoch times from 365 days ago and 364 days ago.

So, conceptually, it's possible to build - as long as you're willing to download your data and manually parse it. Let's see if we can do a little better than that.

Building in Python

We're going to build this using Python3 and the Mastodon.py library.

Install the library on the command line with:

pip3 install -U Mastodon.py

Now, launch Python and load the library:

from mastodon import Mastodon

We'll need an API key. Go to the website of your Mastodon instance. In settings, there should be an option called "Development". Use that to create a new app which has the "Read" scope.

Once created, we will use the "Your access token" which will be a long string of random letters and numbers. In this example, we'll be using abc123. Your real access token will be longer!

Let's set up a connection to your Mastodon instance:

instance = "https://mastodon.example.com"
mastodon = Mastodon( api_base_url=instance, access_token="abc123" )

Next, we need your user ID. This isn't your @ name, instead it is the numerical ID assigned by the server. For this, we need the Verify account credentials API call:

mastodon.me()

That produces:

{
   'id': 7112,
   'username': 'Edent',
   'acct': 'Edent',
   'display_name': 'Terence Eden',
   ...

Looks like I was a pretty early adopter!

Getting the last 20 statuses is:

me = mastodon.me()
my_id = me["id"]
mastodon.account_statuses(id = my_id)

The problem is, that can only receive a maximum of 40 statuses at a time.

statuses = mastodon.account_statuses(id = my_id, limit="40")

We need to use Pagination. The API makes it pretty easy to grab the next page. There's also a call to get every post.

Be warned - this can take a long time. If you have thousands of posts it may take a few minutes. It can also quickly deplete your API rate limits. Use with caution!

We can reduce some of the load by excluding anything you've "boosted".

statuses = mastodon.account_statuses(id = my_id, limit="40", exclude_reblogs=True)
all_statuses = mastodon.fetch_remaining(statuses)

You can run len(all_statuses) to see how many you have retrieved.

The next step is finding all the posts which happened on a certain day each year. Let's say we want every post which happened on a 14th of February.

The library returns timestamps as Python DateTime objects.

status_date = all_statuses[1]["created_at"]

Shows

datetime.datetime(2016, 11, 1, 17, 4, 23, 842000, tzinfo=tzutc())

The datetime library is pretty handy. You can find the day using status_date.day and month with status_date.month.

This means we can loop through every status and show only the ones we care about.

for status in all_statuses:
     if (status["created_at"].day == 14 and status["created_at"].month == 2):
             print(status["uri"])

That will get you a list of URls which contain posts made on a specific day in previous years.

Putting it all together

from mastodon import Mastodon
instance = "https://mastodon.example.com"
mastodon = Mastodon( api_base_url=instance, access_token="abc123" )
me = mastodon.me()
my_id = me["id"]
mastodon.account_statuses(id = my_id)
statuses = mastodon.account_statuses(id = my_id, limit="40", exclude_reblogs=True)
all_statuses = mastodon.fetch_remaining(statuses)
for status in all_statuses:
   if (status["created_at"].day == 14 and status["created_at"].month == 2 and status["created_at"].year < 2022) :
      print(status["uri"])

Building a time machine

All of the above works, but is pretty inefficient because there's no way to search for specific timeframes on Mastodon. Or so I thought!

Gargron

@Eugen Rochko

@Edent Mastodon uses snowflake IDs, which means the IDs correspond to datetimes. You would need to look up how to construct the ID from a date, but you can use this to retrieve posts by date using min_id/max_id.

---

If we can calculate the maximum and minimum IDs for a given day, we will have a much more efficient search!

Let's dive in to the Mastodon Snowflake code. It is really well documented:

Our ID will be composed of the following: 6 bytes (48 bits) of millisecond-level timestamp 2 bytes (16 bits) of sequence data

OK! Let's look at a typical Mastodon Status ID mastodon.social/@Edent/109326536843609210, it was posted on 2022-11-11 at 18:16.

Let's take the ID 109326536843609210 and perform a bitwise shift on it.

print(109326536843609210 >> 16)
1668190564630

Hey! That looks a bit like a UNIX timestamp! The last three numbers will be the sequence, so we can eliminate them and see what happens if we convert it to a timestamp.

from datetime import datetime
datetime.fromtimestamp(1668190564630/1000)
datetime.datetime(2022, 11, 11, 18, 16, 4, 630000)

Nice! So we can go backward and take a date - say this time last year - and convert it to a maximum and minimum ID.

min_id = ( int( datetime(2022,11,11,00,00).timestamp() ) << 16 ) * 1000
max_id = ( int( datetime(2022,11,11,23,59).timestamp() ) << 16 ) * 1000

Which gives us 109322226892800000 and 109327885271040000 respectively.

Let's try that with the API!

Final Code

from datetime import datetime, timedelta
from mastodon import Mastodon

#  Set up access
instance = "https://mastodon.example"
mastodon = Mastodon( api_base_url=instance, access_token="abc123" )

#  Get user's info
me = mastodon.me()
my_id = me["id"]
year_joined = me["created_at"].year

#  Today's date
year_now  = datetime.now().year
month_now = datetime.now().month
day_now   = datetime.now().day

#  Counter
year_counter = year_now

#  Loop through previous years
#  Start with last year and go down until the user joined
while (year_counter >= year_joined ) :
   year_counter -= 1
   #  The end of today is the start of tomorrow
   #  This means yesterday can take into account leap-years
   today_end = datetime(year_counter, month_now, day_now, 00, 00) + timedelta(days=1)
   yesterday_end = today_end - timedelta(days=1)
   #  Bitwise shift the integer representation and convert to milliseconds
   max_id = ( int( today_end.timestamp() )     << 16 ) * 1000
   min_id = ( int( yesterday_end.timestamp() ) << 16 ) * 1000
   #  Call the API
   statuses = mastodon.account_statuses(id = my_id, max_id=max_id, min_id=min_id, limit="40", exclude_reblogs=True)
   #  Fetch further statuses if there are any
   all_statuses = mastodon.fetch_remaining(statuses)
   #  Print the date and URl
   for status in all_statuses :
      print( str(status["created_at"]) + " " + status["uri"] )

Next Steps

It works on my machine! But that's not really good enough. Ideally I'd like to turn this in to a web app which people could use with their own account.

If you're interested in helping out with that grab the code or drop me a line!

You can see it in action partway through this TikTok video.

@cambridgeuniversity

Visit the #Faust Shop and see what happens when you make a deal with your digital double. #Devil #EduTok #Cambridge #Performance #Philosophy

♬ original sound - Cambridge University

Here are some examples of the types of avatars I generated:

First off, how do generative avatars work? Obviously an artist hasn't drawn hundreds of thousands of unique pieces of art. Instead, they have created a few base elements and then use those to generate millions of combinations.

Here's a very quick guide to making very bad generative art.

Start with a random string

Those planet-burning NFTs are derived from a cryptographic hash of something. For example, the address of a cryptocurrency wallet or transaction. We're going to use something much simpler; the humble MD5 algorithm.

MD5 shouldn't be used for anything which needs to be kept safe and secure. But for our purposes, it generates 32 sufficiently-random hexadecimal values. For example d41d8cd98f00b204e9800998ecf8427e.

In our case, we take some data from the user - for example their email address - and then hash it with MD5 to produce 32 hexadecimal characters. Here's some code in PHP:

$hash  =  md5($email);
$characters = str_split($hash);

Extract meaning from noise

Let's take the above MD5 hash and get something useful out of it. For example, the first three characters could be the colour of the background #d41. The next three could be the colour of the avatar's skin #d8c. And so on.

//  A canvas to draw the avatar on
$canvas = imagecreatetruecolor(23, 23);
// Generated colours
$colour_background = imagecolorallocate($canvas,
    hexdec($characters[0].$characters[0]),
    hexdec($characters[1].$characters[1]),
    hexdec($characters[2].$characters[2]),
);

We can also use the characters as the basis for decision-making. For example, if the 7th character is greater than 5, the avatar will have a big nose. If it's less than 5, it will have a small nose.

if ( hexdec($characters[7]) > 5 ) {
    //  Big nose
    imagefilledrectangle($canvas, 10, 12, 10, 13, $colour_nose);
} else {
    imagefilledrectangle($canvas, 10, 12, 10, 12, $colour_nose);
}

We can also use that to decide whether the avatar should have a specific feature. For example, if the 8th character is odd then draw a halo, but don't draw anything if it is even.

Generate base elements

For this project, we didn't need lots of different elements. A few simple ones were enough to generate hundreds of thousands of combinations:

• Background colour
• Skin colour
• Nose colour
• Nose shape
• Lip colour
• Lip shape
• Eye colour
• Eye direction
• Decorations like halos, horns, glasses, and earrings
• Whether it is facing left or right

Limitations

There are a few problems with this approach. There is no guarantee that the background colour will be sufficiently different from the foreground colour. If the mouth is the same colour as the skin, it will disappear.

If the colours are broadly similar, they might be hard to visually distinguish. As in this example:

Because this was a quick project, there wasn't much time to create lots of varieties of decorations. For example, hair is always a rectangle - albeit one which varies in height.

Pixel art is an unsubtle medium. There's no way to add anything small - like zits or eyebrows - to a face which is only a few pixels wide.

Final thoughts

Cryptographic hash visualisations were first conceived, as far as I am aware, at the tail end of the last century. They have been a prominent feature of some PKI packages for over a decade.

Humans have difficulty recognising long strings of digits. But we're reasonably good at recognising images - especially faces. Using generative art to create visually distinct faces could be a reasonable way to help people validate whether they've previously encountered a cryptographic signature.

But, for now, this is just a simple implementation for generating something that looks and behaves like a basic NFT.

This post first appeared on the Faust-Shop website

At the time, they said:

organizations that sign up during the beta period will not ever have to pay for users accepted during that period (provided Google continues to offer the service).

Google still offers the service - since renamed G-Suite, then Workplace, and next week to be renamed Google Plus for Work Home Edition Beta. But they're now going to charge people for it. Fair enough I guess⸮ Of course, Google don't offer a personal plan or a family plan - you have to sign up to an ENTERPRISE PLAN FOR SRS BZNIZ. Which is about £50 per user per year.

So, here's a quick(ish) guide to transferring your domain to a free* @Gmail.com account. * For now. Perhaps they'll start charging tomorrow.

Note: Obviously the better solution is to leave Google and go to Zoho, ProtonMail, or some other company. But some of us are used to the Gmail app or don't have the patience to run our own email server.

Table of Contents

• Sign up for a new Gmail account
• Import your mail
  • Or export / import
• Stop sending mail to Google
• Catch All
• 1. Forward your email
• 2. Check your email
• Send email from Gmail using your domain
• Summing Up
• Fuck You Google

Sign up for a new Gmail account

Go to Gmail.com and sign up for a new account. If your domain was example.biz then I recommend signing up for example.biz@gmail.com. That'll be easy to remember.

Just go through the usual sign up process. Remember to pick a strong password and to turn on 2FA.

Import your mail

Google offers a mail import feature. You can find it in "Settings".

There is some documentation for importing mail

Turn on POP in your old email. Then give your new Gmail account your old account's username and password. For example your.address@example.biz as the username, the server will be pop.gmail.com with port 995. And then your old password.

Note: If you have 2-Factor-Authentication turned on, you will either need to turn it off, or set up an App Specific Password. That will let Gmail log into Gmail to get your Gmail.

This process will take a long time - mine took about 24 hours. Oh, and you will find that Gmail will mark a bunch of legitimate email as spam. Better double check everything!

You'll also probably want to mark everything as "read".

Google doesn't make this stuff easy!

Or export / import

Apparently I have 3.5GB of data in my email. You can download it all via Gmail Export - or you can export via the admin tools.

I went via Google Takeout - which also allowed me to grab anything in Google Docs.

Note: This can take a while - but it is useful to have a backup. It took a couple of hours to generate mine.

Google doesn't support importing a Gmail export. Because they hate you and want you to suffer.

There is a command-line tool for restoring an MBOX backup to Gmail. I've not tried it though.

Stop sending mail to Google

If you set up Google Apps For Your Domain - your MX Records will be directing all email to Gmail's servers. You'll need to change that. Here's where things get tricky.

Log in to your domain management console. I can't tell you how to do this. You will need to go to something like "Zone Editor" where you'll see something like this: You will need to change all of your mail related records so that they point to your web-host's mail provider. Again, I can't help you with this. Speak to your hosting provider to find the details you need.

Catch All

You will need to set up a "catch all" email address - this is sometimes called "default routing". You want any email to @example.biz to be received rather than rejected.

You may need to set up a new username and password. Keep these safe - you may need them later.

Now your email is being received by your host. You have two options

1. Forward your email

This is harder to set up, but I find it to be quicker at delivering email.

Find the email forwarding set up with your domain provider. Set it so that every email to your catch-all is redirected to example.biz@gmail.com

That means each email to your domain will go to your new gmail address.

2. Check your email

This is simpler, but it does mean that your email isn't always delivered straight away.

Back in Gmail, go to settings, and select "Import mail and contacts" Give Gmail your catch-all email's address and password. Google will periodically check your mail and move it to your new Gmail account.

OK! You can now receive whatever@example.biz via Gmail! But what about sending?

Send email from Gmail using your domain

Follow the guide to sending Gmail using an alias. Basically, give them the username and password for the catch-all email address you created.

This will send email via your domain provider's mailservers.

You may need to set up DMARC, SPF, and DKIM on your domain. This is outside the scope of this article and you should speak to your domain provider.

Summing Up

1. Register a new Gmail address
2. Import your old email into your new address
3. Remove the Gmail MX records on your domain
4. Add MX records to point to your host's mailserver
5. Set up a catch-all email address
6. Forward all email to Gmail
7. Set up an email alias which sends via your mail servers

Fuck You Google

Google could have implemented this as a one-click solution. "Hey, if you don't want to pay, click here to transfer to a personal Gmail account. You'll lose all these great workspace features, but keep your email address."

They didn't. Which is a pity.

If you can, switch your email completely away from Google. If you can't, I hope the above is useful.

This time, I'm taking a different Nook, and turning it into a magic gallery. Here's what it looks like in action:

Terence Eden is on Mastodon

@edent

Upcycled an old eReader into an art frame.
Displays a new black & white piece of art from Flickr every few minutes.
Full write-up this weekend, but pretty straightforward to do. pic.x.com/ttvrbhz3ee

---

Video

With the front light off:

With the front light on:

How it works

Every few minutes, it finds a black and white artistic image from Flickr and displays it. Simple!

Inspiration

Ambient home cinema by Tom Whitwell

Build

I've previously replaced the battery on an eInk Nook, so I know how easy they are to disassemble. But for this I wanted to keep the plastic frame on, so that I'd be able to attach a more æsthetically pleasing wooden picture frame.

Finding a picture frame of the right size was a little tricky. The physical dimension of the unit are ~170mm x ~140mm. But the screen is only ~125mm x ~93mm .

I used eFrame - they have a really handy website for custom sized frames and inserts. I ordered a mount for an 135mm x 103mm picture (they cut them slightly smaller but I didn't want any overhang) with 20mm top and bottom borders, and 35mm left and right borders. That gave me enough space to fit the Nook. But, as long as the aperture lets you see the eInk, and masks off the plastic frame, you should be OK.

I was also able to buy a frame without a glass front. As this is my first attempt, I went for simple and cheap wood - rather than something more ornate and artistic. Total cost including shipping was £25.

Attaching the frame

It is useful to be able to get to the main "N" button - and there needs to be a gap for the charging cable. I thought about cutting out a little bit of the frame but, in the end, it was easier to attach the Nook to the frame using velcro and blu-tak. Because of the weight of the Nook, I used some sturdy hooks for the wall mount.

Power

There wasn't enough room in the frame for a normal USB cable, so I used a right-angled micro-USB cable. With the page refreshing every 5 minutes between 0600-2300, it got about 3 days of battery life - less if it used the Glow-Light. So a permanent wired connection seemed sensible.

Background

On the surface this seems quite simple - get a bunch of images and display them. But there are some limitations.

The original Nook Simple Touch is, bless it, an old and slow device. I want to do the minimum amount of processing on it as possible. The screen resolution is only 600x800. I don't want to convert colour artwork to black and white - that will just look messy. I also want the image size to roughly match the screen - so things aren't shrunk down.

So, I want a list of art which meets the following criteria:

• Intended for monochrome display⁰
• Roughly 4:3 aspect ratio
• Proper art¹ by real artists²

I started by looking at the Open Data produced by various museums and art galleries. Unfortunately, none of them indicated whether an artwork was greyscale. So, I turned to Flickr!

Using the Flickr API

Sadly, the Flickr API has been neglected for the best part of a decade. So the documentation is grossly out of date.

Two of the undocumented Search API calls are styles and orientation. I use them to search for black and white images in landscape orientation. You can discover more by using the Flickr search website and opening the "Advanced" panel.

The rest of the API call is quite standard. A keyword to search for ("art") - or whatever you want. Sorted by "interestingness" - although popularity is also a good metric. With safe_search turned off - no prudes in our house. And an image size no wider that 800px - because serving a correctly scaled image means less work for the eReader.

The Code

It is almost embarrassing in its simplicity. It uses Sam Wilson's PHP library for Flickr. It grabs 100 images, and then picks a random one to serve. It is resized to fit in a bounding box of 800x600 and repaginated to centre the image. The image is rotated it 90 degrees and served over HTTP with an appropriate caching policy. Nothing else. Here it is³.

require_once 'vendor/autoload.php';
$flickr = new \Samwilson\PhpFlickr\PhpFlickr("123465789");

$search_terms = ["art", "artistic", "painting", "drawing"];
$search_rand = array_rand($search_terms, 1);

$sort_terms = ["date-posted-asc", "date-posted-desc", "date-taken-asc", "date-taken-desc", "interestingness-desc", "interestingness-asc", "relevance"];
$sort_rand = array_rand($sort_terms, 1);

$s = $flickr->photos()->search(["text"        => $search_terms[$search_rand],
                                "per_page"    => "100",
                                "styles"      => "blackandwhite",
                                "orientation" => "landscape",
                                "sort"        => $sort_terms[$sort_rand],
                                "safe_search" =>"3"
                               ]);

$number_of_photos = count($s["photo"]);

$random_photo = random_int(0, $number_of_photos - 1);

$p = $s["photo"][$random_photo];

$farm   = $p["farm"];
$server = $p["server"];
$id     = $p["id"];
$secret = $p["secret"];
$title  = $p["title"];

$image_url = "https://live.staticflickr.com/{$server}/{$id}_{$secret}_c.jpg";

$image = imagecreatefromjpeg( $image_url );

$cropped = imagecropauto( $image, IMG_CROP_DEFAULT );
if ( $cropped !== false ) { 
   imagedestroy($image);  
   $image = $cropped;
}

$x = imagesx( $image );
$y = imagesy( $image );

$width_ratio  = $x / 800;
$height_ratio = $y / 600;

// Scale
if ( $height_ratio >= $width_ratio ) {
   $new_height = ( 600 / $y ) * $x;
   $scaled = imagescale( $image, $new_height, 600 );
} else {
   $scaled = imagescale( $image, 800, -1 );
}

$x = imagesx( $scaled );
$y = imagesy( $scaled );


// Make a background canvas
$canvas = imagecreatetruecolor( 800, 600 );
$black  = imagecolorallocate( $canvas, 0, 0, 0 );
imagefilledrectangle( $canvas, 0, 0, 800, 600, $black );

$dst_x = ( ( 800 - $x ) / 2 );
$dst_y = ( ( 600 - $y ) / 2 );

imagecopy( $canvas, $scaled, $dst_x, $dst_y, 0, 0, $x, $y );

$rotated = imagerotate( $canvas, 90, 0 );

header('Content-type: image/jpeg');
header("Expires: on, 01 Jan 1970 00:00:00 GMT");
header("Last-Modified: " . gmdate("D, d M Y H:i:s") . " GMT");
header("Cache-Control: no-store, no-cache, must-revalidate");
header("Cache-Control: post-check=0, pre-check=0", false);
header("Pragma: no-cache");

imagejpeg( $rotated );

// Tidy Up
imagedestroy( $image );
imagedestroy( $scaled );
imagedestroy( $rotated );

Displaying

As I mentioned in my previous post about displaying web pages on the Nook - the browser is slow, old, and can't handle https connections.

So I used ElectricSign which is an old Android app. Give it a URL and it will display the contents on screen. It can be set to periodically refresh.

I've had it running for months on a different screen and it hasn't crashed.

That's it!

TODO

• Buy more cheap, 2nd hand eInk screens on eBay. Preferably larger and with higher DPI.
• Get some fancy frames.
• Start a pirate art museum.
• Monetise it by tracking how long people stare at a particular artwork.
• BitCoin?⁴

Thanks for reading

If you've enjoyed this blog post, you can buy me something from my wishlist or sponsor my GitHub. Find out more way to support this blog.

Generate the images

Getting the data is left as an exercise for the reader (sorry!). This loops through the data and generates a separate image for each TreeMap:

for(week in weeks) {
  weekly_data <- subset(file_data, Week == week)

  size <- sqrt(sum(weekly_data$Count)) / 2
  if (size < 40) {
    size <- 40
  }

  map <- ggplot(weekly_data, aes(area = Count, label = paste(Filetype,formatC(Count, big.mark=",") ,sep="\n"), subgroup = Category, fill=Category)) +
    geom_treemap(layout="fixed") +
    geom_treemap_text(colour = "white", place = "centre", grow = TRUE, layout="fixed")

  file_name <- paste("media/", weekly_data$Week[1], ".png", sep="")
  ggsave(file_name, map, width = size, height = size, units = "mm")
}

The width and height are proportionate the the square-root of the size of the data. Annoyingly, ggplot works in millimetres rather than pixels!

If images are too small, R throws an error of "Viewport has zero dimension(s)". So this sets a minimum size. This value was found using trial and error.

The layout is fixed, as per the documentation which keeps the order of the elements and their labels.

Resize and reorientate the images

Now I have a directory of images, each a different size. I want all of them to have the same size canvas and to be placed against the right-hand edge.

mogrify -gravity east -background white -extent 1750x1750 *.png 

That sets the "gravity" to the right - so the original image is centred vertically but is up against the right edge. The extent is the dimension of the new image.

Mogrify overwrites the original images.

Make a video

This is a lazy way to shove all the images into a video.

cat *.png | ffmpeg -f image2pipe -r 10 -vcodec png -i - -vcodec libx264 out.mp4

or

ffmpeg -framerate 8 -pattern_type glob -i '*.png' -c:v libx264 -r 30 -pix_fmt yuv420p out.mp4

Some websites will need further conversion as they have specific codec requirements.

That's it

It isn't the prettiest way to do things, but it seemed pretty effective. If you know of a more efficient - or more R-ish way to accomplish the same animation - please let me know.

So this is how to create a self-hosted Skill, using PHP. It runs on your own server and doesn't require any interaction.

The Skill

At a basic level, all your website has to do is spit out a piece of JSON for Alexa to read out.

//   Set the correct header for JSON data
header('Content-Type: application/json');
//   Set the response
$response = [
  "response" => [
    "outputSpeech" => [
      "type" => "PlainText",
      "text" => "I'm a little teapot"
    ]
  ]
];
echo json_encode($response);

That's it.

This is perfect for when you have a simple query - "Yo! Alexa! Bus time?" - one question, no parameters.

OK, you can make sure that the request genuinely came from Alexa, and do all sorts of certificate checks. But why bother? Totally unnecessary for a personal skill.

Obviously, you can do some programming make the text say $whatever.

That was easy, right?

The Amazon Side

OK kid, this is where it gets complicated.

For this you will need an Amazon developer account. Getting one is beyond the scope of this tutorial.

Go to https://developer.amazon.com/alexa/console/ask

Click the "Create Skill" button.

Give your skill a name. It defaults to USA! USA! USA! English - even though Amazon knows your location and the location of your Echo. If you think AI is hard, that ain't nothing compared to localisation.

You need to select a Custom skill and to "Provision your own endpoint".

If you try to use AWS Lambdas you'll be in for a world of pain. Don't even bother - they are a nightmare for a beginner to use.

Now scroll all the way back to the top of the website to click the "Create Skill" button. You'd think that after filling in a form, the "next" button would be at the bottom. But that's too easy. If you think AI is hard, that ain't nothing compared to designing an easy to use form.

We're going to choose the "Start From Scratch" template. This has nothing to do with the Scratch programming language. Which is a shame, because Scratch is a great language and well designed for complex voice interaction models.

Before doing anything else, set up your "endpoint". This is the website where you uploaded your PHP file. Your website needs to support https - you may need to change the certificate type on this page if Amazon has difficulties with it. Hit "Save Endpoints" - again, it is at the top of the screen. Because people naturally work from the bottom of the form to the top of the form.

Next up, we need a default intent. It doesn't really matter what you put in here. This is not a complex skill which can take multiple routes. We just want something to happen whenever it is launched.

Because Alexa isn't really an AI - we have to give lots of "sample" utterances. So, if you want bus times, you might add "next bus", "buses", "when is my bus" - and every variation you can think of. Now hit "Build Model" at the top. Once that's done, go to test. Set the dropdown to "Development".

That's it! You're finally done! You can test the skill by either typing "whatever" into the text box, or by saying "Alexa! Whatever."

Alexa will read out whatever is in your PHP script.

I wish it were easier. Why can't I just say "When I ask this question, reply with the data from that website." Sites like IFTTT and Zapier make it so easy to create services - I wish there was a Voice Assistant which was easy to make skills for.

I have a bunch of old images which don't have any timestamp associated with them. This quick Python script will add a DateTime EXIF metadata tag to an image.

This uses piexif which can be installed using

pip install piexif

This simple script reads a photo, adds a timestamp, then saves a copy of the new photo.

from PIL import Image
from PIL.ExifTags import TAGS
import piexif
from datetime import datetime

im = Image.open("test.jpg")
exif_dict = piexif.load(im.info["exif"])

exif_dict["0th"][piexif.ImageIFD.DateTime]=datetime.strptime("2000/12/25 12:32","%Y/%m/%d %H:%M").strftime("%Y:%m:%d %H:%M:%S")
exif_bytes = piexif.dump(exif_dict)
im.save("new.jpg", "jpeg", exif=exif_bytes, quality="keep", optimize=True)

A point to note about the last line. By default, PIL saves JPGs with a quality of 75. By using quality="keep" the original quality is preserved. The optimize=True attempts to losslessly improve the image.

A word of warning - timestamps are stored in multiple EXIF locations. If your image already has a timestamp, it is possible that this script won't change it.

Everything here takes place in the Terminal on the Command Line. This should work on Windows and Mac - but I'm using Linux.

Get Python

Open your terminal or command prompt. Type python

You should see something like:

Python 2.7.9 (default, Dec 28 2016, 18:26:44)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

If you do, you have Python installed! Press [ctrl] and d to exit.

If you get an error, you will need to install Python

Install Tweepy

Tweepy is a library which makes it easy to use the Twitter API.

To install it, type:

pip install tweepy

Get API Keys

You will need to visit https://apps.twitter.com/ and follow Twitter's instructions for creating an app.

You will end up with a "Consumer Key" and a "Consumer Secret"

These are the secret tokens which belong to your app. What we need to do next is to sign in to the app and generate your personal tokens.

Get Tokens

• You can save this piece of code as oauth.py
• You run it as python oauth.py

import tweepy

consumer_key = "YOUR CONSUMER KEY HERE"
consumer_secret = "YOUR CONSUMER SECRET HERE"

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.secure = True
auth_url = auth.get_authorization_url()
print 'Visit this URL and authorise the app to use your Twitter account: ' + auth_url
verifier = raw_input('Type in the generated PIN: ').strip()
auth.get_access_token(verifier)

# Print out the information for the user
print "consumer_key        = '%s'" % consumer_key
print "consumer_secret     = '%s'" % consumer_secret
print "access_token        = '%s'" % auth.access_token
print "access_token_secret = '%s'" % auth.access_token_secret

• It will print a URL, open that URL in your browser and it will take you to Twitter.
• Log in to Twitter with your account.
• Twitter will show you a PIN - 6 numbers long.
• Type the PIN into the Command Line.
• The program will display all your API Tokens.

So, how do we use them?

Using your Twitter tokens

• Add your tokens to this code.
• You can save this piece of code as tweet.py
• You run it as python tweet.py
• It will Tweet a message on your behalf.

import tweepy

# Consumer keys and access tokens, used for OAuth
consumer_key        = 'YOUR CONSUMER KEY HERE'
consumer_secret     = 'YOUR CONSUMER SECRET HERE'
access_token        = 'YOUR ACCESS TOKEN HERE'
access_token_secret = 'YOUR ACCESS TOKEN SECRET HERE'

# OAuth process, using the keys and tokens
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

# Creation of the actual interface, using authentication
api = tweepy.API(auth)

# Send the tweet
message = 'This is a test'
api.update_status(message)

Get a list of all* the people you follow

• Add your tokens to this code.
• You can save this piece of code as following.py
• You run it as python following.py
• It will get the ID numbers of everyone* you follow and print them out.

import tweepy

# Consumer keys and access tokens, used for OAuth
consumer_key        = 'YOUR CONSUMER KEY HERE'
consumer_secret     = 'YOUR CONSUMER SECRET HERE'
access_token        = 'YOUR ACCESS TOKEN HERE'
access_token_secret = 'YOUR ACCESS TOKEN SECRET HERE'

# OAuth process, using the keys and tokens
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

# Creation of the actual interface, using authentication
api = tweepy.API(auth)

# Get all the people the user follows
friends = api.friends_ids()

# Print out each one
for id in friends:
    print(id)

* For technical reasons, this will only get the first 5,000 people you follow.

Let's save these to a file. Note the differences in this code and the previous piece of code.

import tweepy
import os

# Consumer keys and access tokens, used for OAuth
consumer_key        = 'YOUR CONSUMER KEY HERE'
consumer_secret     = 'YOUR CONSUMER SECRET HERE'
access_token        = 'YOUR ACCESS TOKEN HERE'
access_token_secret = 'YOUR ACCESS TOKEN SECRET HERE'

# OAuth process, using the keys and tokens
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

# Creation of the actual interface, using authentication
api = tweepy.API(auth)
friends = sorted(api.friends_ids())

friendsFile = open("friends.txt","w")

for id in friends:
    friendsFile.write(str(id) + "\n")

friendsFile.close()

You will now have a file called "friends.txt" which contains the ID Numbers of everyone you follow.

Handy for backing up your account :-)

Terence Eden is on Mastodon

@edent

Replying to @Suw@Suw @paul_clarke Hmmm. I could make a "following backer up thingamajig". Might be of some use. But would anyone use it *before* it happened?

---

Paul Clarke

@paul_clarke

Replying to @edent@edent @Suw I would. But I am on the edge.

---

This bash script moves a top level directory (and all the files and subdirectories under it), to a folder based on the (upper-case) version of the first character of the directory name.

#!/bin/bash
for dir in */ ; do
    start=${dir:0:1}
    mkdir -p ${start^^}
    mv "$dir" ${start^^}
done

Save that as sort.sh, make it executable with chmod +x sort.sh and run with ./sort.sh

It turns this:

.
├── Adella
├── adrianna
├── Barb
├── bebe
├── Carleen
├── catherina

Into:

.
├── A
│   ├── Adella
│   ├── adrianna
├── B
│   ├── Barb
│   └── bebe
├── C
    ├── Carleen
    ├── catherina

How it works

Let's break this down to see what everything does

• for dir in */ ; do
• This loops through the current directory and finds every item which ends with a / - that is, all directories

• start=${dir:0:1}
• Grab a sub-string from the directory name. In this case, the first character.

• mkdir -p ${start^^}
• Make a directory, use the -p flag to make sure we don't get an error when trying to create a duplicate directory. The ^^ command makes things upper case.

• mv "$dir" ${start^^}
• Move the directory we found into the newly created directory.

• done
• Finish the loop

Warnings

• Only works with folders - it won't move any files in your top level directory.
• Non-ASCII characters are supported. For example, 你好 is sorted into 你.
• Handles directories with spaces, punctuation, and numbers at the start.
• No doubt there is some horrific bug lurking in those half-dozen lines. Be a friend and drop me a note in the comments, would you?

But it is annoying for those of us who want something simple - like debug logs or notifications when scripts fail. Or something complex like conditional processing. Or even just a human to talk to when things break.

So, sod the lot of them. I'm slowing replacing my IFTTT "applets" with my own home-brew. Starting with...

RSS & Tumblr

I have a IFTTT recipe which

1. Reads an RSS feed
2. If there is a new item, queue it to Tumblr

Simple! And yet IFTTT randomly fails. It either can't read the RSS, or it doesn't spot new items, or it duplicates all my posts. Oh, and there's no way to set queued posts to automatically Tweet. It basically seems that IFTTT's MVP is certainly minimal, but sadly unviable.

This is a Python script which can be run as often as you like. It reads RSS, checks for new items, queues them, and sets Tweet text. NB - no Python 3 support yet because Tumblr can't be bothered to update their libraries.

Prerequisites

• Install pytumblr
• Install feedparser
• Create an app on Tumblr.
• Generate your Tumblr API tokens

The code

import pytumblr
import feedparser
import sys

id_file = "id_file"

# Authenticate via OAuth
tumblr = pytumblr.TumblrRestClient(
        'AAAAA',
        'BBBBB',
        'CCCCC',
        'DDDDD'
)

feed = feedparser.parse('https://EXAMPLE.COM/FEED.RSS')

#       Read the last id
with open(id_file,'r') as file_:
        last_id = file_.read().strip()

most_recent_id = feed['entries'][0]['id']

if (last_id == most_recent_id):
        # Do nothing
        sys.exit()
else:
        #       Save the new ID for later
        with open(id_file, 'w') as file_:
                file_.write(most_recent_id)

        #       Loop through the posts
        for post in feed['entries']:
                #       If the post has been encountered before, time to end things.
                if (last_id == post["id"]):
                        sys.exit()
                else:
                        #       Queue it on Tumblr
                        tumblr.create_text(
                                "YOUR-TUMBLR-NAME",
                                state="queue",
                                title=post["title"].encode("utf-8"),
                                format="html",
                                body=post["summary"].encode("utf-8")
                        )

Running

I have a crontab set up to run this a few times per day. By default, IFTTT seems to check every hour.

Caveats

• You will need a file called id_file to store the latest ID of the RSS feed.
• If your RSS feed doesn't have unique IDs, this may not work.
• This isn't exactly a robust, error-proof solution - but then, neither is IFTTT.

RAR is a silly and proprietary format. I prefer free software and I find that ZIP files are smaller and decompress faster. Not everyone agrees, and that's fine. Assuming you've downloaded a RAR file and want to convert it to ZIP, what's the easiest way?

Install P7Zip

sudo apt-get install p7zip-full p7zip-rar

Script

This basic bash script will

1. Extract a RAR example.rar file to a temporary directory on a RAM disk.
2. Recompress to example.zip.
3. Delete the temporary directory.

Easy!

It is based on this conversion from CBR to CBZ.

#!/bin/bash
echo "Converting RARs to ZIPs"

# Separate files using ␜ http://graphemica.com/%E2%90%9C.
IFS="␜"

# Use RAM disk for temporary files.
WORKDIR="/dev/shm/"

# Set name for the temp dir. This directory will be created under WORKDIR
TEMPDIR="rar2zip"

# Run using "./rar2zip.sh /full/path/to/files/"
# If no directory is specified, then use the current working directory (".").

if test -z $1; then
   SOURCEDIR=`pwd`
else
   SOURCEDIR="$1"
fi

echo "Using $SOURCEDIR"

# Create an temporary directory to work in.
cd $WORKDIR
mkdir $TEMPDIR
cd $TEMPDIR

# Find all the .rar files in the specified directory.
# Using -iname means it will find .rar .RAR .RaR etc.
# "-printf "%p␜" will cause the file names to be separated by the ␜ symbol,
# rather than the default newline.

for OLDFILE in `find $SOURCEDIR -iname "*.rar" -printf "%p␜"`; do

   # Get the file name without the extension
   BASENAME=`basename "${OLDFILE%.*}"`

   # Path for the file. The ".zip" file will be moved there.
   DIRNAME=`dirname $OLDFILE`

   # Name of the .zip file
   NEWNAME="$BASENAME.zip"

   # Create a temporary folder for unRARed files
   echo "Extracting $OLDFILE"
   mkdir "$BASENAME"
   7z x "$OLDFILE" -O"$BASENAME"
   cd "$BASENAME"

   # Zip the files with maximum compression
   7z a -tzip -mx=9 "$NEWNAME" *
   # Alternative. MUCH SLOWER, but better compression
   # 7z a -mm=Deflate -mfb=258 -mpass=15 -r "$NEWNAME" *

   # Move the new .zip to the directory containing the original ".rar" file
   mv "$NEWNAME" $DIRNAME/"$NEWNAME"

   # Delete the temporary directory
   cd $WORKDIR
   rm -r "$BASENAME"

   # OPTIONAL. Delete the RAR file
   # cd $DIRNAME
   # rm "$OLDFILE"

done

# Delete the temporary directory
cd $WORKDIR
rm -r $TEMPDIR

echo "Conversion Done"

That's it! I suppose it might be nice if there were a simpler way to do it - but this is fairly quick and error-proof.

Here is a quick way to add watermarks to photos and videos. All Linux command line based - so perfect if you've got a lot of images you want to manipulate.

Here is a delightful photo I've taken of a bee covered in pollen. I want to add a little copyright notice to it in order to discourage people using it without permission.

This command uses imagemagick's "annotate" option.

convert bee.jpg -gravity SouthEast -pointsize 16 -font TinyUnicode-Medium -fill "#fffdc3" -annotate +10+10 "(C) @edent" bee1.jpg

It produces this:

As you can see, a small watermark in the bottom right - specified using -gravity SouthEast. I've chosen a yellow colour for the font - in homage to the EURion anti-counterfeiting symbols.

The -annotate command contains the distance in pixels from the edges

For small text, I favour a Pixel Font like TinyUnicode - but feel free to choose one which suits your needs.

In this example, I position the message in the top left, with a larger font size, in pale blue, closer to the horizontal edge and further from the vertical edge.

convert bee.jpg -gravity NorthWest -pointsize 32 -font Helvetica -fill "#cee3f8" -annotate +1+50 "(C) @edent" bee2.jpg

Video

Adding a watermark to a video is more complex. This will require either ffmpeg - which isn't always installed by default on Linux - or avconv (I don't pretend to understand why ffmpeg and avconv split - but there we are).

I am going to overlay a transparent image on top of the video.

I've created this watermark image with a transparent background.

To overlay it, I use:

ffmpeg -i input.mp4 -i watermark.png -filter_complex "overlay=1500:1000" output.mp4

or for avconv's overlay:

avconv -i input.mp4 -i watermark.png -filter_complex 'overlay=x=main_w-overlay_w-10:y=main_h-overlay_h-10' output.mp4

ffmpeg's overlay= option allows me to specify where the top left of the image will appear on the video. So adjust those number based on the resolution of your watermark and of your video.

avconv has a more complex syntax. It's possible to specify the absolute position using overlay=x=1500:y=1000 or to use relative positions with overlay=x=main_w-overlay_w-10:y=main_h-overlay_h-10.

Or, for a one-liner:

ffmpeg -i input.mp4 -vf "drawtext=text='@edent':x=10:y=H-th-10:fontfile=/home/edent/.local/share/fonts/OCRAEXT.TTF:fontsize=15:fontcolor=#cee3f8" -vcodec h264 -strict -2 output.mp4

So, there you have it - hopefully simple ways to watermark your media files.

As part of my "Pirate Museum" I wanted to display 3D scans of statues using WebVR. The only problem is, these files are huge.

Take The Dancing Faun - at full resolution, that's around 230MB. Even on fast broadband that's a pain to download - and even on a fast computer it is slow to render. Once you add several models like that to a scene, everything grinds to a halt.

I wanted a simple way to turn a complex model into something simple - but still recognisable.

On the left you can see the face of the full sized model - on the right is it after being shrunk.

A surprising level of detail remains - and the filesize is a miniscule 1MB. Small enough for a floppy disk!

To do this, I used MeshLab - the Open Source 3D modelling software. Meshlab has a great GUI - but it can be slow to work with, especially if you're trying to change lots of models at once.

The first step is to create a filter script in the Meshlab GUI. This is what I settled on, a script to reduce any model to 32,000 faces.

<!DOCTYPE FilterScript>
<FilterScript>
 <filter name="Quadric Edge Collapse Decimation">
  <Param type="RichInt"   value="32000" name="TargetFaceNum"/>
  <Param type="RichFloat" value="0"     name="TargetPerc"/>
  <Param type="RichFloat" value="1"     name="QualityThr"/>
  <Param type="RichBool"  value="true"  name="PreserveBoundary"/>
  <Param type="RichFloat" value="1"     name="BoundaryWeight"/>
  <Param type="RichBool"  value="true"  name="PreserveNormal"/>
  <Param type="RichBool"  value="false" name="PreserveTopology"/>
  <Param type="RichBool"  value="true"  name="OptimalPlacement"/>
  <Param type="RichBool"  value="true"  name="PlanarQuadric"/>
  <Param type="RichBool"  value="false" name="QualityWeight"/>
  <Param type="RichBool"  value="true"  name="AutoClean"/>
  <Param type="RichBool"  value="false" name="Selected"/>
 </filter>
</FilterScript>

The files is then saved as 32k.mlx.

I followed Andrew Hazelden's blog post to run the script using Meshlab server.

meshlabserver -i original.obj -o small.obj -s 32k.mlx

After a few moments - depending on the speed of your kit and size of model - a reduced size version will be spat out.

It is important to note that the MeshLab work only changes the quality of the model - not its apparent size in 3D space.

Models can be permanently scaled using Obj-Magic. For example, to scale a model to be half its physical size, run: ./obj-magic -s 0.5 -o half.obj original.obj

Finally, we have the issue of rotating models in 3D space. Some models are oriented in surprising ways. This can be done with obj-magic if you already know how you want the model to be rotated. I found it easier to use Blender on some of the models to rotate them and re-centre them.

Blender is notoriously complicated - all you need to do is import the .obj. Select it in the right hand panel. Then select the Cube / Object menu. In there you'll find the controls to shrink and rotate the object.

Once done, select "Export" then "Wavefront (.obj)".

So, with that you can easily shrink 3D models to a more compact file size.

You can visit my Pirate Museum to experience the models.

Get The Fonts

If your computer doesn't have the requite font, install the latest version of Symbola.

Simply open up the .zip file, double click on the .ttf font, then choose "Install".

Find The Character

You almost certainly have the GNU CharMap app installed. If not, run apt-get install gucharmap.

You'll find it in "Accessories" or by running charmap from the terminal.

On the left hand side, you'll see a list of all the Unicode Block. Scroll down to "Emoticons":

Or "Miscellaneous Symbols"

Click on the character you want, then press CTRL+C to copy them to your clipboard.

You can also search through all the character names if you're not sure where it lives.

Enjoy!

😀😁😂😃😄😅😆😇😈😉😊😋😌😍😎😏

public class clickWidget extends AppWidgetProvider
{
@Override
public void onUpdate( Context context, AppWidgetManager appWidgetManager, int[] appWidgetIds )
{
RemoteViews remoteViews =
   new RemoteViews( context.getPackageName(), R.layout.widget );
remoteViews.setImageViewResource(R.id.ImageView01, drawableResourse);

ComponentName myWidget =
   new ComponentName( context, clickWidget.class );

// Create an Intent to launch Browser
Intent intent =
   new Intent(
      Intent.ACTION_VIEW, Uri.parse("http://example.com")
   );
PendingIntent pendingIntent =
   PendingIntent.getActivity(context, 0, intent, 0);

remoteViews.setOnClickPendingIntent(R.id.ImageView01, pendingIntent);
appWidgetManager.updateAppWidget( myWidget, remoteViews);
}

I've used this as the basis of a demo widget - "MI5 Terror Threat Level". The widget displays the UK's Threat Level on your homescreen. Clicking on it takes you to the MI5 page discussing the threat level.

The threat level is determined by parsing the RSS that the security services so helpfully provide. At the moment, the widget keeps a local copy of the graphics because the RSS feed contains references to "localhost" images.

You can download the widget by scanning in this QR code.

MI5 Widget - QR Code