VLC and YouTube will only play "Equirectangular" videos in spherical mode. So, how to convert a dual fisheye to equirectangualr?

The Simple Way

ffmpeg \
  -i original.mp4 \
  -vf "v360=input=dfisheye:output=equirect:ih_fov=189:iv_fov=189" \
  360.mp4

However, this has some "quirks".

The first part of the video filter is v360=input=dfisheye:output=equirect - that just says to use the 360 filter on an input which is dual fisheye and then output in equirectangular.

The next part is :ih_fov=189:iv_fov=189 which says that the input video has a horizontal and vertical field of view of 189°. That's a weird number, right?

You'd kind of expect each lens to be 180°, right? Here's what happens if :ih_fov=180:iv_fov=180 is used:

The lenses overlaps a little bit. So using 180° means that certain portions are duplicated.

I think the lenses technically offer 200°, but the physical casing prevents all of that from being viewed. I got to the value of 189° by trial and error. Mostly error! Using :ih_fov=189:iv_fov=189 get this image which has less overlap:

It isn't perfect - but it preserves most of the image coherence.

Cut Off Images

There's another thing worth noticing - the top, right, bottom, and left "corners" of the circle are cut off. If the image sensor captured everything, the resultant fisheye would look something like this:

I tried repaging the video to include the gaps, but it didn't make any noticeable difference.

Making Equirectangular Videos Work With VLC

Sadly, ffmpeg will not write the metadata necessary to let playback devices know the video is spherical. Instead, according to Bino3D, you have to use exiftool like so:

exiftool \
        -XMP-GSpherical:Spherical="true" \
        -XMP-GSpherical:Stitched="true" \
        -XMP-GSpherical:ProjectionType="equirectangular" \
        video.mp4

Putting It All Together

The LG 360 records audio in 5.1 surround using AAC. That's already fairly well compressed, so there's no point squashing it down to Opus.

The default video codec is h264, but the picture is going to be reprojected, so quality is always going to take a bit of a hit. Pick whichever code you like to give the best balance of quality, file size, and encoding time.

Run:

ffmpeg \
  -i original.mp4 \
  -vf "v360=input=dfisheye:output=equirect:ih_fov=189:iv_fov=189" \
  -c:v libx265 -preset fast -crf 28 -c:a copy \
  out.mp4; exiftool \
        -XMP-GSpherical:Spherical="true" \
        -XMP-GSpherical:Stitched="true" \
        -XMP-GSpherical:ProjectionType="equirectangular" \
        out.mp4

That will produce a reasonable equirectangular file suitable for viewing in VLC or in VR.

If this has been useful to you, please stick a comment in the box!

RSS Club is a series of posts which are only visible to RSS / Atom subscribers. Like you 😃

If you want this for your own WordPress site, here's what you'll need:

1. A blog post which is only visible in RSS / Atom.
2. Which has no HTML rendering on your site.
3. And cannot be found in your site's search.
4. Nor via search engines.
5. Also, doesn't appear on your mailing list.
6. Does not get shared or syndicated to the Fediverse.

(This is a bit more strict than the original rules which allow for web rendering and being found via a search engine.)

Start With A Category

The easiest way to do this in WordPress is via a category - not a tag.

After creating a category on your blog, click the edit link. You will see in the URl bar a tag_id.

Whenever you want to make an RSS-exclusive post, you select the category before you publish.

Disable Display

This code stops any page in the RSS Club category from being displayed on the web.

function rss_club_post_blocker(): void {
    if (    is_singular( "post" )
        &&  has_category( "rss-club" )
        && !current_user_can( "edit_posts" ) )
    {
        status_header( 403 );
        echo "You must be a member of RSS Club to view this content.";
        exit;
    }
}
add_action( "template_redirect", "rss_club_post_blocker" );

Editors can still see it, but everyone else gets a blocked message.

Remove From Site Search and SiteMap

Here's a snippet to stick in your functions.php - it removes the category from any queries unless it is for the admin pages or the RSS feeds.

//  Remove the RSS Club category from search results.
//  $query is passed by reference
function rss_club_search_filter( \WP_Query $query ): void {
    //  Ignore admin screens.
    if ( !is_admin() && !is_feed() ) {
        //  Find the RSS-Club category ID.
        $category = get_category_by_slug( "rss-club" );

        //  Remove it from the search results.
        if ( $category ) {
            $query->set( "category__not_in", [$category->term_id] );
        }       
    }
}
add_action( "pre_get_posts", "rss_club_search_filter" );

This code also redacts that category from the build-in sitemap. Note - the name of the category still shows up in the XML, but it leads to a 404.

Exclude From Email and Social Media RSS Feeds

My mailing list and social media posts are fed from RSS. So how do remove an entire category from an RSS feed?

Simple! Append ?cat=-1234 to the end!

A negative category ID will remove the category from being displayed. So my email subscribers won't see the RSS only content. Of course, they get email-only exclusive posts, so don't feel too bad for them 😊

Fediverse Exclusion

The manual way is easiest. Assuming you have the ActivityPub plugin and a the Share On Mastodon plugin, you can unselect the sharing options before publishing.

If you think you might forget to toggle those boxen, there is a filter for the share plugin:

function rss_club_mastodon_filter( bool $is_enabled, int $post_id ): bool {
    global $exclude;
    if ( has_category( $exclude, $post_id ) ) {
        return false;
    }
    return $is_enabled;
}
add_filter( "share_on_mastodon_enabled", "rss_club_mastodon_filter", 10, 2 );

Similarly, there's a filter for the ActivityPub plugin:

function rss_club_activitypub_filter( bool $disabled, \WP_Post $post ): bool 
{
    global $exclude;
    if ( has_category( $exclude, $post ) ) {
        return true;
    }

    return $disabled;
}
add_filter( "activitypub_is_post_disabled", "rss_club_activitypub_filter", 10, 2 );

Enjoy!

If you've set up your own RSS Club feed, drop me a line so I can subscribe 😊

There are a wide range of social media logins provided by Auth0 - including the usual suspects like Facebook, Twitter, WordPress, Discord, etc. Sadly, there's no support for Mastodon⁰.

All is not lost though. The Auth0 documentation says:

However, you can use Auth0’s Connections API to add any OAuth2 Authorization Server as an identity provider.

You can manually add a single Mastodon instance, but that doesn't work with the decentralised nature of the Fediverse. Instead, I've come up with a manual solution which works with any Mastodon server!

Background

Every Mastodon¹ server is independent. I have an account on mastodon.social you have an account on whatever.chaos. They are separate servers, albeit running similar software. A generic authenticator needs to work with all these servers. There's no point only allowing log ins from a single server.

Fortuitously, Mastodon allows app developers to automatically create new apps. A few simple lines of code and you will have an API key suitable for read-only access to that server. You can read how to instantly create Mastodon API keys or you can steal my PHP code.

User Experience

The user clicks the sign-in button on OpenBenches. They're taken to the Auth0 social login screen:

The user clicks on Mastodon. This is where Auth0's involvement ends!

The user is asked to provide the URl of their instance:

In the background, my server contacts the Mastodon instance and creates a read-only API key.

The user is asked to sign in to Mastodon.

The user is asked to authorise read-only access.

The user is now signed in and OpenBenches can retrieve their name, avatar image, and other useful information. Hurrah!

Auth0

Once you have created a service to generate API keys, it will need to run on a publicly accessible web server. For example https://example.com/mastodon_login.

Here's what you need to do within your Auth0 tennant:

• Authentication → Social → Create Connection
• At the bottom, choose "Create Custom".
• Choose "Authentication" only.
• Give your connection a name. This will be visible to users.
• "Authorization URL" and "Token URL" have the same value - the URl of your service.
• "Client ID" is only visible to you.
• "Client Secret" any random password; it won't be used for anything.
• Leave everything else in the default state.

It should look something like this:

Click the "Create" button and you're (almost) done.

Auth0 Icon

You will need to add a custom icon to the social integration. Annoyingly, there's no way to do it through the web interface, so follow that guide to use the command line.

Done!

I'll admit, this isn't the most straightforward thing to implement. Auth0 could make this easier - but it would still rely on users knowing the URl of their home instance.

That said, the Mastodon API is a delight to work with and the read-only permissions reduce risk for all parties.

To double-down on the annoyance, there's no simple way to turn them off. In part, that is due to the "WordPress Philosophy":

Decisions, not options

[…] Every time you give a user an option, you are asking them to make a decision. When a user doesn’t care or understand the option this ultimately leads to frustration.

I broadly agree with that. Having hundreds of options is a burden for users and a nightmare for maintainers. Do please read this excellent discussion from Tom McFarlin for a more detailed analysis.

But I want to turn things off. Luckily, there is a way. If you're a developer, you can remove a fair number of these "enforced" decisions. Add the following to your theme's functions.php file and watch the mandatory WordPress bloat whither away. I've commented each removal and, where possible, given a source for more information. Feel free to leave a comment suggesting how this script can be improved and simplified.

//  Remove mandatory classic theme.
function disable_classic_theme_styles() {
    wp_deregister_style( "classic-theme-styles" );
    wp_dequeue_style(    "classic-theme-styles" );
}
add_action( "wp_enqueue_scripts", "disable_classic_theme_styles" );

//  Remove WP Emoji.
//  http://www.denisbouquet.com/remove-wordpress-emoji-code/
remove_action( "wp_head",             "print_emoji_detection_script", 7 );
remove_action( "wp_print_styles",     "print_emoji_styles"              );
remove_action( "admin_print_scripts", "print_emoji_detection_script"    );
remove_action( "admin_print_styles",  "print_emoji_styles"              );
//  https://wordpress.org/support/topic/remove-the-new-dns-prefetch-code/
add_filter( "emoji_svg_url", "__return_false" );

//  Stop emoji replacement with images in RSS / Atom Feeds
//  https://danq.me/2023/09/04/wordpress-stop-emoji-images/
remove_filter( "the_content_feed", "wp_staticize_emoji" );
remove_filter( "comment_text_rss", "wp_staticize_emoji" );

//  Remove automatic formatting.
//  https://css-tricks.com/snippets/wordpress/disable-automatic-formatting/
remove_filter( "the_content",  "wptexturize" );
remove_filter( "the_excerpt",  "wptexturize" );
remove_filter( "comment_text", "wptexturize" );
remove_filter( "the_title",    "wptexturize" );

//  More formatting crap.
add_action("init", function() {
    remove_filter( "the_content", "convert_smilies", 20 );
    foreach ( array( "the_content", "the_title", "wp_title", "document_title" ) as $filter ) {
        remove_filter( $filter, "capital_P_dangit", 11 );
    }
    remove_filter( "comment_text", "capital_P_dangit", 31 );    //  No idea why this is separate
    remove_filter( "the_content",  "do_blocks", 9 );
}, 11);

//  Remove Gutenberg Styles.
//  https://wordpress.org/support/topic/how-to-disable-inline-styling-style-idglobal-styles-inline-css/
remove_action( "wp_enqueue_scripts", "wp_enqueue_global_styles" );

//  Remove Gutenberg editing widgets.
//  From https://wordpress.org/plugins/classic-widgets/
//  Disables the block editor from managing widgets in the Gutenberg plugin.
add_filter( "gutenberg_use_widgets_block_editor", "__return_false" );
//  Disables the block editor from managing widgets.
add_filter( "use_widgets_block_editor", "__return_false" );

//  Remove Gutenberg Block Library CSS from loading on the frontend.
//  https://smartwp.com/remove-gutenberg-css/
function remove_wp_block_library_css() {
    wp_dequeue_style( "wp-block-library"       );
    wp_dequeue_style( "wp-block-library-theme" );
    wp_dequeue_style( "wp-components"          );
}
add_action( "wp_enqueue_scripts", "remove_wp_block_library_css", 100 );

//  Remove hovercards on comment links in admin area.
//  https://wordpress.org/support/topic/how-to-disable-mshots-service/#post-12946617
add_filter( "akismet_enable_mshots", "__return_false" );

//  Remove Unused Plugin code.
function remove_plugin_css_js() {
    wp_dequeue_style( "image-sizes" );
}
add_action( "wp_enqueue_scripts", "remove_plugin_css_js", 100 );

//  Remove WordPress forced image size
//  https://core.trac.wordpress.org/ticket/62413#comment:40
add_filter( "wp_img_tag_add_auto_sizes", "__return_false" );

//  Remove <img> enhancements
//  https://developer.wordpress.org/reference/functions/wp_filter_content_tags/
remove_filter( "the_content",  "wp_filter_content_tags", 12 );

//  Stop rewriting http:// URls for the main domain.
//  https://developer.wordpress.org/reference/hooks/wp_should_replace_insecure_home_url/
remove_filter( "the_content", "wp_replace_insecure_home_url", 10 );

//  Remove the attachment stuff
//  https://developer.wordpress.org/news/2024/01/building-dynamic-block-based-attachment-templates-in-themes/
remove_filter( "the_content", "prepend_attachment" );

//  Remove the block filter
remove_filter( "the_content", "apply_block_hooks_to_content_from_post_object", 8 );

//  Remove browser check from Admin dashboard.
//  https://core.trac.wordpress.org/attachment/ticket/27626/disable-wp-check-browser-version.0.2.php
if ( !empty( $_SERVER["HTTP_USER_AGENT"] ) ) {
    add_filter( "pre_site_transient_browser_" . md5( $_SERVER["HTTP_USER_AGENT"] ), "__return_null" );
}

//  Remove shortlink.
//  https://stackoverflow.com/questions/42444063/disable-wordpress-short-links
remove_action( "wp_head", "wp_shortlink_wp_head" );

//  Remove RSD.
//  https://wpengineer.com/1438/wordpress-header/
remove_action( "wp_head", "rsd_link" );

//  Remove extra feed links.
//  https://developer.wordpress.org/reference/functions/feed_links/
add_filter( "feed_links_show_comments_feed", "__return_false" );
add_filter( "feed_links_show_posts_feed",    "__return_false" );

//  Remove api.w.org link.
//  https://wordpress.stackexchange.com/questions/211467/remove-json-api-links-in-header-html
remove_action( "wp_head", "rest_output_link_wp_head" );
//  https://wordpress.stackexchange.com/questions/211817/how-to-remove-rest-api-link-in-http-headers
//  https://developer.wordpress.org/reference/functions/rest_output_link_header/
remove_action( "template_redirect", "rest_output_link_header", 11, 0 );

You can find the latest version of my debloat script in my theme's repo.

If there are other things you find helpful to remove, or a better way to organise this file, please drop a comment in the box.

There are several services which allow you to get favicons based on a domain. But they all have their problems.

• Google
  • Exposes your user's to Google's tracking.
  • Relies on redirects.
• DuckDuckGo
  • Not officially supported by DDG.
• Favicon.is
  • No privacy policy whatsoever.
• Icons.horse
  • Paid service.
  • Only small size icons.
• Favicone
  • No privacy policy.
  • Only small size icons.

I want to show favicons next to specific links, but I don't want to expose my visitors to unnecessary tracking. How can I proxy these images so they are stored and served locally?

There are a few existing services. Some use Cloudflare workers or other cloud services, there are some local-first ones which are unmaintained. But nothing modern, self-hosted, and as easy to deploy as uploading a single PHP file.

So here's my attempt to make something which will preserve user privacy, be reasonably fast, and have moderately up-to-date icons, while remaining fast and efficient.

Getting the domain

Assuming the request comes in to https://proxy.example.com/?domain=bbc.co.uk

PHP has a handy FILTER_VALIDATE_DOMAIN filter which will determine if the string is a domain.

filter_var( $domain, FILTER_VALIDATE_DOMAIN, FILTER_FLAG_HOSTNAME );

Dealing with IDNs

Some domains contain non-ASCII characters - for example https://莎士比亚.org/ - not all favicon services support International Domain Names.

Using the idn_to_ascii() function, it is possible to get the Punycode domain.

$domain = idn_to_ascii("莎士比亚.org");

Getting the image

1. Check if the icon has previously been downloaded.
2. Rotate randomly between a few different Favicon services.
3. Download the icon.
4. Save it somewhere.

Getting the structure right

I know from my work on OpenBenches that storing tens of thousands of files in a single directory can be problematic. So I'll store the retrieved favicon in: /tld/domain/subdomain/

That will make it quick to see if an icon exists. I'll save the file with a filename based on the current timestamp. That will allow me to check if an icon is out of date, and will prevent people downloading the icons directly from me.

Preventing abuse

I don't want anyone but visitors to my site to be able to use this service. So I'll add a (weak) check to see if the request came from my domain.

$referer = parse_url( $_SERVER["HTTP_REFERER"], PHP_URL_HOST );
if ( $referer == "shkspr.mobi") {
   …
}

Some browsers may not send referers for privacy reasons. So they won't see the favicons. But they probably wouldn't have seen the images loaded from a 3^rd party service. So I'll serve a default image.

Putting it all together

You can grab the code from my personal git service.

There used to be a built-in settings tile on stock Android, and some manufacturers still have it, but Google's Pixels don't.

So here's how to make a (fairly) quick shortcut to swap between data SIMs.

First, get the brilliant open source Activity Manager app. It exposes all of the internal activities available in your apps. This lets you create a deep-link shortcut into a specific part of an app.

In Activity Manager, find the Settings app and tap on it.

Search or scroll down to "MobileNetworkListActivity".

Tap the vertical ellipse and then "Create Shortcut".

That will place an icon on your home screen. Tapping the icon will take you directly to your SIM settings. At the bottom is the option to choose your data SIM. Tap it to change your data SIM.

It isn't quite a one-tap solution, but the shortcut is a lot easier than remembering exactly which sub-menu you need to find.

Shotwell stores most of its information in a database. Which I lost. Because I'm an idiot.

But a bunch of metadata is also stored in the image's EXIF metadata!

Most importantly is the "Original File Name" which should become the "Description" in DigiKam. Unfortunately, there's no way to copy those values automatically on import.

So here's a one-liner which will read the "Original File Name" and store it in the "Title" EXIF - ready for DigiKam to parse!

exiftool "-XMP-dc:Title<XMP-getty:OriginalFileName" whatever.jpg

If you want to make sure any existing Title isn't overwritten, use:

exiftool "-XMP-dc:Title<${XMP-getty:OriginalFileName}" -if "not defined $XMP-dc:Title" whatever.jpg

Finally, to do it recursively, across all files:

exiftool -r "-XMP-dc:Title<${XMP-getty:OriginalFileName}" -if "not defined $XMP-dc:Title" /path/to/images

This is perfect for academics wishing to preserve Tweets, journalists wanting to download evidence, or simply embedding content without leaking user data back to Twitter.

tl;dr

You can get the full JSON code of any Tweet by using this API:

https://cdn.syndication.twimg.com/tweet-result?id=123456789&token=01010101010

Add any valid Twitter id, and choose a random number for your token. Done.

Background

Twitter has an "embed" functionality. Websites can import a full copy of a Tweet, including its media and metadata. Twitter's documentation is a little lacklustre but here's a brief explanation of how it works.

Embed Code

Using HTML like this:

<iframe
   src="https://platform.twitter.com/embed/Tweet.html?id=719484841172054016"
   width=512
   height=768></iframe>

Produces an embeddable which looks like this:

API Call

With a bit of sniffing of the traffic, it's possible to see that the iframe eventually calls a URl like this:

https://cdn.syndication.twimg.com/tweet-result?id=719484841172054016&token=123

Visit that and you'll see the JSON code of a Tweet.

Options

• id= this is the numeric ID of the Tweet.
• token= this is the API token. It can be set to a random number. It isn't checked.
• There's an optional lang= which takes BCP47 language codes. For example lang=en or lang=zh. However, they don't seem to make any difference to the output.

Output

Here's the JSON of the above Tweet. As you can see, it includes metadata on the number of replies, favourites, and retweets. There are entities, fully expanded links, and media in a variety of formats. There's also information on whether the post has been edited, if the user is stupid enough to pay for a blue-tick, and the language of the message.

Tweet With Image

{
    "__typename": "Tweet",
    "lang": "en",
    "favorite_count": 4,
    "possibly_sensitive": false,
    "created_at": "2016-04-11T11:18:48.000Z",
    "display_text_range": [
        0,
        120
    ],
    "entities": {
        "hashtags": [],
        "urls": [],
        "user_mentions": [
            {
                "id_str": "23937508",
                "indices": [
                    20,
                    30
                ],
                "name": "BBC Radio 4",
                "screen_name": "BBCRadio4"
            }
        ],
        "symbols": [],
        "media": [
            {
                "display_url": "pic.x.com/6F3ZSiWuIn",
                "expanded_url": "https://x.com/edent/status/719484841172054016/photo/1",
                "indices": [
                    97,
                    120
                ],
                "url": "https://t.co/6F3ZSiWuIn"
            }
        ]
    },
    "id_str": "719484841172054016",
    "text": "Warning! I'll be on @BBCRadio4's You And Yours shortly.\nPlease tune your wirelesses accordingly. https://t.co/6F3ZSiWuIn",
    "user": {
        "id_str": "14054507",
        "name": "Terence Eden is on Mastodon",
        "profile_image_url_https": "https://pbs.twimg.com/profile_images/1623225628530016260/SW0HsKjP_normal.jpg",
        "screen_name": "edent",
        "verified": false,
        "is_blue_verified": false,
        "profile_image_shape": "Circle"
    },
    "edit_control": {
        "edit_tweet_ids": [
            "719484841172054016"
        ],
        "editable_until_msecs": "1460375328174",
        "is_edit_eligible": true,
        "edits_remaining": "5"
    },
    "mediaDetails": [
        {
            "display_url": "pic.x.com/6F3ZSiWuIn",
            "expanded_url": "https://x.com/edent/status/719484841172054016/photo/1",
            "ext_media_availability": {
                "status": "Available"
            },
            "indices": [
                97,
                120
            ],
            "media_url_https": "https://pbs.twimg.com/media/CfwfpnJWwAEXwe3.jpg",
            "original_info": {
                "height": 1280,
                "width": 960,
                "focus_rects": []
            },
            "sizes": {
                "large": {
                    "h": 1280,
                    "resize": "fit",
                    "w": 960
                },
                "medium": {
                    "h": 1200,
                    "resize": "fit",
                    "w": 900
                },
                "small": {
                    "h": 680,
                    "resize": "fit",
                    "w": 510
                },
                "thumb": {
                    "h": 150,
                    "resize": "crop",
                    "w": 150
                }
            },
            "type": "photo",
            "url": "https://t.co/6F3ZSiWuIn"
        }
    ],
    "photos": [
        {
            "backgroundColor": {
                "red": 204,
                "green": 214,
                "blue": 221
            },
            "cropCandidates": [],
            "expandedUrl": "https://x.com/edent/status/719484841172054016/photo/1",
            "url": "https://pbs.twimg.com/media/CfwfpnJWwAEXwe3.jpg",
            "width": 960,
            "height": 1280
        }
    ],
    "conversation_count": 1,
    "news_action_type": "conversation",
    "isEdited": false,
    "isStaleEdit": false
}

Replies

Here's a more complicated example. This Tweet is in reply to another Tweet - so both messages are included:

{
    "__typename": "Tweet",
    "in_reply_to_screen_name": "edent",
    "in_reply_to_status_id_str": "1095653997644574720",
    "in_reply_to_user_id_str": "14054507",
    "lang": "en",
    "favorite_count": 0,
    "created_at": "2019-02-13T12:22:59.000Z",
    "display_text_range": [
        7,
        252
    ],
    "entities": {
        "hashtags": [],
        "urls": [],
        "user_mentions": [
            {
                "id_str": "14054507",
                "indices": [
                    0,
                    6
                ],
                "name": "Terence Eden is on Mastodon",
                "screen_name": "edent"
            }
        ],
        "symbols": []
    },
    "id_str": "1095659600420966400",
    "text": "@edent I can definitely see how this would get in the way of making your day a productive one. Do you find this happens often? If it does, I'd be happy to chat to you about a reliable alternative with us during your lunch break! ☕ PM me for a chat! ^JH",
    "user": {
        "id_str": "20139563",
        "name": "Sky",
        "profile_image_url_https": "https://pbs.twimg.com/profile_images/1674689671006240769/OpfisqRG_normal.jpg",
        "screen_name": "SkyUK",
        "verified": false,
        "verified_type": "Business",
        "is_blue_verified": false,
        "profile_image_shape": "Square"
    },
    "edit_control": {
        "edit_tweet_ids": [
            "1095659600420966400"
        ],
        "editable_until_msecs": "1550062379768",
        "is_edit_eligible": true,
        "edits_remaining": "5"
    },
    "conversation_count": 2,
    "news_action_type": "conversation",
    "parent": {
        "lang": "en",
        "reply_count": 2,
        "retweet_count": 1,
        "favorite_count": 1,
        "possibly_sensitive": false,
        "created_at": "2019-02-13T12:00:43.000Z",
        "display_text_range": [
            0,
            112
        ],
        "entities": {
            "hashtags": [],
            "urls": [],
            "user_mentions": [
                {
                    "id_str": "17872077",
                    "indices": [
                        33,
                        45
                    ],
                    "name": "Virgin Media ❤️",
                    "screen_name": "virginmedia"
                }
            ],
            "symbols": [],
            "media": [
                {
                    "display_url": "pic.x.com/mje6nh38CZ",
                    "expanded_url": "https://x.com/edent/status/1095653997644574720/photo/1",
                    "indices": [
                        113,
                        136
                    ],
                    "url": "https://t.co/mje6nh38CZ"
                }
            ]
        },
        "id_str": "1095653997644574720",
        "text": "Working from home is tricky when @virginmedia goes down so hard even its status page falls over.\nTime for lunch. https://t.co/mje6nh38CZ",
        "user": {
            "id_str": "14054507",
            "name": "Terence Eden is on Mastodon",
            "profile_image_url_https": "https://pbs.twimg.com/profile_images/1623225628530016260/SW0HsKjP_normal.jpg",
            "screen_name": "edent",
            "verified": false,
            "is_blue_verified": false,
            "profile_image_shape": "Circle"
        },
        "edit_control": {
            "edit_tweet_ids": [
                "1095653997644574720"
            ],
            "editable_until_msecs": "1550061043962",
            "is_edit_eligible": true,
            "edits_remaining": "5"
        },
        "mediaDetails": [
            {
                "display_url": "pic.x.com/mje6nh38CZ",
                "expanded_url": "https://x.com/edent/status/1095653997644574720/photo/1",
                "ext_alt_text": "Oops! something's broken! ",
                "ext_media_availability": {
                    "status": "Available"
                },
                "indices": [
                    113,
                    136
                ],
                "media_url_https": "https://pbs.twimg.com/media/DzSLf6sWsAAGWWH.jpg",
                "original_info": {
                    "height": 797,
                    "width": 1080,
                    "focus_rects": [
                        {
                            "x": 0,
                            "y": 192,
                            "w": 1080,
                            "h": 605
                        },
                        {
                            "x": 142,
                            "y": 0,
                            "w": 797,
                            "h": 797
                        },
                        {
                            "x": 191,
                            "y": 0,
                            "w": 699,
                            "h": 797
                        },
                        {
                            "x": 341,
                            "y": 0,
                            "w": 399,
                            "h": 797
                        },
                        {
                            "x": 0,
                            "y": 0,
                            "w": 1080,
                            "h": 797
                        }
                    ]
                },
                "sizes": {
                    "large": {
                        "h": 797,
                        "resize": "fit",
                        "w": 1080
                    },
                    "medium": {
                        "h": 797,
                        "resize": "fit",
                        "w": 1080
                    },
                    "small": {
                        "h": 502,
                        "resize": "fit",
                        "w": 680
                    },
                    "thumb": {
                        "h": 150,
                        "resize": "crop",
                        "w": 150
                    }
                },
                "type": "photo",
                "url": "https://t.co/mje6nh38CZ"
            }
        ],
        "photos": [
            {
                "accessibilityLabel": "Oops! something's broken! ",
                "backgroundColor": {
                    "red": 204,
                    "green": 214,
                    "blue": 221
                },
                "cropCandidates": [
                    {
                        "x": 0,
                        "y": 192,
                        "w": 1080,
                        "h": 605
                    },
                    {
                        "x": 142,
                        "y": 0,
                        "w": 797,
                        "h": 797
                    },
                    {
                        "x": 191,
                        "y": 0,
                        "w": 699,
                        "h": 797
                    },
                    {
                        "x": 341,
                        "y": 0,
                        "w": 399,
                        "h": 797
                    },
                    {
                        "x": 0,
                        "y": 0,
                        "w": 1080,
                        "h": 797
                    }
                ],
                "expandedUrl": "https://x.com/edent/status/1095653997644574720/photo/1",
                "url": "https://pbs.twimg.com/media/DzSLf6sWsAAGWWH.jpg",
                "width": 1080,
                "height": 797
            }
        ],
        "isEdited": false,
        "isStaleEdit": false
    },
    "isEdited": false,
    "isStaleEdit": false
}

Quote Tweets

Here's an example where I have quoted a Tweet:

{
    "__typename": "Tweet",
    "lang": "en",
    "favorite_count": 9,
    "possibly_sensitive": false,
    "created_at": "2022-08-19T13:36:44.000Z",
    "display_text_range": [
        0,
        182
    ],
    "entities": {
        "hashtags": [],
        "urls": [
            {
                "display_url": "gu.com",
                "expanded_url": "http://gu.com",
                "indices": [
                    17,
                    40
                ],
                "url": "https://t.co/Skj7FB7Tyt"
            }
        ],
        "user_mentions": [],
        "symbols": []
    },
    "id_str": "1560621791470448642",
    "text": "Whoever buys the https://t.co/Skj7FB7Tyt domain will effectively get to rewrite history.\nThey can redirect links like these - and change the nature of the content being commented on.",
    "user": {
        "id_str": "14054507",
        "name": "Terence Eden is on Mastodon",
        "profile_image_url_https": "https://pbs.twimg.com/profile_images/1623225628530016260/SW0HsKjP_normal.jpg",
        "screen_name": "edent",
        "verified": false,
        "is_blue_verified": false,
        "profile_image_shape": "Circle"
    },
    "edit_control": {
        "edit_tweet_ids": [
            "1560621791470448642"
        ],
        "editable_until_msecs": "1660918004000",
        "is_edit_eligible": true,
        "edits_remaining": "5"
    },
    "conversation_count": 4,
    "news_action_type": "conversation",
    "quoted_tweet": {
        "lang": "en",
        "reply_count": 131,
        "retweet_count": 1337,
        "favorite_count": 2789,
        "possibly_sensitive": false,
        "created_at": "2018-11-27T15:56:19.000Z",
        "display_text_range": [
            0,
            279
        ],
        "entities": {
            "hashtags": [],
            "urls": [
                {
                    "display_url": "gu.com/p/axa7k/stw",
                    "expanded_url": "https://gu.com/p/axa7k/stw",
                    "indices": [
                        256,
                        279
                    ],
                    "url": "https://t.co/UulPL1CtcK"
                }
            ],
            "user_mentions": [],
            "symbols": []
        },
        "id_str": "1067447032363794432",
        "text": "The Steele Dossier asserted Russian hacking of the DNC was \"conducted with the full knowledge & support of Trump & senior members of his campaign.” Trump's war against the FBI & efforts to obstruct make sense if he thought they could prove it. https://t.co/UulPL1CtcK",
        "user": {
            "id_str": "548384458",
            "name": "Joyce Alene",
            "profile_image_url_https": "https://pbs.twimg.com/profile_images/952257848301498371/5s24RH-g_normal.jpg",
            "screen_name": "JoyceWhiteVance",
            "verified": false,
            "is_blue_verified": true,
            "profile_image_shape": "Circle"
        },
        "edit_control": {
            "edit_tweet_ids": [
                "1067447032363794432"
            ],
            "editable_until_msecs": "1543335979379",
            "is_edit_eligible": true,
            "edits_remaining": "5"
        },
        "isEdited": false,
        "isStaleEdit": false
    },
    "isEdited": false,
    "isStaleEdit": false
}

Downloading Media

Videos are also available to download, with no restrictions, in a variety of resolutions:

   "mediaDetails": [
        {
            "type": "video",
            "url": "https://t.co/Qw1IFom7Fh",
            "video_info": {
                "aspect_ratio": [
                    3,
                    4
                ],
                "duration_millis": 13578,
                "variants": [
                    {
                        "content_type": "application/x-mpegURL",
                        "url": "https://video.twimg.com/ext_tw_video/1432767873504718850/pu/pl/DiIKFNNZLWbLmECm.m3u8?tag=12"
                    },
                    {
                        "bitrate": 632000,
                        "content_type": "video/mp4",
                        "url": "https://video.twimg.com/ext_tw_video/1432767873504718850/pu/vid/320x426/oq2p-t0RJEEKuDD6.mp4?tag=12"
                    },
                    {
                        "bitrate": 950000,
                        "content_type": "video/mp4",
                        "url": "https://video.twimg.com/ext_tw_video/1432767873504718850/pu/vid/480x640/3X8ZsBmXmmaaakmM.mp4?tag=12"
                    },
                    {
                        "bitrate": 2176000,
                        "content_type": "video/mp4",
                        "url": "https://video.twimg.com/ext_tw_video/1432767873504718850/pu/vid/720x960/sS9cLdGn93eUmvKC.mp4?tag=12"
                    }
                ]
            }
        }
    ],

Other Examples

• Multiple Images
• Polls
• Deleted Message
• Summary Cards

Limitations

There are a few small limitations with this approach.

• It doesn't capture replies
  • If the Tweet is in reply to something, it will capture the parent.
  • If the Tweet quotes something, it will capture the quoted Tweet.
• The counts for replies, retweets, and favourites may not be accurate
  • Older messages seem worse for this, but that's a natural part of digital decay.
• Reduced metadata
  • The official API used to tell you which device was used to post the message, user's timezone, and other bits of useful information.
• You need to know the ID of the Tweet
  • There's no way to automatically grab every Tweet by a user, or from a search.
• Sometimes the API stops responding
  • Change the token to another random number.
• Occasionally replies and quotes won't be included
  • Calling the API again often recovers the data.

Python Code

If you're technically inclined, I've written some Python code to automate turning the JSON into HTML.

Have Fun

Remember, the owner of Twitter no longer believes in IP law. So I guess you can go nuts and download all of Twitter's data and use it for any purpose?

I've since re-written it to be faster and more stylistically correct.

It turns this:

<html lang="en-GB"><head><title id="something">Test</title></head><body><h1 class="top upper">Testing</h1><main><p>Some <em>HTML</em> and an <img src="example.png" alt="Alternate Text"></p>Text not in an element<ol><li>List</li><li>Another list</li></ol></main></body></html>

Into this:

<!doctype html>
<html lang=en-GB>
    <head>
        <title id=something>Test</title>
    </head>
    <body>
        <h1 class="top upper">Testing</h1>
        <main>
            <p>
                Some 
                <em>HTML</em>
                 and an 
                <img src=example.png alt="Alternate Text">
            </p>
            Text not in an element
            <ol>
                <li>List</li>
                <li>Another list</li>
            </ol>
        </main>
    </body>
</html>

I say it is "opinionated" because it does the following:

• Attributes are unquoted unless necessary.
• Every element is logically indented.
• Text content of CSS and JS is unaltered. No pretty-printing, minification, or checking for correctness.
• Text content of elements may have extra newlines and tabs. Browsers will tend to ignore multiple whitespaces unless the CSS tells them otherwise.
  • This fucks up <pre> blocks which contain markup.

It is primarily designed to make the markup easy to read. Because according to the experts:

A computer language is not just a way of getting a computer to perform operations but rather … it is a novel formal medium for expressing ideas about methodology. Thus, programs must be written for people to read, and only incidentally for machines to execute.

I'm fairly sure this all works properly. But feel free to argue in the comments or send me a pull request.

Here's how it works.

When is an element not an element? When it is a void!

Modern HTML has the concept of "Void Elements". Normally, something like <a> must eventually be followed by a closing </a>. But Void Elements don't need closing.

This keeps a list of elements which must not be explicitly closed.

$void_elements = [
    "area",
    "base",
    "br",
    "col",
    "embed",
    "hr",
    "img",
    "input",
    "link",
    "meta",
    "param",
    "source",
    "track",
    "wbr",
];

Tabs 🆚 Space

Tabs, obviously. Users can set their tab width to their personal preference and it won't get confused with semantically significant whitespace.

$indent_character = "\t";

Setting up the DOM

The new HTMLDocument should be broadly familiar to anyone who has used the previous one.

$html = '<html lang="en-GB"><head><title id="something">Test</title></head><body><h1 class="top upper">Testing</h1><main><p>Some <em>HTML</em> and an <img src="example.png" alt="Alternate Text"></p>Text not in an element<ol><li>List</li><li>Another list</li></ol></main></body></html>>'
$dom = Dom\HTMLDocument::createFromString( $html, LIBXML_NOERROR, "UTF-8" );

This automatically adds <head> and <body> elements. If you don't want that, use the LIBXML_HTML_NOIMPLIED flag:

$dom = Dom\HTMLDocument::createFromString( $html, LIBXML_NOERROR | LIBXML_HTML_NOIMPLIED, "UTF-8" );

To Quote or Not To Quote?

Traditionally, HTML attributes needed quotes:

<img src="example.png" class="avatar no-border" id="user-123">

Modern HTML allows those attributes to be unquoted as long as they don't contain ASCII Whitespace or certain other characters

For example, the above becomes:

<img src=example.png class="avatar no-border" id=user-123>

This function looks for the presence of those characters:

function value_unquoted( $haystack )
{
    //  Must not contain specific characters

    $needles = [ 
        //  https://infra.spec.whatwg.org/#ascii-whitespace
        "\t", "\n", "\f", "\n", " ", 
        //  https://html.spec.whatwg.org/multipage/syntax.html#unquoted 
        "\"", "'", "=", "<", ">", "`" ];
    foreach ( $needles as $needle )
    {
        if ( str_contains( $haystack, $needle ) )
        {
            return false;
        }
    }
    //  Must not be null
    if ( $haystack == null ) { return false; }
    return true;
}

Re-re-re-recursion

I've tried to document this as best I can.

It traverses the DOM tree, printing out correctly indented opening elements and their attributes. If there's text content, that's printed. If an element needs closing, that's printed with the appropriate indentation.

function serializeHTML( $node, $treeIndex = 0, $output = "")
{
    global $indent_character, $preserve_internal_whitespace, $void_elements;

    //  Manually add the doctype to start.
    if ( $output == "" ) {
        $output .= "<!doctype html>\n";
    }

    if( property_exists( $node, "localName" ) ) {
        //  This is an Element.

        //  Get all the Attributes (id, class, src, &c.).
        $attributes = "";
        if ( property_exists($node, "attributes")) {
            foreach( $node->attributes as $attribute ) {
                $value = $attribute->nodeValue;
                //  Only add " if the value contains specific characters.
                $quote = value_unquoted( $value ) ? "" : "\"";

                $attributes .= " {$attribute->nodeName}={$quote}{$value}{$quote}";
            }
        }

        //  Print the opening element and all attributes.
        $output .= "<{$node->localName}{$attributes}>";

    } else if( property_exists( $node, "nodeName" ) &&  $node->nodeName == "#comment" ) {
        //  Comment
        $output .= "<!-- {$node->textContent} -->";
    }

    //  Increase indent.
    $treeIndex++;
    $tabStart = "\n" . str_repeat( $indent_character, $treeIndex ); 
    $tabEnd   = "\n" . str_repeat( $indent_character, $treeIndex - 1);

    //  Does this node have children?
    if( property_exists( $node, "childElementCount" ) && $node->childElementCount > 0 ) {

        //  Loop through the children.
        $i=0;
        while( $childNode = $node->childNodes->item( $i++ ) ) {

            //  Is this a text node?
            if ($childNode->nodeType == 3 ) {
                //  Only print output if there's no HTML inside the content.
                //  Ignore Void Elements.
                if ( 
                      !str_contains( $childNode->textContent, "<" ) && 
                    property_exists( $childNode, "localName" ) && 
                          !in_array( $childNode->localName, $void_elements ) ) 
                {
                    $output .= $tabStart . $childNode->textContent;
                }
            } else {
                $output .= $tabStart;
            }

            //  Recursively indent all children.
            $output = serializeHTML( $childNode, $treeIndex, $output );
        };

        //  Suffix with a "\n" and a suitable number of "\t"s.
        $output .= "{$tabEnd}"; 

    } else if ( property_exists( $node, "childElementCount" ) && property_exists( $node, "innerHTML" ) ) {
        //  If there are no children and the node contains content, print the contents.
        $output .= $node->innerHTML;
    }

    //  Close the element, unless it is a void.
    if( property_exists( $node, "localName" ) && !in_array( $node->localName, $void_elements ) ) {
        $output .= "</{$node->localName}>";
    }

    //  Return a string of fully indented HTML.
    return $output;
}

Print it out

The serialized string hardcodes the <!doctype html> - which is probably fine. The full HTML is shown with:

echo serializeHTML( $dom->documentElement );

Next Steps

Please raise any issues on GitLab or leave a comment.

PHP 8.4 introduces a new Dom\HTMLDocument class it is a modern HTML5 replacement for the ageing XHTML based DOMDocument. You can read more about how it works - the short version is that it reads and correctly sanitises HTML and turns it into a nested object. Hurrah!

The one thing it doesn't do is pretty-printing. When you call $dom->saveHTML() it will output something like:

<html lang="en-GB"><head><title>Test</title></head><body><h1>Testing</h1><main><p>Some <em>HTML</em> and an <img src="example.png"></p><ol><li>List</li><li>Another list</li></ol></main></body></html>

Perfect for a computer to read, but slightly tricky for humans.

As was written by the sages:

A computer language is not just a way of getting a computer to perform operations but rather … it is a novel formal medium for expressing ideas about methodology. Thus, programs must be written for people to read, and only incidentally for machines to execute.

HTML is a programming language. Making markup easy to read for humans is a fine and noble goal. The aim is to turn the single line above into something like:

<html lang="en-GB">
    <head>
        <title>Test</title>
    </head>
    <body>
        <h1>Testing</h1>
        <main>
            <p>Some <em>HTML</em> and an <img src="example.png"></p>
            <ol>
                <li>List</li>
                <li>Another list</li>
            </ol>
        </main>
    </body>
</html>

Cor! That's much better!

I've cobbled together a script which is broadly accurate. There are a million-and-one edge cases and about twice as many personal preferences. This aims to be quick, simple, and basically fine. I am indebted to this random Chinese script and to html-pretty-min.

Step By Step

I'm going to walk through how everything works. This is as much for my benefit as for yours! This is beta code. It sorta-kinda-works for me. Think of it as a first pass at an attempt to prove that something can be done. Please don't use it in production!

Setting up the DOM

The new HTMLDocument should be broadly familiar to anyone who has used the previous one.

$html = '<html lang="en-GB"><head><title>Test</title></head><body><h1>Testing</h1><main><p>Some <em>HTML</em> and an <img src="example.png"></p><ol><li>List<li>Another list</body></html>'
$dom = Dom\HTMLDocument::createFromString( $html, LIBXML_NOERROR, "UTF-8" );

This automatically adds <head> and <body> elements. If you don't want that, use the LIBXML_HTML_NOIMPLIED flag:

$dom = Dom\HTMLDocument::createFromString( $html, LIBXML_NOERROR | LIBXML_HTML_NOIMPLIED, "UTF-8" );

Where not to indent

There are certain elements whose contents shouldn't be pretty-printed because it might change the meaning or layout of the text. For example, in a paragraph:

<p>
    Some 
    <em>
        HT
        <strong>M</strong>
        L
    </em>
</p>

I've picked these elements from text-level semantics and a few others which I consider sensible. Feel free to edit this list if you want.

$preserve_internal_whitespace = [
    "a", 
    "em", "strong", "small", 
    "s", "cite", "q", 
    "dfn", "abbr", 
    "ruby", "rt", "rp", 
    "data", "time", 
    "pre", "code", "var", "samp", "kbd", 
    "sub", "sup", 
    "b", "i", "mark", "u",
    "bdi", "bdo", 
    "span",
    "h1", "h2", "h3", "h4", "h5", "h6",
    "p",
    "li",
    "button", "form", "input", "label", "select", "textarea",
];

The function has an option to force indenting every time it encounters an element.

Tabs 🆚 Spaces

Tabs, obviously. Users can set their tab width to their personal preference and it won't get confused with semantically significant whitespace.

$indent_character = "\t";

Recursive Function

This function reads through each node in the HTML tree. If the node should be indented, the function inserts a new node with the requisite number of tabs before the existing node. It also adds a suffix node to indent the next line appropriately. It then goes through the node's children and recursively repeats the process.

This modifies the existing Document.

function prettyPrintHTML( $node, $treeIndex = 0, $forceWhitespace = false )
{    
    global $indent_character, $preserve_internal_whitespace;

    //  If this node contains content which shouldn't be separately indented
    //  And if whitespace is not forced
    if ( property_exists( $node, "localName" ) && in_array( $node->localName, $preserve_internal_whitespace ) && !$forceWhitespace ) {
        return;
    }

    //  Does this node have children?
    if( property_exists( $node, "childElementCount" ) && $node->childElementCount > 0 ) {
        //  Move in a step
        $treeIndex++;
        $tabStart = "\n" . str_repeat( $indent_character, $treeIndex ); 
        $tabEnd   = "\n" . str_repeat( $indent_character, $treeIndex - 1);

        //  Remove any existing indenting at the start of the line
        $node->innerHTML = trim($node->innerHTML);

        //  Loop through the children
        $i=0;

        while( $childNode = $node->childNodes->item( $i++ ) ) {
            //  Was the *previous* sibling a text-only node?
            //  If so, don't add a previous newline
            if ( $i > 0 ) {
                $olderSibling = $node->childNodes->item( $i-1 );

                if ( $olderSibling->nodeType == XML_TEXT_NODE  && !$forceWhitespace ) {
                    $i++;
                    continue;
                }
                $node->insertBefore( $node->ownerDocument->createTextNode( $tabStart ), $childNode );
            }
            $i++; 
            //  Recursively indent all children
            prettyPrintHTML( $childNode, $treeIndex, $forceWhitespace );
        };

        //  Suffix with a node which has "\n" and a suitable number of "\t"
        $node->appendChild( $node->ownerDocument->createTextNode( $tabEnd ) ); 
    }
}

Printing it out

First, call the function. This modifies the existing Document.

prettyPrintHTML( $dom->documentElement );

Then call the normal saveHtml() serialiser:

echo $dom->saveHTML();

Note - this does not print a <!doctype html> - you'll need to include that manually if you're intending to use the entire document.

Licence

I consider the above too trivial to licence - but you may treat it as MIT if that makes you happy.

Thoughts? Comments? Next steps?

I've not written any formal tests, nor have I measured its speed, there may be subtle-bugs, and catastrophic errors. I know it doesn't work well if the HTML is already indented. It mysteriously prints double newlines for some unfathomable reason.

I'd love to know if you find this useful. Please get involved on GitLab or drop a comment here.

Let's dive in!

Background

HTML has six levels of headings¹ - <h1> is the main heading for content, <h2> is a sub-heading, <h3> is a sub-sub-heading, and so on.

Together, they form a hierarchy.

Heading Example

HTML headings are expected to be used a bit like this (I've nested this example so you can see the hierarchy):

<h1>The Theory of Everything</h1>
   <h2>Experiments</h2>
      <h3>First attempt</h3>
      <h3>Second attempt</h3>
   <h2>Equipment</h2>
      <h3>Broken equipment</h3>
         <h4>Repaired equipment</h4>
      <h3>Working Equipment</h3>
…

What is the purpose of a table of contents?

Wayfinding. On a long document, it is useful to be able to see an overview of the contents and then immediately navigate to the desired location.

The ToC has to provide a hierarchical view of all the headings and then link to them.

Code

I'm running this as part of a WordPress plugin. You may need to adapt it for your own use.

Load the HTML

This uses PHP's DOMdocument. I've manually added a UTF-8 header so that Unicode is preserved. If your HTML already has that, you can remove the addition from the code.

//  Load it into a DOM for manipulation
$dom = new DOMDocument();
//  Suppress warnings about HTML errors
libxml_use_internal_errors( true );
//  Force UTF-8 support
$dom->loadHTML( "<!DOCTYPE html><html><head><meta charset=UTF-8></head><body>" . $content, LIBXML_NOERROR | LIBXML_NOWARNING );
libxml_clear_errors();

Using PHP 8.4

The latest version of PHP contains a better HTML-aware DOM. It can be used like this:

$dom = Dom\HTMLDocument::createFromString( $content, LIBXML_NOERROR , "UTF-8" );

Parse the HTML

It is not a good idea to use Regular Expressions to parse HTML - no matter how well-formed you think it is. Instead, use XPath to extract data from the DOM.

//  Parse with XPath
$xpath = new DOMXPath( $dom );

//  Look for all h* elements
$headings = $xpath->query( "//h1 | //h2 | //h3 | //h4 | //h5 | //h6" );

This produces an array with all the heading elements in the order they appear in the document.

PHP 8.4 querySelectorAll

Rather than using XPath, modern versions of PHP can use querySelectorAll:

$headings = $dom->querySelectorAll( "h1, h2, h3, h4, h5, h6" );

Recursive looping

This is a bit knotty. It produces a nested array of the elements, their id attributes, and text. The end result should be something like:

array (
  array (
    'text' => '<h2>Table of Contents</h2>',
    'raw' => true,
  ),
  array (
    'text' => 'The Theory of Everything',
    'id' => 'the-theory-of-everything',
    'children' => 
    array (
      array (
        'text' => 'Experiments',
        'id' => 'experiments',
        'children' => 
        array (
          array (
            'text' => 'First attempt',
            'id' => 'first-attempt',
          ),
          array (
            'text' => 'Second attempt',
            'id' => 'second-attempt',

The code is moderately complex, but I've commented it as best as I can.

//  Start an array to hold all the headings in a hierarchy
$root = [];
//  Add an h2 with the title
$root[] = [
    "text"     => "<h2>Table of Contents</h2>", 
    "raw"      => true, 
    "children" => []
];

// Stack to track current hierarchy level
$stack = [&$root]; 

//  Loop through the headings
foreach ($headings as $heading) {

    //  Get the information
    //  Expecting <h2 id="something">Text</h2>
    $element = $heading->nodeName;  //  e.g. h2, h3, h4, etc
    $text    = trim( $heading->textContent );   
    $id      = $heading->getAttribute( "id" );

    //  h2 becomes 2, h3 becomes 3 etc
    $level = (int) substr($element, 1);

    //  Get data from element
    $node = array( 
        "text"     => $text, 
        "id"       => $id , 
        "children" => [] 
    );

    //  Ensure there are no gaps in the heading hierarchy
    while ( count( $stack ) > $level ) {
        array_pop( $stack );
    }

    //  If a gap exists (e.g., h4 without an immediately preceding h3), create placeholders
    while ( count( $stack ) < $level ) {
        //  What's the last element in the stack?
        $stackSize = count( $stack );
        $lastIndex = count( $stack[ $stackSize - 1] ) - 1;
        if ($lastIndex < 0) {
            //  If there is no previous sibling, create a placeholder parent
            $stack[$stackSize - 1][] = [
                "text"     => "",   //  This could have some placeholder text to warn the user?
                "children" => []
            ];
            $stack[] = &$stack[count($stack) - 1][0]['children'];
        } else {
            $stack[] = &$stack[count($stack) - 1][$lastIndex]['children'];
        }
    }

    //  Add the node to the current level
    $stack[count($stack) - 1][] = $node;
    $stack[] = &$stack[count($stack) - 1][count($stack[count($stack) - 1]) - 1]['children'];
}

Missing content

The trickiest part of the above is dealing with missing elements in the hierarchy. If you're sure you don't ever skip from an <h3> to an <h6>, you can get rid of some of the code dealing with that edge case.

Converting to HTML

OK, there's a hierarchical array, how does it become HTML?

Again, a little bit of recursion:

function arrayToHTMLList( $array, $style = "ul" )
{
    $html = "";

    //  Loop through the array
    foreach( $array as $element ) {
        //  Get the data of this element
        $text     = $element["text"];
        $id       = $element["id"];
        $children = $element["children"];
        $raw      = $element["raw"] ?? false;

        if ( $raw ) {
            //  Add it to the HTML without adding an internal link
            $html .= "<li>{$text}";
        } else {
            //  Add it to the HTML
            $html .= "<li><a href=#{$id}>{$text}</a>";
        }

        //  If the element has children
        if ( sizeof( $children ) > 0 ) {
            //  Recursively add it to the HTML
            $html .=  "<{$style}>" . arrayToHTMLList( $children, $style ) . "</{$style}>";
        } 
    }

    return $html;
}

Semantic Correctness

Finally, what should a table of contents look like in HTML? There is no <toc> element, so what is most appropriate?

ePub Example

Modern eBooks use the ePub standard which is based on HTML. Here's how an ePub creates a ToC.

<nav role="doc-toc" epub:type="toc" id="toc">
<h2>Table of Contents</h2>
<ol>
  <li>
    <a href="s01.xhtml">A simple link</a>
  </li>
  …
</ol>
</nav>

The modern(ish) <nav> element!

The nav element represents a section of a page that links to other pages or to parts within the page: a section with navigation links. HTML Specification

But there's a slight wrinkle. The ePub example above use <ol> an ordered list. The HTML example in the spec uses <ul> an unordered list.

Which is right? Well, that depends on whether you think the contents on your page should be referred to in order or not. There is, however, a secret third way.

Split the difference with a menu

I decided to use the <menu> element for my navigation. It is semantically the same as <ul> but just feels a bit closer to what I expect from navigation. Feel free to argue with me in the comments.

Where should the heading go?

I've put the title of the list into the list itself. That's valid HTML and, if my understanding is correct, should announce itself as the title of the navigation element to screen-readers and the like.

Conclusion

I've used slightly more heading in this post than I would usually, but hopefully the Table of Contents at the top demonstrates how this works.

If you want to reuse this code, I consider it too trivial to licence. But, if it makes you happy, you can treat it as MIT.

Thoughts? Comments? Feedback? Drop a note in the box.

After some back-and-forth with the manufacturer, they agreed to send me their Android SDK and documentation. Sadly, the PDF they sent me was riddled with errors and the software library is also a bit dodgy. So, with the help of Edward Toroshchyn and a hefty amount of automated boiler-plate, I managed to get it working.

The full code is open source, but here's a quick walk-through of the important bits.

First, the AAR library needs to be imported into the project. Place it in app/libs and then include it in the Gradle build file:

dependencies {
    implementation(files("libs/badge_nfc_api-release.aar"))
}

The key to getting it working is in the Android permissions. It needs Bluetooth, NFC, and location. So the manifest has to contain:

<uses-permission android:name="android.permission.BLUETOOTH"/>
<uses-permission android:name="android.permission.BLUETOOTH_ADMIN"/>
<uses-permission android:name="android.permission.BLUETOOTH_SCAN"/>
<uses-permission android:name="android.permission.BLUETOOTH_CONNECT"/>
<uses-permission android:name="android.permission.NFC"/>
<uses-permission android:name="android.permission.NFC_TRANSACTION_EVENT"/>
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION"/>
<uses-permission android:name="android.permission.ACCESS_FINE_LOCATION"/>

The following imports are needed from the Android Archive library:

import cn.manytag.badge_nfc_api.manager.BadgeWriteManager
import cn.manytag.badge_nfc_api.manager.OnNFCReaderCallback
import cn.manytag.badge_nfc_api.manager.OnBluetoothCallBack
import cn.manytag.badge_nfc_api.manager.OnSendImageCallback

The library needs to be initialised with:

val state = BadgeWriteManager.getInstance().init(this)

When the phone reads the NFC tag, it gets a bunch of information:

BadgeWriteManager.getInstance().setOnNFCReaderCallback(object : OnNFCReaderCallback {
    override fun onReaderMessage(i: Int, tag: Tag) {
        if (i == 0) {
            BadgeWriteManager.getInstance().readNFC(tag)
        }
    }

    //  Get the data from the badge
    override fun onReaderType(tag: Tag, isodep: IsoDep, i: Int, type: String, size: String, color: String) {
        if (i == 0) {
            nfcData = """
                NFC Tag Detected!!!
                Tag: $tag
                IsoDep: $isodep
                Type: $type
                Size: $size
                Color: $color
            """.trimIndent()
            colorFromNFC = color
            tagObject = tag
            isoDepObject = isodep
            badgeType = type
            badgeSize = size
        }
    }
})

The color is most important right now. It says whether the badge is black and white, or black and white and red, or black and white and red and yellow.

After picking an image from the filesystem, it needs to be dithered into the correct colour format:

processedBitmap = originalBitmap?.let { bitmap ->
    colorFromNFC?.let { color ->
        BadgeWriteManager.getInstance().processImage(bitmap, color)
    }
}

Finally, the processed image needs to be converted to bytes and then sent to the badge via Bluetooth:

if (processedBitmap != null && badgeType != null && badgeSize != null && colorFromNFC != null) {
    val imgData = BadgeWriteManager.getInstance().getImageData(processedBitmap!!, colorFromNFC!!)

    BadgeWriteManager.getInstance().sentImageResource(
        tagObject!!, isoDepObject!!, imgData, badgeType!!, badgeSize!!, colorFromNFC!!
    )
}

I realise this is a bit "draw the rest of the owl" but that should be enough to get you started on building an app which can communicate with these badges.

The app I've built isn't the prettiest in the world but at least it works. It scans a badge, gets its info, picks an image, dithers it, then sends it to the badge.

You can play with the code on CodeBerg.

Here's how the admin list of posts looks to me:

I don't want it to look like that. I want it in RFC3339 format.

I know what you're thinking, just change the default date display - but that only seems to work in some areas of WordPress. It doesn't change the column-date format. Here's what mine is set to:

So that doesn't work.

Instead, you need to use the slightly obscure post_date_column_time filter

Add this to your theme's functions.php:

//  Admin view - change date format
function rfc3339_post_date_time( $time, $post ) {
    //  Modify the default time format
    $rfc3339_time = date( "Y-m-d H:i", strtotime( $post->post_date ) );
    return $rfc3339_time;
}
add_filter( "post_date_column_time", "rfc3339_post_date_time", 10, 2 );

And, hey presto, your date column will look like this:

Obviously, you can change that code to whichever date format you prefer.

There is no way to update the logo of a custom social connection on Auth0 without using the command line. On literally every other service I've used, there's a little box to upload a logo. But Okta have a funny idea of what developers want.

And, to make matters worse, their documentation contains an error! They don't listen to community requests or take bug reports, so I'm blogging in the hope that this is useful to you.

The Command

curl --request PATCH \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-H 'Authorization: Bearer eyJhb...ZEQ' \
  --url 'https://whatever.eu.auth0.com/api/v2/connections/con_qwerty123456' \
  --data ' ... '

You will also need to supply some JSON in the data parameter. I've formatted it to be easier to read than the garbage documentation. All of these fields are mandatory.

{
  "options": {
    "client_id": "your-app-id",
    "client_secret": "Shhhhhh!",
    "icon_url": "https://example.com/image.svg",
    "scripts": {
      "fetchUserProfile": "???"
    },
    "authorizationURL": "https://example.com/oauth2/authorize",
    "tokenURL": "https://example.com/oauth2/token",
    "scope": "auth"
  },
  "display_name": "Whatever"
}

OK, but how do you get all those values?

• Bearer token:
  • Create a management token
  • The only scope it needs is update:connections
• URl
  • This is your normal Auth0 domain name.
  • The Connection ID at the end can be found in the dashboard of your social connection
    
• Client ID & Secret
  • You set these in the social connection's dashboard.
• icon_url
  • Public link to an image. It can be an SVG.
• fetchUserProfile
  • Whatever code you want to run. If you don't want any, you can't leave it blank. So type in a couple of characters.
• authorizationURL and tokenURL
  • Wherever you want to redirect users to
• display_name
  • What you want to show to the user

This is such a load of bollocks! Is it really that hard for the Okta team to put an input field with "type the URl of your logo"?

<a href="https://shkspr.mobi/blog/2024/12/change-wordpress-fragment-links-in-rss-feeds-to-be-permalinks/#where-is-this-a-problem>Jump%20to%20heading</a>/code/prepSo%20when%20someone%20clicks%20on%20a%20link,%20they%20a%20href="#where-is-this-a-problem">go straight to the relevant section.  For example, they might want to skip straight to how to fix it.


Isn't that clever?

Where is this a problem?

This works great when someone is on my website. They're on the page, and a fragment links straight to the correct section of that page.

But some people view this blog in RSS & Atom feeds - and those feeds also power my newsletter.

When those people see a fragment, it is devoid of its original context. So they end up going to some random location, or my homepage.

How to fix it?

Stick this into your WordPress theme's functions.php file:

//  In the RSS feed, change #whatever to <permalink>#whatever
function rewrite_fragment_links_in_rss($content) {
    global $post;

    //  Ensure this is a feed
    if ( is_feed() && $post instanceof WP_Post ) {
        //  Get the permalink
        $base_url = get_permalink( $post );

        //  Regex to get href="https://shkspr.mobi/blog/2024/12/change-wordpress-fragment-links-in-rss-feeds-to-be-permalinks/#%20%20%20%20%20%20%20%20$content%20=%20preg_replace_callback(%20%20%20%20%20%20%20%20%20%20%20%20"/href=["\']#([^"\']+)["\']/',
            function ( $matches ) use ( $base_url ) {
                return 'href="' . esc_url( $base_url . '#' . $matches[1] ) . '"';
            },
            $content
        );
    }

    return $content;
}

//  Hook into feed filters for both excerpts and full content
add_filter( "the_excerpt_rss",  "rewrite_fragment_links_in_rss" );
add_filter( "the_content_feed", "rewrite_fragment_links_in_rss" );


That listens out for the RSS feed being generated and replaces #whatever with https://shkspr.mobi/blog/2024/12/change-wordpress-fragment-links-in-rss-feeds-to-be-permalinks#whatever

Nifty!

Hopefully, if you click on the links in my emails and feeds, it should take you to the right place now.
]]>
					
					https://shkspr.mobi/blog/2024/12/change-wordpress-fragment-links-in-rss-feeds-to-be-permalinks/feed/
			1
		
		
			
		
		
		https://shkspr.mobi/blog/2024/11/an-easy-guide-to-bluesky-verification/
					https://shkspr.mobi/blog/2024/11/an-easy-guide-to-bluesky-verification/#comments
				
		Tue, 19 Nov 2024 12:34:21 +0000
				
		
		
		
		
		https://shkspr.mobi/blog/?p=54022

					
										The new Twitter-Wannabe BlueSky has an interesting approach to verifying accounts. Rather than you sending in your passport, or paying a 3rd party, or bribing an employee - you can self-verify for free!


This opens up verification to small organisations, individuals, and anyone who wants to prove who they are. Brilliant!

Verification means that your @username will change to @Your.Website.com - this means that everyone can see your BlueSky account is owned by that specific website. When you change your name, you keep all your followers and posts.

Here are some organisations and people at risk of impersonation who have already done this:


UK Newspaper https://bsky.app/profile/theguardian.com
Trade Union https://bsky.app/profile/utaw.tech
Labour MPs https://bsky.app/profile/sarahowen.org.uk
Small Publisher https://bsky.app/profile/canongate.co.uk
Fun Website https://bsky.app/profile/openbenches.org


There is an easy way to get verified and a hard way.  Let's do the easy way!

1) Sign Up For BlueSky

Sign up and register a username. This can be anything you want. For example, I registered edent.bsky.social

2) Change Your User ID

Follow these steps:


Visit https://bsky.app/settings
Scroll down and select "Change Handle"
Click "I have my own domain"
Select "No DNS Panel". The screen should look like this:




Type in the domain name you want to verify
Click "Copy File Contents"


Keep this web page open.

3) Copy and Save Your DID

On your clipboard, you will have a bit of text which looks like this did:plc:dip7ueksh627fxacagfrdyz2

Save it in a text file called atproto-did



It is very important that the file doesn't end with .txt - it must be called atproto-did and nothing else.

The file should only contain the text you copied. Nothing else.

4) Upload The File To Your Website

This is the only technical bit of the process.  You need the ability to upload a file to your website.  I don't know whether you use FTP, a control panel, or email things to the person who manages your site.

You need to save the atproto-did file in a folder called /.well-known/

If that folder doesn't exist, create it. The folder name must be typed exactly like that, with the dot at the start.

You can check it has worked by visiting YourWebsite.com/.well-known/atproto-did

If you can see your DID, it worked!

5) Change Your Username

Go back to the "Change Handle" web page you opened in Step 2.

Click "Verify Text File" and then "Update".

6) That's It!

Feel free to share this guide with people and organisations who want to get verified on BSky.

Leave a comment if you found it useful or want me to clarify something.
]]>
					
					https://shkspr.mobi/blog/2024/11/an-easy-guide-to-bluesky-verification/feed/
			14
		
		
			
		
		
		https://shkspr.mobi/blog/2024/11/a-simple-and-free-way-to-post-rss-feeds-to-threads/
					https://shkspr.mobi/blog/2024/11/a-simple-and-free-way-to-post-rss-feeds-to-threads/#comments
				
		Wed, 06 Nov 2024 12:30:00 +0000
				
		
		
		
		
		
		https://shkspr.mobi/blog/?p=53776

					
										Threads is Meta's attempt to disrupt the social media landscape. Whether you care for it or not, there are a lot of users there. And, sometimes, you have to go where the audience is.


Here's how I build a really simple PHP tool to post to Threads using their official API.  This allows you to send a single status update programatically, or regularly send new items from your RSS feed to an account.

You can see the bot in action at https://www.threads.net/@openbenches_org

Get the code

The code is available as Open Source. It should be fairly self explanatory for a moderately competent programmer - but feel free to open an issue if you think it is confusing.

Get it working


Create an account on Threads (duh!) - this involves signing up to Instagram.
Create a Facebook Developer account.
Create an app which requests the Threads posting API.


You do not need to publish this app if you're only using it yourself.

Create a User Token using the "User Token Generator"
Get your Threads account's User ID with:


curl -s -X GET "https://graph.threads.net/v1.0/me?ields=id,username,name,threads_profile_picture_url,threads_biography&access_token=TOKEN"
(Yes, ields. If you use fields you get something else!)

Clone the RSS2Threads repo and stick it on a webserver somewhere.
Rename config.sample.php to config.php and add your feeds' details, along with your ID and Token.
Run php rss2threads.php


And that's it!

The service will download your RSS feed, check if it has posted the entries to Threads and, if not, post them.

How I built it

Shoulders of giants, and all that! I have been using Thomas Nesges's RSS2BSky for auto-posting to BlueSky. I also used Jesse Chen's Python Threads example code.

Posting is a two stage process.


POST the URl encoded text to:


https://graph.threads.net/USER_ID/threads?text=My%20post&access_token=TOKEN&media_type=TEXT
If successful, the API will return a Creation ID.

POST the Creation ID to:


https://graph.threads.net/USER_ID/threads_publish?creation_id=CREATION_ID&access_token=TOKEN
If successful, the API will return a Post ID.



Successful RSS posts are stored in a simple SQLite database. If an RSS entry was posted successfully, it won't be reposted.

Caveats


There are no unit tests, fuzzing, or exception handling. It's assumed you're running this on well-formed RSS that you trust.
The Threads API is slow! It takes ages for a post to be sent to it.
Getting a Threads API token is difficult and the margin is too small for me to explain it here.


Feedback

Please leave a comment here or on the code repository.
]]>
					
					https://shkspr.mobi/blog/2024/11/a-simple-and-free-way-to-post-rss-feeds-to-threads/feed/
			1
		
		
			
		
		
		https://shkspr.mobi/blog/2024/10/using-phplist-for-a-blogs-newsletter/
					https://shkspr.mobi/blog/2024/10/using-phplist-for-a-blogs-newsletter/#comments
				
		Thu, 31 Oct 2024 12:34:36 +0000
				
		
		
		
		
		
		
		https://shkspr.mobi/blog/?p=53583

					
										Some people like to receive this blog via email. I previously used JetPack to send out subscriber messages - but it became increasingly clear that Automattic isn't a good steward of such things.  I couldn't find any services which would let me send a few thousand subscribers a few emails per week, at zero cost.


So, redecentralise!

I installed phpList which is an open source email campaign tool.  My webhost - Krystal - had a one-click install option. But, phpList isn't quite one-click for sending out a regular blog newsletter.  I found the set-up to be quite confusing, so here are the steps I took to turn an RSS feed into an Email Newsletter for free.

Install the plugins


Navigate to Config → Manage plugins
Enable "CommonPlugin"
Add the RSS Feed Plugin using the Plugin package URL https://github.com/bramley/phplist-plugin-rssfeed/archive/master.zip


Configure the RSS Feed Plugin


Navigate to Config → Settings
Scroll down to the RSS Settings
Set both Minimum and Maximum number of items to 1

That will ensure you only send the latest RSS item as your newsletter.
Set "Use the item summary content (the description or summary element) instead of the content element" to "No". This will allow the full text of the RSS item to be sent.


Edit config.php

For some reason, you need to manually edit this file in a text editor, rather than a GUI.


Set define('USE_REPETITION', 1); - this allows the newsletter to be sent whenever there is a new RSS item.
Set define('CLICKTRACK', 0); - this removes tracking links from your emails. I don't care who opens my emails or what they click on.


Add The Campaign


Go to  Campaigns → Send a campaign.
Start a new campaign.


Tab 1


Campaign subject should be [RSSITEM:TITLE] - that will make the subject line the same as your post's title
Compose message should be [RSS] - that will ensure the contents come from your RSS feed.


Tab 2


Add your RSS feed's URl
Order items "Newest" first - to get the most recent item.
Add a custom HTML template. I used one from https://emailframe.work/


<div style="margin:0; padding:0; background-color:#F2F2F2;">
  <h1><a href="[URL]">[TITLE]</a></h1>
  <table width="100%" border="0" cellpadding="0" cellspacing="0" bgcolor="#F2F2F2">
      <tr>
          <td valign="top">
              [CONTENT]
          </td>
      </tr>
  </table>
</div>


Tab 3


Send as HTML


Tab 4


"Stop sending after" - choose the furthest date in the future possible.
"Repeat campaign every" - I chose "hour". That should check the RSS feed each hour.


Tab 5


"Lists" - pick the email list you want to send from.


Tab 6


You should be finished! It will tell you if there are any errors.
Place the campaign in the queue for processing.


WordPress Sign Up Form

You can either redirect users to your phpList subscription page, or put a form directly on your site.

<form method="post" action="/YourSubscribePage/?p=subscribe&id=1" name="subscribeform">
    <label for="email">Email address:</label>
    <input type="email" name="email" required="required" placeholder="" size="40" id="email">
    <input type="hidden" name="htmlemail" value="1">
    <input type="hidden" name="list[2]" value="signup">
    <input type="hidden" name="listname[2]" value="newsletter">
    <div style="display:none">
        <input type="text" name="VerificationCodeX" value="" size="20">
    </div>
    <input type="submit" name="subscribe" value="Subscribe">
</form>


Adjust the hidden parameters based on your list.

If in doubt, go to Config →  Subscribe pages, and generate a new subscribe page. Then copy the form from that.

Cron Jobs

You need two cron jobs set up.

Update the RSS feed

I run this every hour:

/usr/bin/php /path/to/YourSubscribePage/admin/index.php -p get -m RssFeedPlugin -c /path/to/YourSubscribePage/config/config.php

Process the Queue

I run this a few minutes after the RSS feed is updated

/usr/bin/php -q /path/to/YourSubscribePage/admin/index.php -p processqueue -c /path/to/YourSubscribePage/config/config.php >/dev/null

And then...

That should be it.  There are lots of options which you can fiddle around with. But the above should be enough to get your first newsletter out.

Huge thanks to Duncan Cameron for graciously answering my noddy questions and helping me out with the config.
]]>
					
					https://shkspr.mobi/blog/2024/10/using-phplist-for-a-blogs-newsletter/feed/
			2
		
		
			
		
		
		https://shkspr.mobi/blog/2024/05/hdr-on-a-pioneer-vsx-933/
					https://shkspr.mobi/blog/2024/05/hdr-on-a-pioneer-vsx-933/#comments
				
		Wed, 29 May 2024 11:34:49 +0000
				
		
		
		
		
		https://shkspr.mobi/blog/?p=50627

					
										I bloody hate hardware manufacturers. I wanted to use HDR on my PlayStation 5. The console supports it, my TV supports it, my amp supports it, my cables support it. Yet it wasn't working.  I tried everything - updating firmware, replacing cables, and even reading the manual. Nothing.


And then I stumbled on the answer thanks to a random forum post.

Perform the following procedure when the unit is on.


While pressing DIMMER on the main unit, press AUTO/DIRECT to display the current setting on the display. While this is being displayed, while pressing DIMMER on the main unit, repeatedly press AUTO/DIRECT to switch the setting.
To exit the settings, release your finger. After a few seconds, the display goes out and the switching is complete.


Once the setting was changed to "Enhanced" HDR worked! But why isn't it in the manual? A bit of searching for the text finds a file called manual/sup/upd/hdmi_4k_pio.pdf .

So I assume that this is a supplement meant to update the original manual - it is mentioned as new functionality introduced after a firmware update. But why isn't it in the main manual?

If you visit the VSX-933 product page you can download the manual, but there's no mention of a supplement or an update.  The original manual was released in 2018, and the supplement in 2019.

I wonder what other features this amp is hiding that Pioneer simply haven't bothered to tell anyone about?
]]>
					
					https://shkspr.mobi/blog/2024/05/hdr-on-a-pioneer-vsx-933/feed/
			3
		
		
			
		
		
		https://shkspr.mobi/blog/2024/05/link-relalternate-typetext-plain/
					https://shkspr.mobi/blog/2024/05/link-relalternate-typetext-plain/#comments
				
		Fri, 10 May 2024 11:34:57 +0000
				
		
		
		
		https://shkspr.mobi/blog/?p=50490

					
										Hot on the heels of yesterday's post, I've now made all of this blog available in text-only mode.


Simply append .txt to the URl of any page and you'll get back the contents in plain UTF-8 text. No formatting, no images (although you can see the alt text), no nothing!


Front page https://shkspr.mobi/blog/.txt
This blog post https://shkspr.mobi/blog/2024/05/link-relalternate-typetext-plain/.txt
A tag https://shkspr.mobi/blog/tag/solar.txt


This was slightly tricky to get right!  While there might be an easier way to do it, here's how I got it to work.

Firstly, when someone requests /whatever.txt, WordPress is going to 404 - because that page doesn't exist. So, my theme's functions.php, detects any URls which end in .txt and redirects it to a different template.

//  Theme Switcher
add_filter( "template_include", "custom_theme_switch" );
function custom_theme_switch( $template ) {

    //  What was requested?
    $requested_url = $_SERVER["REQUEST_URI"];

    //  Check if the URL ends with .txt
    if ( substr( $requested_url, -4 ) === ".txt")  {    
        //  Get the path to the custom template
        $custom_template = get_template_directory() . "/templates/txt-template.php";
        //  Check if the custom template exists
        if ( file_exists( $custom_template ) ) {
            return $custom_template;
        }
    }

    //  Return the default template
    return $template;
}


The txt-template.php file is more complex.  It takes the requested URl, strips off the .txt, matches it against the WordPress rewrite rules, and then constructs the WP_Query which would have been run if the .txt wasn't there.

//  Run the query for the URl requested
$requested_url = $_SERVER['REQUEST_URI'];    // This will be /whatever
$blog_details = wp_parse_url( home_url() );  // Get the blog's domain to construct a full URl
$query = get_query_for_url( 
    $blog_details["scheme"] . "://" . $blog_details["host"] . substr( $requested_url, 0, -4 )
);

function get_query_for_url( $url ) {
    //  Get all the rewrite rules
    global $wp_rewrite;

    //  Get the WordPress site URL path
    $site_path = parse_url( get_site_url(), PHP_URL_PATH ) . "/";

    //  Parse the requested URL
    $url_parts = parse_url( $url );

    //  Remove the domain and site path from the URL
    //  For example, change `https://example.com/blog/2024/04/test` to just `2024/04/test`
    $url_path = isset( $url_parts['path'] ) ? str_replace( $site_path, '', $url_parts['path'] ) : '';

    //  Match the URL against WordPress rewrite rules
    $rewrite_rules = $wp_rewrite->wp_rewrite_rules();
    $matched_rule = false;

    foreach ( $rewrite_rules as $pattern => $query ) {
        if ( preg_match( "#^$pattern#", $url_path, $matches ) ) {
            $matched_rule = $query;
            break;
        }
    }

    //  Replace each occurrence of $matches[N] with the corresponding value
    foreach ( $matches as $key => $value ) {
        $matched_rule = str_replace( "\$matches[{$key}]", $value, $matched_rule );
    }

    //  Turn the query string into a WordPress query
    $query_params = array();
    parse_str(
        parse_url( $matched_rule, PHP_URL_QUERY), 
        $query_params
    );

    //  Construct a new WP_Query object using the extracted query parameters
    $query = new WP_Query($query_params);

    //  Return the result of the query
    return $query;
}


From there, it's a case of iterating over the posts returned by the query. You can see the full code on my GitLab.
]]>
					
					https://shkspr.mobi/blog/2024/05/link-relalternate-typetext-plain/feed/
			4