Graphing the connections between my blog posts

Terence Eden — Thu, 09 Jan 2025 12:34:56 +0000

I love ripping off good ideas from other people's blogs. I was reading Alvaro Graves-Fuenzalida's blog when I saw this nifty little force-directed graph:

When zoomed in, it shows the relation between posts and tags.

In this case, I can see that the posts about Small Gods and Pyramids both share the tags of Discworld, Fantasy, and Book Review. But only Small Gods has the tag of Religion.

Isn't that cool! It is a native feature of Quartz's GraphView. How can I build something like that for my WordPress blog?

Aim

Create an interactive graph which shows the relationship between a post, its links, and their tags.

It will end up looking something like this:

You can get the code or follow along to see how it works.

This is a multi-stage process. Let's begin!

What We Need

When on a single Post, we need the following:

The tags assigned to that Post.
Internal links back to that Post.
Internal links from that Post.
The tags assigned to links to and from that Post.

Tags assigned to that Post.

This is pretty easy! Using the get_the_tag_list() function we can, unsurprisingly, get all the tags associated with a post.

$post_tags_text = get_the_tag_list( "", ",", $ID );
$post_tags_array = explode( "," , $post_tags_text );

That just gets the list of tag names. If we want the tag IDs as well, we need to use the get_the_tags() function.

$post_tags = get_the_tags($ID);
$tags = array();
foreach($post_tags as $tag) {
    $tags[$tag->term_id] = $tag->name; 
}

Internal links back to the Post is slightly trickier. WordPress doesn't save relational information like that. Instead, we get the Post's URl and search for that in the database. Then we get the post IDs of all the posts which contain that string.

//  Get all the posts which link to this one, oldest first
$the_query = new WP_Query(
    array(
        's' => $search_url,
        'post_type' => 'post',
        "posts_per_page" => "-1",
        "order" => "ASC"
    )
);

//  Nothing to do if there are no inbound links
if ( !$the_query->have_posts() ) {
    return;
}

Backlinks' Tags

Once we have an array of posts which link back here, we can get their tags as above:

//  Loop through the posts
while ( $the_query->have_posts() ) {
    //  Set it up
    $the_query->the_post();
    $id                  = get_the_ID();
    $title               = esc_html( get_the_title() );
    $url                 = get_the_permalink();
    $backlink_tags_text  = get_the_tag_list( "", ",", $ID );
    $backlink_tags_array = explode( "," , $backlink_tags_text );
}

Links from the Post

Again, WordPress's lack of relational links is a weakness. In order to get internal links, we need to:

Render the HTML using all the filters
Search for all <a href="…">
Extract the ones which start with the blog's domain
Get those posts' IDs.

Rendering the content into HTML is done with:

$content = apply_filters( "the_content", get_the_content( null, false, $ID ) );

Searching for links is slightly more complex. The easiest way is to load the HTML into a DOMDocument, then extract all the anchors. All my blog posts start /blog/YYYY so I can avoid selecting links to tags, uploaded files, or other things. Your blog may be different.

$dom = new DOMDocument();
libxml_use_internal_errors( true ); //  Suppress warnings from malformed HTML
$dom->loadHTML( $content );
libxml_clear_errors();

$links = [];
foreach ( $dom->getElementsByTagName( "a" ) as $anchor ) {
    $href = $anchor->getAttribute( "href" );
    if (preg_match('/^https:\/\/shkspr\.mobi\/blog\/\d{4}$/', $href)) {
        $links[] = $href;
    }
}

The ID of each post can be found with the url_to_postid() function. That means we can re-use the earlier code to see what tags those posts have.

Building a graph

OK, so we have all our constituent parts. Let's build a graph!

Graphs consist of nodes (posts and tags) and edges (links between them). The exact format of the graph is going to depend on the graph library we use.

I've decided to use D3.js's Force Graph as it is relatively simple and produces a reasonably good looking interactive SVG.

Imagine there are two blog posts and two hashtags.

const nodes = [
    { id: 1, label: "Blog Post 1",    url: "https://example.com/post/1", group: "post" },
    { id: 2, label: "Blog Post 2",    url: "https://example.com/post/2", group: "post" },
    { id: 3, label: "hashtag",        url: "https://example.com/tag/3",  group: "tag"  },
    { id: 4, label: "anotherHashtag", url: "https://example.com/tag/4",  group: "tag"  },
];

Blog Post 1 links to Blog Post 2.
Blog Post 1 has a #hashtag.
Both 1 & 2 share #anotherHashtag.

const links = [
    { source: 1, target: 2 },
    { source: 3, target: 1 },
    { source: 4, target: 1 },
    { source: 4, target: 2 },
];

Here's how to create a list of nodes and their links. You will need to edit it for your own blog's peculiarities.

<?php 
// Load WordPress environment
require_once( "wp-load.php" );

//  Set up arrays for nodes and links
$nodes = array();
$links = array();

//  ID of the Post
$main_post_id = 12345;

//  Get the Post's details
$main_post_url   = get_permalink( $main_post_id );
$main_post_title = get_the_title( $main_post_id );

//  Function to add new nodes
function add_item_to_nodes( &$nodes, $id, $label, $url, $group ) {
    $nodes[] = [ 
        "id"    => $id, 
        "label" => $label, 
        "url"   => $url, 
        "group" => $group
    ];

}

//  Function to add new relationships
function add_relationship( &$links, $source, $target ) {
    $links[] = [
        "source" => $source,
        "target" => $target
    ];
}

//  Add Post to the nodes
add_item_to_nodes( $nodes, $main_post_id, $main_post_title, $main_post_url, "post" );

//  Get the tags of the Post
$main_post_tags = get_the_tags( $main_post_id );

//  Add the tags as nodes, and create links to main Post
foreach( $main_post_tags as $tag ) {
    $id   = $tag->term_id;
    $name = $tag->name;

    //  Add the node
    add_item_to_nodes( $nodes, $id, $name, "https://shkspr.mobi/blog/tag/" . $name, "tag" );
    //  Add the relationship
    add_relationship( $links, $id, $main_post_id );
}

//  Get all the posts which link to this one, oldest first
$the_query = new WP_Query(
    array(
        's'              => $main_post_url,
        'post_type'      => 'post',
        "posts_per_page" => "-1",
        "order"          => "ASC"
    )
);

//  Nothing to do if there are no inbound links
if ( $the_query->have_posts() ) {
    //  Loop through the posts
    while ( $the_query->have_posts() ) {
        //  Set up the query
        $the_query->the_post();
        $post_id = get_the_ID();
        $title = esc_html( get_the_title() );
        $url   = get_the_permalink();

        //  Add the node
        add_item_to_nodes( $nodes, $post_id, $title, $url, "post" );
        //  Add the relationship
        add_relationship( $links, $post_id, $main_post_id );

        //  Get the tags of the Post
        $post_tags = get_the_tags( $post_id );

        //  Add the tags as nodes, and create links to main Post
        foreach($post_tags as $tag) {

            $id   = $tag->term_id;
            $name = $tag->name;

            //  Add the node
            add_item_to_nodes( $nodes, $id, $name, "https://shkspr.mobi/blog/tag/" . $name, "tag" );
            //  Add the relationship
            add_relationship( $links, $id, $post_id );
        }

    }
}

//  Get all the internal links from this post
//  Render the post as HTML
$content = apply_filters( "the_content", get_the_content( null, false, $ID ) );

//  Load it into HTML
$dom = new DOMDocument();
libxml_use_internal_errors( true );
$dom->loadHTML( $content );
libxml_clear_errors();

//  Get any <a href="…" which starts with https://shkspr.mobi/blog/
$internal_links = [];
foreach ( $dom->getElementsByTagName( "a" ) as $anchor ) {
    $href = $anchor->getAttribute( "href" );
    if (preg_match('/^https:\/\/shkspr\.mobi\/blog\/\d{4}$/', $href)) {
        $internal_links[] = $href;
    }
}

//  Loop through the internal links, get their hashtags
foreach ( $internal_links as $url ) {
    $post_id = url_to_postid( $url );
    //  Get the Post's details
    $post_title = get_the_title( $id );

    //  Add the node
    add_item_to_nodes( $nodes, $post_id, $post_title, $url, "post" );
    //  Add the relationship
    add_relationship($links, $main_post_id, $post_id );

    //  Get the tags of the Post
    $post_tags = get_the_tags( $post_id );

    //  Add the tags as nodes, and create links to main Post
    foreach( $post_tags as $tag ) {
        $id   = $tag->term_id;
        $name = $tag->name;

        //  Add the node
        add_item_to_nodes( $nodes, $id, $name, "https://shkspr.mobi/blog/tag/" . $name, "tag" );
        //  Add the relationship
        add_relationship( $links, $id, $post_id );
    }
}

//  Deduplicate the nodes and links
$nodes_unique = array_unique( $nodes, SORT_REGULAR );
$links_unique = array_unique( $links, SORT_REGULAR );

//  Put them in the keyless format that D3 expects
$nodes_output = array();
$links_output = array();

foreach ( $nodes_unique as $node ) {
    $nodes_output[] = $node;
}

foreach ( $links_unique as $link ) {
    $links_output[] = $link;
}

//  Return the JSON
echo json_encode( $nodes_output, JSON_PRETTY_PRINT );
echo "\n";
echo json_encode( $links_output, JSON_PRETTY_PRINT );

Creating a Force Directed SVG

Once the data are spat out, you can include them in a web-page. Here's a basic example:

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <title>Force Directed Graph</title>
        <script src="https://d3js.org/d3.v7.min.js"></script>
    </head>
    <body>
        <svg width="800" height="600">
            <defs>
                <marker id="arrowhead" markerWidth="10" markerHeight="7" refX="10" refY="3.5" orient="auto" fill="#999">
                <path d="M0,0 L10,3.5 L0,7 Z"></path>
                </marker>
            </defs>
        </svg>
        <script>

            const nodes = [];
            const links = [];

            const width  = 800;
            const height = 600;

            const svg = d3.select("svg")
                .attr( "width",  width  )
                .attr( "height", height );

            const simulation = d3.forceSimulation( nodes )
                .force( "link",   d3.forceLink( links ).id( d => d.id ).distance( 100 ) )
                .force( "charge", d3.forceManyBody().strength( -300 ) )
                .force( "center", d3.forceCenter( width / 2, height / 2 ) );

            //  Run simulation with simple animation
            simulation.on("tick", () => {
                link
                    .attr("x1", d => d.source.x)
                    .attr("y1", d => d.source.y)
                    .attr("x2", d => d.target.x)
                    .attr("y2", d => d.target.y);   node
                    .attr("transform", d => `translate(${d.x},${d.y})`);
            });

            // Draw links
            const link = svg.selectAll( ".link" )
                .data(links)
                .enter().append("line")
                .attr( "stroke", "#999" )
                .attr( "stroke-width", 2 )
                .attr( "x1", d => d.source.x )
                .attr( "y1", d => d.source.y )
                .attr( "x2", d => d.target.x )
                .attr( "y2", d => d.target.y )
                .attr( "marker-end", "url(#arrowhead)" );

            //  Draw nodes
            const node = svg.selectAll( ".node" )
                .data( nodes )
                .enter().append( "g" )
                .attr( "class", "node" )
                .attr( "transform", d => `translate(${d.x},${d.y})` )
                .call(d3.drag() //  Make nodes draggable
                    .on( "start", dragStarted )
                    .on( "drag",  dragged )
                    .on( "end",   dragEnded ) 
                );

            //  Add hyperlink
            node.append("a")
            .attr( "xlink:href", d => d.url ) //    Link to the node's URL
            .attr( "target", "_blank" ) //  Open in a new tab
            .each(function (d) {
                const a = d3.select(this);
                //  Different shapes for posts and tags
                if ( d.group === "post" ) {
                    a.append("circle")
                        .attr("r", 10)
                        .attr("fill", "blue");
                } else if ( d.group === "tag" ) {
                    //  White background rectangle
                    a.append("rect")
                            .attr("width", 20)
                            .attr("height", 20)
                            .attr("x", -10)
                            .attr("y", -10)
                            .attr("fill", "white"); 
                    // Red octothorpe
                    a.append("path")
                            .attr("d", "M-10,-5 H10 M-10,5 H10 M-5,-10 V10 M5,-10 V10") 
                            .attr("stroke", "red")
                            .attr("stroke-width", 2)
                            .attr("fill", "none");
                }
                //  Text label
                a.append( "text")
                    .attr( "dy", 4 )
                    .attr( "x", d => ( d.group === "post" ? 12 : 14 ) )
                    .attr( "fill", "black" )
                    .style("font-size", "12px" )
                    .text( d.label );
            });

            //  Standard helper functions to make nodes draggable
            function dragStarted( event, d ) {
                if ( !event.active ) simulation.alphaTarget(0.3).restart();
                d.fx = d.x;
                d.fy = d.y;
            }
            function dragged( event, d ) {
                d.fx = event.x;
                d.fy = event.y;
            }
            function dragEnded( event, d ) {
                if (!event.active) simulation.alphaTarget(0);
                d.fx = null;
                d.fy = null;
            }

        </script>
    </body>
</html>

Next Steps

It needs a bit of cleaning up if I want to turn it into a WordPress plugin. It might be nice to make it a static SVG rather than relying on JavaScript. And the general æsthetic needs a bit of work.

Perhaps I could make it 3D like my MSc Dissertation?

But I'm pretty happy with that for an afternoon hack!

You can get the code if you want to play.

Surrey Police and the Case of The Misleading Pie Charts

Terence Eden — Mon, 18 Mar 2013 07:30:10 +0000

Surrey County Council have sent every household in the county a booklet explaining how our council tax is being spent. Within it is a highly political comment from Kevin Hurley, the newly elected Police and Crime Commissioner.

He presents a pie chart showing how the police force spend its money. Take a look at it and ask yourself this question: what percentage is spent on "Employees".

~~Please use this poll to record your guess - answers at the end of this blog.~~

Pie charts have a long and noble history. They were popularised by Florence Nightingale and were hugely effective in helping politicians understand the causes of death among soldiers during the Crimean War.

As we understand more about the human brain and how we perceive shapes, it is becoming clear that pie charts are ineffective for representing complex information.

2D pie charts can still serve a useful purpose in limited circumstances. The real problem is with 3D pie charts. As far as I can tell, these abominations were popularised by Microsoft's Excel charting software.

3D charts distort the view of the data in such a way that it becomes increasingly hard to understand the information being presented. A picture being worth 1000 words, allow me to demonstrate:

So, just how bad is Surrey Police's Pie Chart? In an extremely scientific study of asking half a dozen people, they all guessed between 75% and 85%. That's quite a wide range considering it's a multi-million pound difference.

On the opposite page to the pie chart is this summary of spending.

In slightly more readable format, it is:

Category	£	%
Employees	£181.70	81.9%
Premises	£8.00	3.6%
Supplies	£27.20	12.3%
Transport	£5.00	2.3%
Total	£216.90	97.7%

A few interesting things to note here.

Firstly, how do we calculate the percentages? The total spend isn't mentioned in the report (£216.90). If we use that, "Employees" accounts for 81.9% of spending.

If we take into account the gross expenditure (£207.70) the figure jumps to 87.5%.

Secondly, if we do assume that we're using the unreported total spend - there is at least 2% missing. Some of which can be explained by rounding - but I wonder what the rest of the money is spent on.

Given the above, I don't think the provided pie chart allows Surrey residents to see an accurate view of how their hard earned money is being spent.

Hopefully, this side-by-side - of the above data - will show you how 3D pie charts distort data and end up misleading their audience.

With this overlay, we can see the distortion much more clearly. The smaller sections of the chart look disproportionately larger. It's time to announce a zero tolerance crackdown on dodgy data representation.

graphs – Terence Eden’s Blog

Graphing the connections between my blog posts

Surrey Police and the Case of The Misleading Pie Charts