I seem to recall that our student pub charged about £2 per pint. And minimum wage around that time was £4 per hour. So a drink was 30 minutes' wages.

Today the minimum wage is about £12 and that pint cost me £6. So, again, about half an hour.

But the human memory is fickle! Let's get some actual historical data.

The UK's Office for National Statistics maintains a dataset of historic draught lager prices.

Well, my memory wasn't too hazy! About £2 when I was at uni. The national average price now is about a fiver - so the London premium wasn't too outrageous.

But how does that compare to wages? The history of the minimum wage is complicated - with several different bands being introduced. It ends up looking something like this:

So I grabbed the most recent data and plotted the ratio between the cost of draught lager and minimum wage:

Ah! It turns out that the cost of beer as a ratio to minimum wage is pretty consistent - somewhere between 27 to 40 minutes. Right now, draught lager is cheaper in terms of minimum wage than it has ever been!

Obviously, averages hide all sorts of sins. I'm sure your favourite brand of premium Bohemian pilsner has dramatically risen in price. And minimum wage doesn't necessarily mean disposable income. And you now have a student loan repayment rather than cash being dropped into your account. And the music they play in pubs is crap these days. And you back hurts ever since you tried to match your younger team members pint for pint and slipped in a puddle of your own sick.

Remember, nostalgia is actively dangerous to your mental health.

has anyone else noticed that food tasted better in the past? it was mushy and easy to eat. and the spoon would come at you like an airplane

— leon (@leyawn.bsky.social) 2025-12-05T21:38:21.731Z

Me? I prefer to look at how many people visit my blog from each site. It is an imperfect measure - and a vain one - but lets me know where I should be spending my time. No point posting on a network which is just bots talking to each other, right?

Earlier this year I built a stats-counter for my blog. Every time someone clicks from a website which links to my blog, it records that visit in a database. I get to see which blog posts are doing numbers, and where those numbers came from.

Until fairly recently, the Mastodon social network didn't send referer details. I thought that reduced the visibility of the network and lobbied for it to change. As various Mastodon servers upgrade, and admins opt-in, it is becoming more apparent just how much traffic originates from the Fediverse.

Over the last few weeks, here's how many people have clicked from BlueSky and Mastodon to one of my blog posts.

Total	Source
1,607	bsky.app
752	mastodon.social

At first glance, it doesn't look good for our elephantine friends, does it? The butterfly sends over twice the traffic. Game over!

But, of course, while Mastodon.social is the biggest instance - it is far from the only one. What happens if we slide down the long tail? Here's all the Mastodon-ish instances which sent me over 10 clicks.

Total	Source
193	phanpy.social
120	android-app://org.joinmastodon.android/
106	infosec.exchange
62	mas.to
59	mstdn.social
55	social.vivaldi.net
49	wandering.shop
48	fosstodon.org
33	mathstodon.xyz
27	mastodon.online
26	mastodon.scot
24	app.wafrn.net
19	indieweb.social
18	social.lol
17	tech.lgbt
17	toot.wales
16	en.osm.town
16	feditrends.com
14	mstdn.ca
14	piefed.social
12	wetdry.world
11	c.im
11	mastodon.nl
51	Sites sending < 10 clicks

Ah! Add them all up and you get a grand total of 1,773 visitors from Mastodon-powered sites. That's more than BlueSky.

Now, there are some obvious caveats to the data:

• I have a smaller follower count on BlueSky than I do on Mastodon.
• My posts may appeal more to one demographic than another.
• People may have strict privacy controls which suppress the true volume of visitors.
• There's no way to measure how long someone spends reading my posts.
• RSS and newsletter visitors aren't counted.
• Clicks from apps may not always show a referer.
• Some people may be on multiple services.
• Fediverse users can follow the post directly, so don't need to visit the site to read it.

And yet… no matter how you slice it, Fediverse servers are sending as much traffic as BlueSky!

I think this is brilliant. Web services should be able to scale from small to big - and each ActivityPub-powered site helps power the open Internet.

Just for completeness, this is how Reddit, Facebook, LinkedIn, Twitter, and Lemmy do over the same period:

Total	Source
1,158	reddit.com
585	android-app://com.reddit.frontpage/
76	facebook.com
76	https://old.reddit.com/r/programming/
56	https://www.reddit.com/r/programming/
52	youtube.com
41	t.co
38	https://old.reddit.com/r/todayilearned/comments/1nsw7f4/til_in_mongolia_instead_of_a_street_address_a/
31	linkedin.com
27	android-app://io.syncapps.lemmy_sync/
27	https://www.reddit.com/r/todayilearned/comments/1nsw7f4/til_in_mongolia_instead_of_a_street_address_a/
22	https://old.reddit.com/r/programming/comments/1n96ftn/40_years_later_are_bentleys_programming_pearls/
22	lemmy.ca
17	android-app://com.linkedin.android/
16	lemmy.dbzer0.com
14	feddit.org
11	https://www.reddit.com/r/programming/comments/1n96ftn/40_years_later_are_bentleys_programming_pearls/
10	discuss.tchncs.de
10	l.instagram.com
8	lemmy.blahaj.zone
6	https://www.reddit.com/r/GrapheneOS/comments/1m2l84b/considering_making_the_switch_does_google_pay/
6	reddthat.com

If you add up all the Lemmy instances, they send about as much traffic as Facebook and LinkedIn combined. That's not a huge surprise - those platforms hate anyone clicking away to the wider web.

Twitter is basically the Dead Internet. I'm no longer on there, but I do occasionally search it to see who is sharing my posts. The popular posts I write get shared a lot - sometimes by accounts with huge followers - yet there are no comments or retweets and barely and clicks.

I don't do Instagram or Threads, and that might be reflected in their low numbers. But I'm not active on YouTube either - yet people there occasionally link back to me.

Final Thoughts

Firstly, my stats only represent my site. Your site might be very different.

Secondly, I've ignored search engine traffic, big blogs, newsletters, and other sources.

Thirdly, and most importantly, this isn't a competition! The desire for a "winner-takes-all" service is dangerous and disturbing. An ecosystem is at its most vibrant when there are multiple participants each thriving in their own niche.

I want a thousand sites, running a hundred different software stacks, some of which only serve a dozen people, or even a lone participant.

Diversity is strength.

What is a broken link?

Every day, I look at the On This Day page of my blog and look at that day's historic posts. I click on every link to see if it is still working. If it isn't, I have a few options.

1. If the site is working, but the content has moved, I change the link to point at the content.
2. If the site is dead, or the content isn't there, I stick the URl into the WayBack Machine and pick the archive closest to the publication date.
3. If the archive doesn't have it, I try to find a link from around the same date and point it to that.
4. If not, I remove the link.

Why don't I just cURL the URl and see if it responds with a 2xx HTTP code?

I used to use an automated checker to test my links. But spam sites lie. Lots of the dead sites have been taken over by scammers, spammers, and AI grifters. They return an HTTP OK and then serve up advertising. Yeuch! So now I spend each morning Tending To My Digital Garden and manually checking the links.

So, let's scan through every link within my blog posts and see which ones have been archived.

Results

Here's a stacked chart showing blog posts from 2008 until today:

And, if you prefer percentages, here's another chart:

Rather unsurprisingly, the older posts have the most link-rot.

Here's the data if you want to play with it yourself:

Year,Total,Archives
2008,159,27
2009,1335,156
2010,1166,91
2011,1130,85
2012,1392,98
2013,1889,84
2014,1858,35
2015,3016,43
2016,2274,37
2017,1309,17
2018,1561,25
2019,1580,18
2020,3362,33
2021,3621,40
2022,2904,22
2023,2358,15
2024,2136,19

Deficiencies

There will be some links that I haven't spotted. There will be dead links in review metadata and in comments. My code might not have accounted for some weird edge cases in the HTML.

But, generally, it should be a roughly accurate assessment.

Generating the data

Using the WP CLI, this can be run with wp eval-file archive.php:

<?php
//  Load the WordPress environment
require_once( "wp-load.php" );

//  Loop through the desired years
for ( $year = 2008; $year <= 2024; $year++ ) {
        //  Initialise the counters
        $total_links   = 0;
        $archive_links = 0;

        //  Get all the published posts for that year
        $posts = get_posts( [
            "date_query" => [
                [ "year" => $year, ],
            ],
            "post_type"   => "post",
            "post_status" => "publish",
            "numberposts" => -1,
        ] );

        //  Loop through each post
        foreach ( $posts as $post ) {
            $ID = $post->ID;

            //  Render the content of the post
            $content = apply_filters( "the_content", get_the_content( null, false, $ID ) );

            //  Count total occurrences of "<a href="
            preg_match_all( '/<a href=/i', $content, $matches );
            $total_links += count( $matches[0] );

            //  Count total occurrences of "<a href=" which contain "archive.org"
            preg_match_all( '/<a href=["\'][^"\']*archive.org/i', $content, $archive_matches );
            $archive_links += count( $archive_matches[0] );
        }

    //  Display the results as CSV
    echo "{$year},";
    echo "{$total_links},";
    echo "{$archive_links}\n";
}

But I am vain. I want all my old JetPack stats to appear in Koko so I can look back on the glory days of blogging.

Koko has two main tables. The first is a summary table called wpbp_koko_analytics_site_stats :

That's the total number of visitors and page views for each date.

Then there's a more detailed breakdown at wpbp_koko_analytics_post_stats

That shows every individual post's number of views and visitors per day.

WordPress's database is MySQL. It can handle CSV imports. So, given these tables are pretty simple, it is possible to get the old WordPress stats, convert them to CSV, and then import them.

Let's go!

As described in a previous post, the JetPack stats API is fairly basic. It doesn't differentiate between visitors and pageviews. So, for this import, we'll pretend they're the same.

This will be a 4 step process.

1. Get all your stats in JSON format

This code loops through your stats and downloads a JSON file for each one.

You will need:

• Your API key - find it at https://apikey.wordpress.com/
• Your blog's web address - I assume you know this
• The earliest date you have for JetPack - you will need to find this yourself

import requests
import datetime
import os
import json

# Directory to save the JSON files
save_dir = "jetpack_stats"
os.makedirs(save_dir, exist_ok=True)

# URL of the API
base_url = "https://stats.wordpress.com/csv.php?api_key=123456789012"+\
           "&blog_uri=https://example.com/"+\
           "&table=postviews"+\
           "&days=1"+\
           "&format=json"+\
           "&limit=-1"+\
           "&end="

# Make API call and save the response
def fetch_and_save_json(date):
    # Format the date as ISO8601 (YYYY-MM-DD)
    formatted_date = date.isoformat()

    # Make the API call
    url = f"{base_url}{formatted_date}"
    response = requests.get(url)

    if response.status_code == 200:
        data = response.json()
        file_name = f"{formatted_date}.json"
        file_path = os.path.join(save_dir, file_name)
        with open(file_path, "w") as f:
            json.dump(data, f, indent=4)

        print(f"Saved {formatted_date}")
    else:
        print(f"Failed! {formatted_date} status code: {response.status_code}")

# Iterate over a date range
start_date = datetime.date(2020,  1 , 1)
end_date   = datetime.date(2024, 10, 30)

# Loop through all dates 
current_date = start_date
while current_date <= end_date:
    fetch_and_save_json(current_date)
    current_date += datetime.timedelta(days=1)

2. Generate The Individual Posts' Stats

This takes those JSON files and creates a single CSV ready to upload to wpbp_koko_analytics_post_stats.

import os
import json
import csv

# Directory where the JSON files are stored
json_dir = "jetpack_stats"

# List to hold the loaded data
all_data = []

json_files = sorted( [ file for file in os.listdir( json_dir ) if file.endswith( ".json" ) ] )

# Loop through all files in the directory
for file_name in json_files :
    if file_name.endswith( ".json" ) :
        file_path = os.path.join(json_dir, file_name)
        with open(file_path, "r") as json_file :
            data = json.load(json_file)
            all_data.append(data)  # Add the data to the list

print(f"Total files loaded: {len(all_data)}")
with open( "wpbp_koko_analytics_post_stats.csv", "w", newline="") as csvfile:
    csvwriter = csv.writer( csvfile, delimiter="," )
    for stat in all_data:
        for views in stat[0]["postviews"] :
            csvwriter.writerow( [ stat[0]["date"], views["post_id"], views["views"], views["views"] ] )

3. Generate the total site views

It is possible to get this separately from the JetPack API using &table=views - but that's a lot more API calls. So we're just going to sum it up instead 😄

This, again, inserts a dummy value for visitors.

import pandas as pd

input_csv  = "wpbp_koko_analytics_post_stats.csv"
output_csv = "wpbp_koko_analytics_site_stats.csv"

column_names = ['Date', 'Post ID', 'Visitors', 'Page Views']
df = pd.read_csv( input_csv, names=column_names )

# Group by Date and sum the Page Views
df_grouped = df.groupby( "Date" )["Page Views"].sum().reset_index()

# Add a new column with a copy of the Page Views
df_grouped['Visitors'] = df_grouped['Page Views']

# Rename the Page Views column to Total Page Views
df_grouped = df_grouped.rename(columns={"Page Views": "Total Page Views"})

# Write the dataframe to the output CSV file
df_grouped.to_csv( output_csv, index=False,  header=False )

4. Upload to MySQL

You're on your own here, Sparky. If you have something like PHPMyAdmin, you should be able to load the file in directly. Anything else… good luck!

Once done, your stats dashboard should be filled with historic data.

Huge thanks to Koko Analytics for providing such a great tool and answering my questions.

New investors are often told that, when investing for the long term rather than chasing individual stocks, it is better to be invested for the longest possible time rather than trying to do "dollar cost averaging". DCA is the process of spreading out over time the purchasing of your investments. That way, you don't lose it all if the market drops the day after you invest.

Let me explain...

Imagine that it is 1994 and your rich uncle, Scrooge McDuck, has decided to gift you $1,200 per year. How generous!

He has stipulated that you must invest it in the S&P 500 - that's the top 500 companies in the world⁰.

He gives you two choices:

• Put $1,200 in on the 1st of January every year.
• Put $100 in on the 1st of the month every year.

How much money do you make in each scenario?

Get The Data

Kaggle has a download for the historic S&P 500 data. It goes from 1993 to 2024.

The data looks like this:

Experiment 1 - Time In The Market

Here's the algorithm we want to run.

1. Start in 1994
2. Set the investment as 1200
3. Get the Opening price of the first entry of the year
4. Get the Closing price of the last entry of the year
5. Calculate the percentage difference
6. Multiply the investment by the growth / fall
7. Add 1200 to the investment
8. Repeat from (3) for the next year.

Here's the code. I've made some assumptions - for example there are no trading fees, you buy at the opening price, and fractional dollars disappear. I'm aware this doesn't track perfectly but it isn't intended to; this is a rough and ready reckoner.

import numpy as np
import locale

#   Set for American currency
locale.setlocale( locale.LC_ALL, 'en_US.UTF-8' )

#   Location of the data
file_path = 'data.csv'

#   Load CSV into a NumPy array
data = np.genfromtxt(file_path, delimiter=',', names=True, dtype=None, encoding='utf-8')

# Count the number of unique years
unique_years = 0

#   Total investment
total_investment = 0

#   Yearly investment
yearly_cash = 1200

#   Start with an opening of 0
opening = 0

#   Loop through the years
for year in range( 1994, 2024 ):
    #   Add our yearly amount
    total_investment += yearly_cash

    #   Get data for the current year
    data_year = data[data['Year'] == year]

    #   Opening Price
    if ( 0 == opening ) :
        opening = data_year['Open'][0]

    #   Closing Price
    closing = data_year['Close'][-1]

    #   Percentage Change
    change = ( closing - opening) / opening
    movement = "✅ +" if change > 0 else "❌ -"

    #   Calculate change in investment
    total_investment = int( (1 + change) * total_investment )

    #   Print the running total
    print ( f"{movement}{abs(change*100):05.2f}% {year} Total investment is " + locale.currency( total_investment, grouping=True ) )

    #   Set the new opening price
    opening = closing

    #   Increment the number of years seen
    unique_years += 1

print ( f"After {unique_years} years you have invested {locale.currency( yearly_cash * unique_years , grouping=True )}" )
print ( f"Your total amount is {locale.currency( total_investment, grouping=True )}" )

It spits out:

After 30 years you have invested $36,000

Your total amount is $203,445

Experiment 2 - Dollar Cost Averaging

OK! Can we beat that with DCA? In this scenario you take your uncle's money and invest 1/12th of it on the first trading day of every month.

import numpy as np
import locale

#   Set for American currency
locale.setlocale( locale.LC_ALL, 'en_US.UTF-8' )

#   Location of the data
file_path = 'data.csv'

#   Load CSV into a NumPy array
data = np.genfromtxt(file_path, delimiter=',', names=True, dtype=None, encoding='utf-8')

#   Total investment
total_investment = 0

#   Monthly investment
monthly_cash = 100

#   Number of months run
number_of_months = 0

#   Start with an opening of 0
opening = 0

#   Loop through the years
for year in range( 1994, 2024 ):

    #   Get data for the current year
    data_year = data[data['Year'] == year]

    #   There should be 12 months in every year, but let's double check!
    unique_months = np.unique(data_year['Month'])

    #   Loop through the months
    for month in unique_months:
        #   Add our monthly amount
        total_investment += monthly_cash

        # Filter data for the current month
        data_month = data_year[data_year['Month'] == month]

        #   Opening Price
        if ( 0 == opening ) :
            opening = data_month['Open'][0]

        #   Closing Price
        closing = data_month['Close'][-1]

        #   Percentage Change
        change = ( closing - opening) / opening

        movement = "✅ +" if change > 0 else "❌ -"

        #   Calculate change in investment
        total_investment = int( (1 + change) * total_investment )

        #   Print the running monthly total
        print ( f"{movement}{abs(change*100):05.2f}% {year}/{month:02.0f} Total investment is " + locale.currency( total_investment, grouping=True ) )

        #   Set the new opening price
        opening = closing

        #   Increment the number of months
        number_of_months += 1
    #   Yearly total
    #print ( f"{movement}{abs(change*100):05.2f}% {year}/{month:02.0f} Total investment is " + locale.currency( total_investment, grouping=True ) )

print ( f"Your total amount is {locale.currency( total_investment, grouping=True )} after {number_of_months} months" )

The end result?

Your total amount is $193,891 after 360 months

That's $9,554 worse than shoving all the money in on January 1st.

Which... Look, $10k is $10k. I'd rather have it than not. But, in the context of these numbers, it doesn't feel significant. Does it?

Let's Graph It!

Conclusion

Dollar Cost Averaging is fine⁰. Over a long enough time you're probably marginally better off with lump-sum investing. But there's not much in it.

I'm sure you can construct scenarios where DCA is slightly preferential, and not every investment tracks the S&P500, and I'm sure my maths might be a little wonky, and obviously this is not financial advice. But, yeah, DCA if you want to; the difference in this example appears to be minimal.

We all know that solar panels' efficiency wilts in the heat, but do they get a tan work standing in the English rain?

At the beginning of August we installed a 4.8kWh solar battery to supplement our 5kW of solar panels.

The battery provides a CSV of readings taken every 15 minutes. It measures solar power, household usage, and battery usage. August, looked like this:

Not massively helpful. But, with a little bit of 🐍 Python and 🐼 Pandas, I worked out the following:

🏠 Our home used 290kWh

We're tracking pretty close to the UK average of about 10kWh per day. Our average of 9.4kWh each day is perhaps slightly higher than normal for a 2-person household. But we work from home regularly and have a lot of hungry smarthome gadgets.

🔌 Only 15% of our electricity came from the grid

The sun doesn't shine at night. Duh! But the battery usually provides most of our power after sunset. The battery can only discharge at a maximum of 2.4kW, I think. So if we use the electric shower, oven, or other high power appliances, then we draw from the grid.

So, what did that 44kWh cost us?

💷 Normally, we'd pay £90 for August's electricity. We only paid £14!

The price you pay for electricity depends on where you are in the UK and what tariff you're on. With a mix of solar and battery, we cut our August bill by 85%.

But what's the mix between solar direct and solar delayed?

🌞 Solar gave us 46% of our electricity

About 135kWh of our month electricity needs was met directly from solar. That means photons hit the panels, they bounced down into the inverter, and then straight into the wires in our walls, where they were gobbled up by laptops, TVs, and toasters.

That just leaves the battery…

🔋 Battery storage gave us 39% of our electricity

We used about 113kWh from stored solar. An average of 3.6kWh per day. Perhaps that means our 4.8kWh battery is over specced? I'm not so sure. Some days we use over 100% of our solar battery capacity.

Of course, not all of the solar power can get used or stored. Once the battery is full, that electricity has to go somewhere…

🔙 We sold 140kWh of solar back to the grid

Our solar power feeds into our local grid for our neighbours to use. We sell the electricity at market rates - which change every 30 minutes. This made us £13.

📈 Total cost for August's electricity? £1.

Yup! For the whole month of August, our electricity bill was £1.

(Plus the standing charge, of course!)

Disclaimer

OK, time for a little bit of a sanity check.

Firstly, these data are drawn directly from the battery. It has clamps over our import and export wires to monitor what the household is doing. These are broadly accurate - I estimate less than 2% different from what our smart meter and inverter report.

Secondly, the battery groups up the stats every 15 minutes. So, again, that's likely to introduce some errors into the data.

Thirdly, prices for both import and export can vary massively. Our export price in particular varies depending on demand.

Fourthly, these data were gathered in South-East London on an East / West split solar site. Your panels will be in a different location and will perform differently.

Fifthly, the price of panels and battery storage is high. If you can afford the up-front capital costs of an installation, I think it makes sense to do so. The payback period is usually under 10 years. But can be considerably shorter during a time of rising energy costs.

I want more stats!

Every day at sunset, my solar panels publish their generation stats to GitLab. You can download all the data from 2020 and see how much solar generation we've had.

If you need more, I published 5 years of minute-by-minute solar generation as Open Data from our previous house. This dataset has been cited in several academic papers.

I'm considering whether to release my daily usage statistics. At the moment, it feels a little invasive. You can tell when I put the kettle on in the morning, see when I load a tumble-dryer, and calculate just how long I use the oven for. Perhaps you can even analyse the overnight fluctuations and work out what model of fridge I have. I don't think you can tell what video content I'm watching because it's hidden in the noise of my other appliances. But you could probably tell if I was home or not.

Here's a typical daily graph. What do you think you can figure out from this?

So I think I'll release it in a year's time. That's a decent balance between openness and privacy.

I hope you've found this blog post useful. If you have, you can support me by:

• Switching to Octopus Energy - if you join, we both get £50. They do dynamic pricing for import and export. And, even better, they have an API so you can query your energy usage.
• Supporting OpenBenches - it's a crowdsourced site of memorial benches run by me and my wife.
• You can also buy me a book to read.

Thanks!

My suggestion is, as always, sortition.

Let me explain.

Most top flight universities around the world have the same problem. They have space for 100 students on a specific course. 15,000 apply. How do they select the best-of-the-best-of-the-best?

My answer is - they don't.

They should ignore extra-curricular activities (that tend to bias against poorer students with less free time). Ignore admission essays (which aren't read, can be creepily voyeuristic, and are probably copy-and-pasted). Ignore relevant experience (which, again, can be bought). Ignore whether someone's parents went to the school. Ignore skin colour, religion, sex, gender, credit-score, disability, and everything else¹.

Instead, find a somewhat objective method of filtering the applicants. For example, academic score².

Take the names of everyone who meets or exceeds the minimum standard. Put the names in a big hat. Pull out 100 names. Those are the students you admit. Done.

It's cheap, fast, simple, and - above all - ruthlessly fair.

Everyone who is eligible has an equal chance of getting in. There can be no accusations of favouritism, discrimination, or bias. You can't easily buy your way in, seduce a professor, or concoct a heartbreaking story to sway someone's opinion.

Might the university miss out on some stellar minds it really wanted to attract? Probably. Will the university get some undiscovered gems they would have otherwise overlooked? Probably. If the people applying to the university are broadly representative of society, will the intake be similarly representative? Probably.

I am convinced that sortition (and similar stochastic methods) are fairer way of apportioning limited resources.

Now, before you go storming my barricades, I'd like to point out some of the flaws in this scheme.

Firstly, it's probably impossible to set a completely unbiased academic test. Even if it were, rich students have always been able to buy academic tutors to help them cram and study. It also means that a talented student who has, for example, gone to a crappy school or suffered a bereavement may not meet the strict criteria.

Secondly, if a university puts all of its recruitment efforts into specific schools - they're the ones more likely to apply. For example, if they never bother to send recruiters to schools for blind students, will those students be less inclined to apply?

Thirdly, what happens if one year the admissions happen to be 100% white students? Or zero disabled students make the cut? Or no men get in? That is statistically likely to happen. And our monkey-brains just aren't wired to cope with that level of "unfairness". We want randomness which looks random, not that is random.

Fourthly, how does this make students feel? If you know that you only got in because of a dice roll, what does that do to your mental health? I notice no one seems to worry about the psychology of students who only got in on a legacy basis. Or a sports scholarship. But I think sortition might cause both imposter syndrome in those that get in, and hostility in those that didn't.

Perhaps the answer is a mix of traditional and sortition? Give 50% of places to those who can write a good essay, wow an interviewer, and generally play the game. Give the remaining 50% to those students who are merely lucky.

Of course, unless such admissions data are kept secret, there will always be an uncomfortable urge to find out who "deserved" their place and who were merely lucky. Which, in a sense, is no different to how opponents of Affirmative Action feel today.

I am quite serious about this. It seems that the two options facing universities - and anyone else oversubscribed with qualified candidates - are either to fully embrace your current discriminatory practices, or just pick people at random.

I'll go one step further. If US universities are no longer allowed to positively discriminate on any characteristic, then it strikes me that a lottery of qualified applicants is the only solution they can adopt.

I don't think the world is ready for this level of cold-blooded rationality. We want processes which feel fair and which appear reward deserving students. Sortition asks us to confront the very real possibility that our traditions and processes have been monumentally unfair.

• What is the acceptable failure rate?

For this, I had to teach them two concepts.

1. False Positives. For example, telling someone they have cancer when they don't.
2. False Negatives. For example, telling someone they don't have cancer when they do.

There is a cost associated with both of these errors. In the first case, it is the monetary cost of unnecessary treatment and the emotional cost to the patient. In the second, it is the monetary and reputational cost of getting sued for negligence as well as the emotional cost to the patient.

Here's a handy chart:

So, we're agreed! Let's eliminate false results. Luckily, that's pretty easy.

If we want to eliminate all False Positives, we just give everything a negative result.

If we want to eliminate all False Negatives, we just give everything a positive result.

Ah. Do you see the problem? We want both to be zero. But humans - and the processes they create - are fallible. There will always be an error somewhere.

So how much error is acceptable?

You may have heard of the maxim "It is better that ten guilty persons escape than that one innocent suffer." - known as Blackstone's ratio. That suggests that a high False Negative rate is an acceptable consequence of the legal system - because a False Positive is unconscionable⁰.

The problem of False results increases with the total number of results. Let's take, for example, spam email. If you are manually looking over every email you receive - say 10 per day¹ - and judging whether they are spam or not, then it's relatively easy to get zero false results.

Now imagine that you're sorting through a million emails. Is that one about Viagra spam - or is it a genuine message from your pharmacy? That one is from your mum - but it's a forwarded chain hoax - is that spam? And so on.

What's the right balance between False Positive and False Negative? Would you rather have genuine email fall into the spam trap, or occasionally see spam in your inbox?

Now apply this thinking to any large-scale process.

For example, responding to abuse requests on social media. I'm sure we've all had the experience of finding blatantly illegal or abusive content, reporting it, and getting back the message "Sorry, we don't think this violates our guidelines."

It's annoying and distressing to encounter a False Negative in the wild.

And, I'm sure many of us will have experienced being notified that our completely innocuous post somehow "went against community guidelines" and was blocked.

It's annoying and distressing to encounter a False Positive in the wild.

But the most painful thing is that we have no way of judging is how many True Positives and Negatives there were.

The Trilemma

The classic choice we are asked to make is:

1. Fast
2. Cheap
3. Reliable

• Pick any two

If we moderated our own content, it would be reliable and cheap - but it wouldn't be fast.

Things like Bayesian Filters are automated tools which compare messages to a list of known "good" and "bad" messages. They're very quick, very cheap, but their reliability is poor because spammers learn how to bypass them.

So we tend to do one of two things:

1. Outsourced labour
2. AI

Using real humans in a low-cost area is cheap. Get enough of them and they are fast. But the more you get, the more expensive it is. And the reliability is uneven.

So AI comes to the rescue. It is blazing fast. The more you use it, the cheaper it gets. And the more you correct it, the more reliable it gets.

But that reliability always comes back to the acceptable number of False Positives vs False Negatives.

I've seen some people say that social networks should have zero-tolerance for abuse. I agree with that. I'd happily ban all content that is racist, sexist, homophobic, transphobic, casteist etc².

But what's an acceptable failure rate? With a million abusive messages coming in every minute, how many is acceptable number to incorrectly mark as benign?

If you say zero, then you have to accept an increase in False Positives - innocent messages getting incorrectly blocked.

The reverse is also true.

And I think that's where people get stuck. We have to accept that failure is an option. More than that, it is an inevitability. We have to make peace with the fact that False Positives and False Negatives can only ever be reduced - not entirely eliminated.

But, there's something we blokes can be grateful for. No matter what grief the world throws at us, it'll always be the women's fault that there aren't enough babies!

The other day, I saw this headline:

The Business Insider article - in the "Economy" section - focussed solely on the lady-folk. Why do those pesky dames refuse to procreate? About half-way through, the journalist linked to the original study. Did it put the blame squarely on the female of the species?

Among parents and non-parents alike, men and women are equally likely to say they will probably not have kids (or more kids) in the future.

Ah.

It turns out, it isn't just the fault of the chicks. Us dudes (assuming you are one) are also equally to blame. Not that you'd know it from the media!

Here's a selection of recent headlines. I've deliberately selected fairly mainstream Anglosphere publications rather than, say, single-interest magazines.

And on and on it goes!

Try to find articles in those same publications - or anywhere else - which ask "Why are young men not becoming fathers?" Or "10 reasons why men aren't choosing fatherhood - you'll never guess number 7!!" Or "Gen-Z Blokes Back-off Baby-making".

Where's the article entitled "I don't want to be a dad… ever!"?

At best, you find the occasional article like this: But that's really about men who want kids, but haven't got around to it.

Or this: Which quite reasonably says:

Why, oh why, does nobody ever ask about the men?

Statistics

OK, so no one is asking men why they don't want kids. What about the statistics?

I've previously ranted about the way the UK's Office for National Statistics talks about childfree intentions. By law, the ONS has to gather data about live births. In its latest statistical bullitein, it says:

The total fertility rate (TFR) reached a record low in 2020, decreasing to 1.58 children per woman.

OK, but how many children were there per man? Is this one hyper-fertile chap running around impregnating everyone?

The average age (as measured by standardised mean age) of mothers at childbirth remained the same as 2019, at 30.7 years … Fertility rates for woman over the age of 40 decreased for the first time since 2013.

And what's the fertility rate of the father? Are young-bucks opting out? Are older daddies now the norm?

On the one hand, it is easier to gather data from the mother - there's reasonably solid evidence that the baby is hers. People can and do lie about who the sperm-donator are. But that seems to me like a weak reason to not attempt to gather the data.

Well, the good news is - the data are gathered! The ONS query builder shows age of father. Here are the age of fathers & mothers in 2019:

(Ew at the number of geriatric skeeves perving on teenagers!)

The full dataset only has 2017-2020 - and looks like this for the age of fathers over time:

Now, this doesn't talk about how many kids each man has fathered. But it is pretty clear that younger guys are dropping out of fatherhood at a higher rate than older ones. And, eyeballing it, at about the same rate as women.

Why is this presented as a woman-only problem?

Sexism.

Sorry, were you expecting a more detailed explanation? OK. Lots of sexism.

There is a stereotypical perception that men generally don't want kids. That's not true - I know loads of mates who were desperate to become fathers.

And there is an equal and opposite presumption that all women want to breed. But it is only that stereotype which seems to be the target. Men simply don't get an equal amount of grief in the media.

As people get better access to education and jobs - they see the structural advantages to the way (some) child-free people live their lives. And they see the misery inflicted on (some) people by having kids. This affects men and women equally.

As people take a rational look at both the economy and the health of the planet - they realise the monumental unfairness of bringing life into the world. Again, both men and women take a long-term rational view of their lives and the lives of potential children.

You can't escape the fact that you need a womb in order to give birth. But, as the old song goes, it takes two to tango. "Blame" for the lack of babies needs to be shared equally.

Do busier hospitals have higher survival rates? How many trees are there on the planet? Why do old men have big ears? David Spiegelhalter reveals the answers to these and many other questions - questions that can only be addressed using statistical science.

Statistics has played a leading role in our scientific understanding of the world for centuries, yet we are all familiar with the way statistical claims can be sensationalised, particularly in the media. In the age of big data, as data science becomes established as a discipline, a basic grasp of statistical literacy is more important than ever.

In The Art of Statistics, David Spiegelhalter guides the reader through the essential principles we need in order to derive knowledge from data. Drawing on real world problems to introduce conceptual issues, he shows us how statistics can help us determine the luckiest passenger on the Titanic, whether serial killer Harold Shipman could have been caught earlier, and if screening for ovarian cancer is beneficial.

Any book which starts with a link to a GitHub repo full of the code used to produce the statistics is bound to be good!

This is an excellent introduction and overview to why statistics are presented the way they are.

In some parts, the book is similar to the Feynmann report on the Challenger disaster. It takes a calm, dispassionate look at heartbreaking data to try to make sense of tragedy. Being able to take emotion and fear out of data is a core part of investigating the world. Something that isn't always easy to remember.

Spiegelhalter gently eviscerates journalists who have no concept of how stats work and just publish whichever big, eye-catching number they latch onto. But I found it hard not to sympathise with people who skim a poorly-constructed press release looking for the interesting headline.

The book makes brilliant use of modern ebook features - it separates footnotes and citations, it hyperlinks reference, and makes good use of crosslinking. That said, they are sometimes so dense that it's difficuly to navigate using a touchscreen.

Sadly, all the diagrams and charts are in black and white - an understandable cost saving for a print book, but a splash of colour might have livened up the ebook.

If you're involved in stats or data science then you probably won't find anything new in here - but it's a good reminder about how much work it takes to make statistics accessible for the lay reader.

There are some areas which it does impress though.

Scale.

Half-a-dozen years ago - during one of VR's periodic hype-phases - an employer asked me and my team to "do something interesting" with all the expensive VR kit they'd bought on a whim.

We looked at virtual store walkthroughs, simulating the view from a theatre seat, or a view of the country from above overlaid with shiny graphics. None really sparked the imagination.

Then, we had the genius idea of graphs.

Yup! Bar charts!

The tyranny of the A4 print out can't be overstated. Every graph is reduced to the same size. Differences are erased. The massive disparity between columns are squished down. Even a 65 inch, 4K TV doesn't really show scale.

But VR does!

We tried a few different things - riding roller coasters over the data, sky diving through the numbers, even rendering the data as monsters and shooting them with lasers.

But, in the end, the thing that impressed was scale.

Here's a quick demo I knocked up in A-Frame. (I don't have the original data or hoards of graphic designers to make things look halfway decent. Nor a copy of Unity. Just fill in with your imagination.)

It's a pretty OK way to visualise the data. Imagine some labels and axis on there - you can see how wandering around it would give you a sense of the data.

"Now," you tell your CEO, "gaze upwards..."

Whoa! From playing around with small numbers to suddenly being confronted by the overwhelming power of a giant number.

Look, is this the best way to visualise data? No! Probably not. Is it a good way to visualise data? Also no!

But it is good at preventing a certain class of problem when viewing data:

Most of the time, statistical graphs are static. Occasionally, they're animated. But when you need to impress on someone the magnitude of a number - sometimes VR is the answer.

In How to Make the World Add Up, Tim Harford draws on his experience as both an economist and presenter of the BBC’s radio show ‘More or Less’ to take us deep into the world of disinformation and obfuscation, bad research and misplaced motivation to find those priceless jewels of data and analysis that make communicating with numbers so rewarding. Through vivid storytelling he reveals how we can evaluate the claims that surround us with confidence, curiosity and a healthy level of scepticism. It is a must-read for anyone who cares about understanding the world around them.

This is a lovely and useful book. It contains ten simple rules for dating my teenage daughter making sense of statistics. Every day we're bombarded with hundreds of seemingly contradictory conclusions from an array of confusing statistics. How do we sort the bullish claims from the bullshit?

I was particularly impressed with the book's full throated endorsement of open data and open source. If statistics aren't public - they can't be criticised. If their methodology is secret - they can't be examined. Both are needed for a healthy statistical debate.

The thing which struck me the most was just how easy the rules are to follow. And then I was swiftly batted around the head by the realisation that I only tend to apply them to statistics I'm predisposed to disagree with. All humans are fallible - and this book is a constant reminder that we all need to up our game.

Two very mild points of criticism.

There isn't much new here. If you've been following along with the statistic discourse, or are a regular reader/listener of Tim's work. But it put together well, and provides a great overview of why we trust statistics as well as how to trust them.

Many of the statistics and anecdotes are about America. I appreciate that their hegemony looms large in our legend - but a few more stories from outside their sphere of influence would have been interesting. Perhaps, unsurprisingly, the book is being released in the USA and Canada as "The Data Detective".

Thanks to NetGalley for the review copy. The book is available in all good bookshops (and, statistically, a few bad ones) now:

It can be summed up thus:

Commentators said our startup wouldn't work. Our startup did, in fact, suceed. ∴ All commentators are wrong about whether startups will work.

This exhibits one of my favourite logical fallacies - survivorship bias. If you only look at the winners, the world looks very different from reality. It's why we think that products made in the past were of superior quality - only the ones which managed not to deteriorate over the last 50 years are with us because the crappy ones died long ago.

For us to examine whether there's any merit to ignoring the commentariat we need to look at, broadly speaking, four categories of companies:

• Commentators said it sucked, and they were wrong.
• Commentators said it sucked, and they were right.
• Commentators said it rocked, and they were wrong.
• Commentators said it rocked, and they were right.

By picking on only the first category, M.G. Siegler is committing the Texas Sharp-Shooter Fallacy.

A Texan fires a dozen shots at the side of a barn, half of them miss. He draws a target around the area that he hit and declares himself the best gun-slinger in the west.

It's tempting - far too tempting - to fall for the lie that because "they all laughed at Christopher Columbus", they are equally misdirected when laughing at us.

Every successful company had a legion of people telling them that they were doomed. Let's take a quick look at some examples and counter-examples:

You can add your own selection of companies and products to that table depending on your prejudices.

It's never a great idea to read the bottom half of the web - the idiots, trolls, sycophants, and spammers will drive you crazy. But, like the proverbial stopped clock, they are occasionally right.

Don't give up on your dreams just because some prat says you'll never make the big time. Equally, don't be deluded into thinking that if they're disparaging you, then you must be on the right path. There have been many people who were ignored, laughed at, and attacked who still lost!

He presents a pie chart showing how the police force spend its money. Take a look at it and ask yourself this question: what percentage is spent on "Employees".

Please use this poll to record your guess - answers at the end of this blog.

Pie charts have a long and noble history. They were popularised by Florence Nightingale and were hugely effective in helping politicians understand the causes of death among soldiers during the Crimean War.

As we understand more about the human brain and how we perceive shapes, it is becoming clear that pie charts are ineffective for representing complex information.

2D pie charts can still serve a useful purpose in limited circumstances. The real problem is with 3D pie charts. As far as I can tell, these abominations were popularised by Microsoft's Excel charting software.

3D charts distort the view of the data in such a way that it becomes increasingly hard to understand the information being presented. A picture being worth 1000 words, allow me to demonstrate:

So, just how bad is Surrey Police's Pie Chart? In an extremely scientific study of asking half a dozen people, they all guessed between 75% and 85%. That's quite a wide range considering it's a multi-million pound difference.

On the opposite page to the pie chart is this summary of spending.

In slightly more readable format, it is:

A few interesting things to note here.

Firstly, how do we calculate the percentages? The total spend isn't mentioned in the report (£216.90). If we use that, "Employees" accounts for 81.9% of spending.

If we take into account the gross expenditure (£207.70) the figure jumps to 87.5%.

Secondly, if we do assume that we're using the unreported total spend - there is at least 2% missing. Some of which can be explained by rounding - but I wonder what the rest of the money is spent on.

Given the above, I don't think the provided pie chart allows Surrey residents to see an accurate view of how their hard earned money is being spent.

Hopefully, this side-by-side - of the above data - will show you how 3D pie charts distort data and end up misleading their audience.

With this overlay, we can see the distortion much more clearly. The smaller sections of the chart look disproportionately larger. It's time to announce a zero tolerance crackdown on dodgy data representation.

I remember, several years ago reading a short piece of speculative fiction which postulated that China would go to war over access to women. Generations of female infanticide would leave the country with a severe gender imbalance. Hoards of men would be unable to find a wife, would become violent, and would lead their country into bitter conflicts with other countries in order to capture their women.

A simplistic approach, perhaps. There are many perceived consequences of an unchecked male population growth. But gender imbalance stemming from female infanticide is a real problem around the world.

The Indian Census has released some very detailed statistics for the country's population. This chart shows the male:female ratio in Dehli.

Now, there are several reasons why there may be fewer women than men in Delhi. Economic migration may play a part for example. What we can look at with some accuracy is the birth rate.

In most human societies, more boys are born than girls. In 2011, the global sex ratio was 984 females per 1,000 males.

In India, the rate is 940.

In Delhi? It's 866.

In rural Delhi? 809.

Delhi has 1.2 million more men than women.

By way of comparison, In the UK as a whole there are 970 males to every 1,000 females (according to the ONS mid-2007 population estimates). Of course, the UK doesn't have an exemplary record when it comes to sex crimes.

There are many factors to the continued female harassment around the world. I don't want to appear to be an apologist for sexual violence, or single this out as the sole cause for these shameful crimes. However, I can't help but wonder whether populations with such unhealthy sex ratios provide an environment which fosters a truly terrible attitude to women.

If you see that a light - or anything else - is damaged, you can scan the QR code and report the issue. There's even a phone number and vanilla URL for those who aren't quite up to speed with new technology.

There's only one slight issue - the QR code points to this site.

The fact that the landing page isn't mobile friendly is bad enough, but what's worse is that they completely fail to take advantage of sending the user to a precise URL.

What should happen is that the URL should be something like

http://qr.camden.gov.uk/light/22

There are two solid reasons for doing this.

1. It saves the user time - take them directly to where they need to go.
2. You can track where users are scanning your codes.

QR codes are free to generate, and cheap to print. You don't need to be stuck in the old way of thinking about how you link the physical world to the digital world.

Imagine if, after scanning, the user was told "You're reporting street light 22 (Bedford Square) as broken. Click here to confirm." And then, perhaps, on the next page "We'd like to stay in touch with you - please enter your details here." which could even be a Facebook / Twitter login.

As the owner of the QR, you can see exactly where and when people are scanning - at the moment, all the council knows is that one of the thousands of QR codes was scanned, but not which one.

It also means that users don't have to fiddle around on their phone's screen too much in order to report a problem. At the moment, they have to navigate through a complex site, fill in a form, and then hope they remembered which number street lamp was busted.

It could be as simple as scan - click - done. A win for users, and a nice set of analytics to monitor.

Take this recent example from Vodafone.

Aside from the obvious downsides (user doesn't know where the link will take them, if it's compatible, link looks like gobbledegook, etc) there is a rather more interesting issue.

Goo.gl - along with many other URL shortners - give everyone access to your statistics.

Visiting http://goo.gl/NdVGI+ will let you see everything.

I don't know how many messages were sent out, so it's hard to calculate a response rate.

From the map, I can see the two target areas were UK and (unsurprisingly) Greece. There are a few hits in other countries - that could just be people connecting via WiFi networks routed through that country.

There are also a few hits from iPhones - suggesting either the targeting wasn't accurate, or people are spoofing the iOS user agent.

Still, it's an interesting insight into Vodafone's marketing.

On Friday, October 30, 2009, I switched on WordPress statistics so I could get a better idea of what was popular on my site. My average traffic back then was 80 visits per day. Not bad for a backwater blog.

Since then, my writing has been getting better (I hope), my content has become more interesting, and I've had several stories which have spread far and wide.

The blog now gets ~600 visits per day.

What I find interesting is that it's some of the older posts which get the most regular traffic. This post about getting Android to run on Nokia phones is often in my "top ten" list - despite being published over three years ago.

May 12, 2012 was the busiest day with 17,186 views thanks to this article about the SIM-less Phone.

Arbitrary Milestones

I'm not sure how accurate the stats are, but since they were set up (two years after this blog started) I've had 470,034 views in total.

So, my goal is to get to the half million mark. Hopefully by the end of NaBloPoMo or, more realistically, by the end of 2012.

Here we go!

The Terence Eden of October 2009 was a busy chap! 22 blog posts! What a guy :-)

One post which caught my eye recently, was asking "What are the browser statistics for 10 Downing Street?"

Here was their answer

UK Prime Minister

@10DowningStreet

Replying to @edent@edent Top are: IE7 22%, IE8 20%, IE6 12%, Firefox3.5.3 9%, FF3.5.2 7%, FF3.0.14 5%, FF3.0.13 4%, Safari 4.0.3 4%, Chrome 2.0.172.43 2%

---

So, three years later, how have things changed?

Firstly, I asked the team behind the (still in beta) GOV.UK

Terence Eden is on Mastodon

@edent

Anyone from @GDSTeam know if they'll be releasing browser statistics - or any other user metrics? shkspr.mobi/blog/2009/10/b…

---

Sam Sharpe

@SamJSharpe

Replying to @edent@edent very rough stats (-bots -spiders): FF 28%, IE<6 10%, IE7 12%, IE8 21%, IE9+ 13%, Safari 11%, Opera 3.5% /cc @GDSTeam

---

Then, I asked for the whole of the parliament.uk space

Terence Eden is on Mastodon

@edent

Replying to @blangry@blanalive If you've got any going. Would be better to have them published regularly than on an ad-hoc basis.

---

Alex

@blangry

Replying to @edent@edent IE8 23% IE9 17% Chrome 21/22 16% FF15 7.7% IE7 6.5% Safari 4% IE6 1.6%

---

These are, as they say, to be taken with a pinch of salt.

It's interesting to see the collapse of IE6, and the huge rise in Chrome. More interesting still, is the difference between the general parliamentary site and GOV.UK. There also doesn't appear to be a significant mobile presence - unless they're rolled up into the main stats.

One thing is for sure, it would be great to see these sort of official statistics regularly. Not least to counter the "must support IE6" and "Who on Earth uses Chrome" crowds.

sigh A non-mobile site. With an Adobe Flash plugin in the top right which won't work on any iPhones. Why on Earth do marketing companies insist on pointing phones to non-mobile sites. It really bemuses me. Stations rarely have good signal (too many people leads to local network congestion) and, besides, large sites are a right pig to use on a small screen.

Stats

I've blogged several times about using Bit.ly links in QR codes. With a little bit of hacking (adding at + character to the end of the URL) we can see how many people have been scanning the code.

I don't know how many of these tickets have been printed. That might be a really good conversion rate - but I doubt it. I only noticed the QR code because someone had dropped their ticket and it landed face-down. Realistically, how many people look at the back of their tickets?

The best campaign in the world would fail if it's not put in front of an audience.