Some people are @whatever.tld and others are @cool.subdomain.funny.lol.fwd.boring.tld

I wanted to know what the distribution is of these domain names. For example, are there more .uk users than .org users?

Shut up and show me the results

You can play with the interactive data

Oh, and the large number of .gy domains is due to The Fediverse Bridge.

Getting the data

BlueSky has an open "firehose" of the data passing through it. Following the sample code I listened for public interactions - people posting, liking, or follows.

From there, I grabbed every username which wasn't on the default .bsky.social domain. I left the code running for a few days until I had over 22,000 usernames.

Note, these data are all public - although I'm not sure if users necessarily realise that. It doesn't include lurkers (people who don't interact). Some of the accounts may have been moved, banned, or deleted.

Drawing a TreeMap

I used Plotly's TreeMap library to draw a static map of all the Top Level Domains (TLD).

As you can see, .com dominates the landscape - but there are quite a few country code TLDs in there as well.

Public Suffixes

Domain names have the concepts of Public Suffixes. For example, users can register domains at .co.uk and .org.uk as well as just plain .uk. The Python tldextract library allowed me to see which domains were public suffixes, so I could attach them to their parent TLD.

I then drew a TreeMap showing this.

Note! You'll need to hack your Plotly installation to allow empty leaf nodes to get in the same style as the first map.

So what? What next?

• Not everyone from, say, Brazil will have a .br domain name - but it is fascinating to see which countries dominate.
• It might be fun to go full "Information Is Beautiful" and turn each ccTLD into its country's flag.
• Are there ethical implications of recording the fact that an account has publicly shared themselves on a social network?
• What percentage of all users have a domain name handle?

Get the code

Everything is open source on GitHub.

As pointed out by Dave a company called Nordigen was set up to provide a free Open Banking service. It was quickly bought by GoCardless who said:

We believe access to open banking data should be free. We can now offer it at scale to anyone - developers, partners and Fintechs - looking to solve customer problems.

And, I'm delighted to report, it works! As a solo developer you can get access to your own data for free via the GoCardless APIs.

You'll get back a JSON file from each of your banks and credit cards with information like this in it:

{
   "bookingDate": "2023-07-11",
   "bookingDateTime": "2023-07-11T20:52:05Z",
   "transactionAmount": {
       "amount": "-2.35",
       "currency": "GBP"
   },
   "creditorName": "GREGGS PLC",
   "remittanceInformationUnstructured": "Greggs PLC, London se1",
   "merchantCategoryCode": "5812",
   "internalTransactionId": "123456789"
}

For foreign exchange, transactions look like this:

{
   "bookingDate": "2023-10-01",
   "bookingDateTime": "2023-10-01T21:41:40Z",
   "transactionAmount": {
      "amount": "-0.82",
      "currency": "GBP"
    },
    "currencyExchange": {
      "instructedAmount": {
         "amount": "1.00",
         "currency": "USD"
      },
      "sourceCurrency": "USD",
      "exchangeRate": "1.2195",
      "targetCurrency": "GBP"
   },
   "creditorName": "KICKSTARTER.COM",
   "remittanceInformationUnstructured": "Kickstarter.com, Httpswww.kick, 1.0 U.S. DOLLAR USA",
   "merchantCategoryCode": "5815",
   "internalTransactionId": "987654321"
}

Depending on your card and the transaction type, you might also get a few more bits of metadata.

Get started at https://gocardless.com/bank-account-data/. From there, it's a case of following the quickstart guide.

A few niggles

There's a bit of bouncing around. You've got to get an API key, get the institution ID, sign in, get redirected, get an ID from the callback, then get the bank account details. And then you can get the transactions!

Oh, and the access token only lasts a short while, so you'll need to either re-auth or use a refresh token.

Bank authorisation only lasts 90 days, so you'll have to refresh your details every 3 months. That's standard across all opening banking, but a bit of a pain.

GoCardless have pretty comprehensive bank coverage but they are missing a few which you might find useful.

Because there are so many financial institution in there, you might find it difficult to work out which one you need to log in to. For example, if you have a Barclays Credit Card, which of these is the right one for you?

{
    "id": "BARCLAYCARD_COMMERCIAL_BUKBGB22",
    "name": "Barclaycard Commercial Payments",
    "bic": "BUKBGB22",
    "transaction_total_days": "730",
    "countries": [
      "GB"
    ],
    "logo": "https://cdn.nordigen.com/ais/BARCLAYCARD_COMMERCIAL_BUKBGB22.png"
  },
  {
    "id": "BARCLAYCARD_BUKBGB22",
    "name": "Barclaycard UK",
    "bic": "BUKBGB22",
    "transaction_total_days": "730",
    "countries": [
      "GB"
    ],
    "logo": "https://cdn.nordigen.com/ais/BARCLAYCARD_COMMERCIAL_BUKBGB22.png"
  },
  {
    "id": "BARCLAYS_BUSINESS_BUKBGB22",
    "name": "Barclays Business",
    "bic": "BUKBGB22",
    "transaction_total_days": "730",
    "countries": [
      "GB"
    ],
    "logo": "https://cdn.nordigen.com/ais/BARCLAYS_WEALTH_BUKBGB22.png"
  },
  {
    "id": "BARCLAYS_CORPORATE_BUKBGB22",
    "name": "Barclays Corporate",
    "bic": "BUKBGB22",
    "transaction_total_days": "730",
    "countries": [
      "GB"
    ],
    "logo": "https://cdn.nordigen.com/ais/BARCLAYS_WEALTH_BUKBGB22.png"
  },
  {
    "id": "BARCLAYS_BUKBGB22",
    "name": "Barclays Personal",
    "bic": "BUKBGB22",
    "transaction_total_days": "730",
    "countries": [
      "GB"
    ],
    "logo": "https://cdn.nordigen.com/ais/BARCLAYS_WEALTH_BUKBGB22.png"
  },
  {
    "id": "BARCLAYS_WEALTH_BUKBGB22",
    "name": "Barclays Wealth",
    "bic": "BUKBGB22",
    "transaction_total_days": "730",
    "countries": [
      "GB"
    ],
    "logo": "https://cdn.nordigen.com/ais/BARCLAYS_WEALTH_BUKBGB22.png"
  },

But, overall, it's an excellent service. Now I just need to find / write something to ingest the data and do something with it!

The full report quotes lots of people - not just me! - and is mostly about "Live data" i.e.

data that is transparent, readily accessible, and shared widely – is a prerequisite for innovative, sustainable, and human-centric enterprises.

Naturally, there are a wide range of opinions in there. But I'm heartened that Open Data is seen as critical for Net Zero. And it is great to see open APIs and Open Source getting a mention.

You can download the Digital Radar Report to read it for yourself.

The Splitline Algorithm is, conceptually, very simple:

1. Divide the entire map in half based on population.
2. Repeat (1) for each half.
3. Once you have reached the target number of voters in each segment, stop.

There's an excellent paper about how to apply this to the USA's districts.

But, I couldn't find anything which applied the algorithm to the UK. So I thought I'd give it a go for fun².

First, start with the Office for National Statistics' Population Density Map

Before we even begin, there are a few obvious issues. Firstly, the map isn't contiguous. So what happens to the Shetland Isles (population 22,000) and the Isle of Wight (population 140,000)? There are, on average, 100,000 people to every MP³. Do Shetland Islanders want to be lumped together with 78k other people from the mainland who might not necessarily share their values? Do the people from the Isle of Wight want to be split in half with one half being tied to non-islanders?

The second (related) issue is NI. I'm not going to get into the long history there⁴. But I think it is fair to say that an algorithmic segmentation might cause a few raised eyebrows. So I'm going to concentrate on England, Scotland, and Wales for this section.

Here's a really naïve (and inaccurate) split based on eyeballing it.

Applying the algorithm again, and we can split the area into four equal population parts:

Repeat a few hundred times and you have equal population constituencies.

You could slice the country into long horizontal strips. That would be equal - but not necessarily practical.

OK, that's enough mucking around, time to try applying it for real.

Getting the data

The first question is what resolution of population do you want? Using country-level population density isn't fine-grained enough. Using existing constituency data is just going to replicate the existing boundaries. Like this:

So... street level?

ONS have local authority level data - which isn't quite granular enough for my purposes.

Instead, I downloaded a 1.2GB CSV from Data for Good at Meta (previously Facebook). The data looks like this:

"Lat","Lon","Population"
"50.62069444448494","-2.023750000001619","0.785894169690091"
"54.91486111115504","-1.378472222223325","3.3208914089403367"
"52.725416666708846","0.11152777777786699","1.116925979478443"
"52.72736111115329","0.12402777777787699","1.116925979478443"
"52.779583333375555","0.100694444444525","1.3609065999360417"

They have a GeoTIFF which renders the whole of the UK like this:

Zooming in to Edinburgh, shows the city is well-populated but not the countryside around it.

London, however, is dense. With occasional pockets of emptiness.

(Notes to self, to make a GeoTIFF into a browseable web map, run:

gdal_translate -of VRT -ot Byte -scale population_gbr_2019-07-01.tif temp.vrt

Then:

gdal2tiles.py temp.vrt

Finally, change to the newly generate directory and run python3 -m http.server 9000 and - hey presto - web maps!)

Python and Pandas Oh My!

import pandas as pd
df = pd.read_csv("population_gbr_2019-07-01.csv")
total_population = df['Population'].sum()

Gets us a total population of 66,336,531. Which looks right to me! Let's say we want 100,000 people (not voters) per constituency. That'd give us 663 areas - which is about what the UK has in the House of Commons⁵.

OK, which way do we want to split these data? A proposal by Brian Langstraat suggests splitting only in the horizontal and vertical directions.

First, let's sort the data South to North.

df = df.sort_values(by = 'Lat', ignore_index = True)

Which gives us

                Lat       Lon  Population
0         49.864861 -6.399306    0.573312
1         49.868194 -6.393472    0.573312
2         49.874306 -6.369583    0.573312
3         49.884306 -6.342083    0.573312
4         49.886528 -6.341806    0.573312
...             ...       ...         ...
19232801  60.855417 -0.886250    0.109079
19232802  60.855417 -0.885694    0.109079
19232803  60.855417 -0.885417    0.109079
19232804  60.855694 -0.884861    0.109079
19232805  60.855972 -0.884028    0.109079

Now we need to add up the Population until we reach total_population / 2 - that will tell us where to make the first cut.

half_population = total_population / 2
index = 0
cumulative_total = 0
for x in df["Population"] :
     if (cumulative_total >= half_population):
             print(str(index))
             break
     else :
             cumulative_total += x
             index += 1

That tells us that the row which is halfway through the population is 8,399,921.

df.iloc[8399921]

Gives us a Latitude of 52.415417 - which is Huntington. So a properly bisected map of the UK's population looks like:

50% of the population live above the black line, 50% live below it.

Let's take the top half and split it vertically.

df = df[8399921:]
df = df.sort_values(by = 'Lon', ignore_index = True)
total_population = df['Population'].sum()
half_population = total_population / 2
index = 0
cumulative_total = 0
for x in df["Population"] :
     if (cumulative_total >= half_population):
             df.iloc[index]
             break
     else :
             cumulative_total += x
             index += 1

Which gives us 52.675972,-2.082917 - an industrial estate in Wolverhampton.

In this map, 25% of the total population live to the East of the black line, and 25% to the West:

And this is where we start to see one of the problems with the naïve splitting algorithm. A chunk of Aberdeen has been sliced off from its neighbours. We can see that there will be a likely constituency of Shetlands, a bit of Aberdeen, and a slice of North-East England. These may not share common needs!

Straight-line slicing bisects otherwise "natural" groupings of people. Sure, gerrymandering is bad - but this sort of divvying up makes for the strangest bedfellows.

Shortest Splitline

The Shortest Splitline Algorithm doesn't is similar to the above but, rather than restricting itself to vertical and horizontal lines, looks for the line with the shortest distance which contains 50% of the population.

A Different Approach - South Up Algorithm

Let's just start at the bottom left of the map and work our way up.

Here's the South West (Scilly Isles not shown⁶.):

Let's plot everything, just to make sure the data are all there:

import matplotlib.pyplot as plt
df.plot(x="Lon", y="Lat", kind="scatter", c="black")
plt.show()

OK! Let's grab the first 100,000 people.

df = df.sort_values(by = ['Lat', 'Lon'], ignore_index=True)
target = 100000
index = 0
cumulative_total = 0
for x in df["Population"] :
     if (cumulative_total >= target):
             df.iloc[index]
             break
     else :
             cumulative_total += x
             index += 1

area = df[:index]

area.plot(x="Lon", y="Lat", kind="scatter", c="black")
plt.show()

Which results in:

Hurrah! Scillies and South West England! Exactly 100,000 people live in that area.

Let's do the next few in different colours

df = df.sort_values(by = ['Lat', 'Lon'], ignore_index=True)
df['Colour'] = pd.Series(dtype="U") # Add a Unicode string column for colour
target = 100000
index = 0
cumulative_total = 0

#   There is probably a much more efficient way to do this loop
for x in df["Population"] :
     if (cumulative_total <= target):
             df.Colour.iloc[index] = "mistyrose"
     elif (cumulative_total > target and cumulative_total <= target * 2):
             df.Colour.iloc[index] = "peru"
     elif (cumulative_total > target * 2 and cumulative_total <= target * 3):
             df.Colour.iloc[index] = "mediumpurple"
     elif (cumulative_total > target * 3 and cumulative_total <= target * 4):
             df.Colour.iloc[index] = "olivedrab"
     elif (cumulative_total > target * 4):
             break
     cumulative_total += x
     index += 1

area = df[:index]

area.plot(x="Lon", y="Lat", kind="scatter", c="Colour")
plt.show()

That's (mostly) going South to North, so we get those unnatural looking stripes which have weird incongruent chunks.

Bubble Split Algorithm

Rather than drawing lines, let's use a "Bubble Split" approach. Starting in, for example, the most South Westerly point in the dataset and then growing to its neighbours until it hits a population of 100,000.

This will use's SciPy's KDTree Algorithm

from scipy import spatial
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
df = pd.read_csv("population_gbr_2019-07-01.csv")
df = df.sort_values(by = ['Lat', 'Lon'], ignore_index=True)
points = df[["Lat", "Lon"]].to_numpy()
kdtree = spatial.KDTree(points)

# To find the nearest neighbour of a specific point:
kdtree.query( [59.1,-6.2] )[1]

counter = 1
population = 0
target = 100

while (population <= target):
   nearest_index = kdtree.query( [59.1,-6.2], [counter] )[1]
   population += df.loc[nearest_index, "Population"].values[0]
   counter += 1

population

Looping through is very slow and crawls to a halt after a few thousand iterations. So let's cheat. This grabs the nearest million points and finds their total population.

nearest_million = kdtree.query( [59.1,-6.2], 1000000 )[1]
df["Population"].iloc[ nearest_million ].sum()

There's no way to iterate through the results, so its easiest to grab a bunch and iterate through that instead.

counter = 0
population = 0 
target = 100000

while (population <= target):
    end     = (counter + 1) * 10000
    start   =  counter * 10000
    population += df["Population"].iloc[ nearest_million[start:end] ].sum()
    print("On " + str(end) + " Pop: " + str(population))
    counter += 1

These can now be plotted using:

indices = kdtree.query( [59.1,-6.2], end )[1]
to_plot = df.iloc[ indices ]

KDTrees are not designed to be altered - so deleting nodes from them is impossible. Instead, the nodes have to be deleted from the data, and then a new KDTree constructed.

index_to_delete = kdtree.query( [59.1,-6.2], end )[1]
df = df.drop(index = index_to_delete)
points = df[["Lat", "Lon"]].to_numpy()
kdtree = spatial.KDTree(points)

Bounding Boxes

Drawing a box around some points is useful. It provides a geographic border and also means we don't need to worry about map colouring algorithms.

For this, we'll use SciPy's ConvexHull algorithm:

import matplotlib.pyplot as plt
from scipy.spatial import ConvexHull, convex_hull_plot_2d
import numpy as np

indices = kdtree.query( [x,y], end )[1]

area = df.iloc[ indices ]

s_array = area[["Lat", "Lon"]].to_numpy()
hull = ConvexHull(s_array)
plt.plot(s_array[:,0], s_array[:,1], 'o') # Remove this to only display the hull

for simplex in hull.simplices:
    plt.plot(s_array[simplex, 0], s_array[simplex, 1], 'k-')

plt.show()

Here's the result - can you spot what I did wrong?

Putting it all together

This scrap of code reads the data, sorts it, constructs a KDTree, starts at the South West tip, finds the 100,000 people nearest to that point, and draws a bounding box around them:

#   Import the libraries
import pandas as pd
import matplotlib.pyplot as plt
from scipy import spatial
from scipy.spatial import ConvexHull, convex_hull_plot_2d
import numpy as np

#   Read the data
df = pd.read_csv("population_gbr_2019-07-01.csv")

#   Sort the data
df = df.sort_values(by = ['Lat', 'Lon'], ignore_index=True)

#   Create KDTree
points = df[["Lat", "Lon"]].to_numpy()
kdtree = spatial.KDTree(points)

#   Most South Westerly Point
sw_lat, sw_lon = points[0]

#   Get first 100,000 people
counter = 0
population = 0 
increment = 5000
target = 100000

while (population <= target):
    end     = (counter + 1) * increment
    start   =  counter * increment
    population += df["Population"].iloc[ nearest_million[start:end] ].sum()
    print("On " + str(end) + " Pop: " + str(population))
    counter += 1

#   Get the index numbers of the points with 100,000 people
indices = kdtree.query( [sw_lat, sw_lon], end )[1]

#   A separate DataFrame for drawing
to_plot = df.iloc[ indices ]

#   Flip Lat & Lon, because Lon is the X Co-ord, Lat is Y Co-ord
plot_array = to_plot[["Lon", "Lat"]].to_numpy()

#   Calculate the bounding box
hull = ConvexHull(plot_array)

#   Plot the points
plt.plot(plot_array[:,0], plot_array[:,1], 'o') # Remove this to only display the hull

#   Draw the hull
for simplex in hull.simplices:
    plt.plot(plot_array[simplex, 0], plot_array[simplex, 1], 'k-')

#   Display the plot
plt.show()

Which produces:

Running that a few more times gives this (sorry for chopping off the Scilly Isles):

Can you see why I call this "Bubble Split"?

Already we can see the limits to this approach. The orange-coloured subdivision has a little incongruent bit across the estuary of the River Fal.

Here's the (hugely inefficient and slow) code to generate 40 areas of roughly 100,000 people:

#   Import the libraries
import pandas as pd
import matplotlib.pyplot as plt
from scipy import spatial
from scipy.spatial import ConvexHull, convex_hull_plot_2d
import numpy as np

def get_100k_people(df, nearest_million) :
    counter = 0
    population = 0 
    increment = 1000
    target = 100000

    while (population <= target):
        end     = (counter + 1) * increment
        start   =  counter * increment
        population += df["Population"].iloc[ nearest_million[start:end] ].sum()
        #print("On " + str(end) + " Pop: " + str(population))
        counter += 1

    return end

def plot_hull(df, indices) :
    #   A separate DataFrame for drawing
    to_plot = df.iloc[ indices ]

    #   Flip Lat & Lon, because Lon is the X Co-ord, Lat is Y Co-ord
    plot_array = to_plot[["Lon", "Lat"]].to_numpy()

    #   Calculate the bounding box
    #hull = ConvexHull(plot_array)

    #   Plot the points
    plt.plot(plot_array[:,0], plot_array[:,1], 'o', markersize=1) # Remove this to only display the hull

    #   Draw the hull
    #for simplex in hull.simplices:
    #    plt.plot(plot_array[simplex, 0], plot_array[simplex, 1], 'k-')

#   Read the data
df = pd.read_csv("population_gbr_2019-07-01.csv")

#   Sort the data
df = df.sort_values(by = ['Lat', 'Lon'], ignore_index=True)

for areas in range(40):
    #   Create KDTree
    points = df[["Lat", "Lon"]].to_numpy()
    kdtree = spatial.KDTree(points)

    #   Most South Westerly Point
    sw_lat, sw_lon = points[0]

    #   Get the nearest 1 million points
    nearest_million = kdtree.query( [sw_lat, sw_lon], 1000000 )[1]

    #   How many points contain a cumulative total of 100k people
    end = get_100k_people(df, nearest_million)

    #   Get the index numbers of those points
    indices = kdtree.query( [sw_lat, sw_lon], end )[1]

    #    Draw
    plot_hull(df, indices)

    #   Delete used Indices
    df = df.drop(index = indices)
    df = df.reset_index(drop = True)

#   Display the plot
plt.show()

Other Choices

I made the rather arbitrary choice to start in the South West and proceed Northwards. What if, instead, we start with the point with the lowest population density and work upwards.

Here's a video of the sequence:

As you can see, it starts off pretty well, but the final few areas are randomly distributed throughout the map. I kinda like the idea of a meta-constituency of small villages. But I'm not sure if that's practical!

This next video starts with the highest population density and works downwards:

Terence Eden is on Mastodon

@edent

Replying to @edentHere's the reverse.
Start at the most densely populated point.
Draw a randomly coloured shape around the ~100,000 nearest people.
Find the next most dense point which hasn't already been drawn.
Repeat ~650 times.
This would be a *terrible* system for creating constituencies. pic.x.com/uqifiakbmp

---

Terence Eden is on Mastodon

@edent

Replying to @edentHere are the two different approaches. Click for massive.

Left starts at the lowest density.
Right starts at the highest density.

Fascinating to see where they diverge, and which bits look more "natural".
Anyway, go play with maps, data, & algorithms. It's fun! pic.x.com/dpcbsbymsb

---

NB, there's no guarantee that the generated images will have dimensions be divisible by two - so here's some hacky ffmpeg code to crop the images! ffmpeg -framerate 1 -pattern_type glob -i '*.png' -c:v libx264 -r 30 -pix_fmt yuv420p -vf "crop=trunc(iw/2)*2:trunc(ih/2)*2" output.mp4

Or, to scale the video to 1280 wide ffmpeg -framerate 1 -pattern_type glob -i '*.png' -c:v libx264 -r 30 -pix_fmt yuv420p -vf scale=1280:-2 output.mp4

Algorithms aren't neutral

It's tempting to think that computer code is neutral. It isn't. Even something as seemingly innocuous as choosing a starting point can cause radical change. It may be aesthetically pleasing to draw straight lines on maps - but it can cause all sorts of tension when communities are divided, or combined, against their will⁷.

It's a fun exercise to take population density data and play around with it algorithmically. It shows the power and the limitations of automated decision making.

Usually, this causes technologists to roll their eyes. They have a vision of a CEO grandly staring at a giant projection screen, watching the pretty graphs go up and down, and making real-time decisions about Serious Business. Ugh! What a waste of time!

The thing is - that's not what a dashboard is for. And that's generally not why a CEO wants it.

A dashboard shows that you have access to your data. And that is a huge deal.

If you are able to successfully build a dashboard, that means you have demonstrated that you have the ability to get:

• Live data
• Historic data
• Comparative metrics
• Access to multiple databases
• External data sources
• ...and a dozen more things.

Most data are locked away. Either in Excel sheets, or behind a dozen different logins for a dozen different systems - none of which can talk to each other. The data (if they are even kept at all) are each in a different format and mildly incompatible with each other.

A dashboard isn't there to be used. It is there to prove that the data are easily accessible, comparable, and trackable. Only once that is done can they be actionable.

Trapped data is useless data.

It's the same reason that some people launch Twitter-bots at hackathons (myself included!). Is there any great skill in making a data feed push a snippet of text to an API? Of course not - it's trivial! But it is a reasonably powerful statement that a novice can gain access to several different data-sources in an afternoon and do something - anything - with them.

Yes, I'm sure there are some Board Members who wake up every day and track whether the EBITDA correlates with the humidity in the call centre. And, no doubt, there's a CFO somewhere whose pulse races when they see a figure dip bellow two standard-deviations of some metric. But try to think of a dashboard not as an interface, but as tangible proof that data can be created, stored, retrieved, and compared.

People used to pay-per-minute for telephone calls. Some numbers were "zero-rated". That is, if you called them you wouldn't be charged. At first⁰ it was calls to the emergency services which were free. Businesses and other organisations realised that it was good customer service to provide a free-to-call number. Generally speaking, this means that the called-party pays the phone company for incoming call rather than the caller paying¹. Thus 0800² numbers were born.

Why don't we have something similar for the mobile web? Lots of people pay per MB for data - or have limited data caps. Wouldn't it be nice if sites could say "don't charge the user for this - charge our site instead!"?

Clearly, what's needed is a new HTTP Header. Something like x-no-data-charges.

Simples‽

We Already Have This

Last year, I had a small part to play in the scheme to zero-rate data for UK mobiles accessing the NHS websites:

Terence Eden is on Mastodon

@edent

You can now access most NHS services from your mobile, with no data charges.
In awe of the team who pulled this together very quickly.
gov.uk/government/new…gov.uk
Mobile networks remove data charges for online NHS coronavirus advice
Free access to online NHS services will be available for as long as coronavirus (COVID-19) remains widespread in the UK.

---

Later, this was extended to several important charities.

In these cases, the mobile networks got together and agreed to waive the charges. The sites didn't need to do any special configurations at their end³.

Many years ago, when Facebook still pretended that it cared about bringing free Internet to the developing world, it offered poor countries free access to Facebook. And maybe a few other sites.

I tried to get this blog added to the free programme - but nothing ever came of it.

Terence Eden is on Mastodon

@edent

Does anyone know how I get my site zero-rated on @internet_org?

---

Similarly, lots of mobile networks in the UK will give you UNLIMITED NETFLIX which doesn't eat into your regular data cap. How does a Netflix competitor get on to that deal? Pay up! Do big deals with big companies. Small fry need not apply.

There are problems with this.

Here's what happened when Wikipedia and Facebook were made free-to-browse in Angola:

Wikimedia and Facebook have given Angolans free access to their websites, but not to the rest of the internet. So, naturally, Angolans have started hiding pirated movies and music in Wikipedia articles and linking to them on closed Facebook groups, creating a totally free and clandestine file sharing network in a country where mobile internet data is extremely expensive.

Motherboard - "Angola’s Wikipedia Pirates Are Exposing the Problems With Digital Colonialism"

Could this happen here?

Imagine that *.NHS.uk was zero-rated for mobile users. Is there a tiny cottage hospital running an outdated webserver somewhere on the NHS estate? I'll bet there is⁴. How long before pirates would start abusing that service? Sure, the infosec teams would shut it down quickly. But that's a hassle.

What about the embedded content on a page? No one hosts their own videos. So do we have to zero-rate YouTube? What about the CDNs serving all the CSS and JavaScript?

Who pays?

What does it realistically cost to deliver a MB of data? Over a satellite phone, about US$245 for 100MB. No webmaster wants to pay that!

How would you even invoice for this properly? It's hard⁵ to abuse the phone network so much that you'd bankrupt an 0800 number. But I'm sure sender-pays-data would be abused the instant it was turned on.

Is this a good idea?

No. But sometimes it's fun to run with an idea to see how awful it is.

Imagine that you work at a company which sells widgets. Each widget has a unique serial number. The number is a fixed length, and can contain leading zeros. That is, the following are all valid identifiers:

• 00001
• 01010
• 12345

What data type would you use to store these data in a database?

This is one of those strong opinions, weakly held. I'm not sure there's a right answer to it. A quick survey on Twitter⁰ was inconclusive:

Terence Eden is on Mastodon

@edent

You want to record a fixed-length number in your database (i.e. always 5 digits long).
The number can have leading zeros (e.g. 00123).
What data type do you use?

---

Numeric with ZEROFILL: (61)
61
Varchar or Text: (94)
94
Something else (what?): (12)
12

---

Here, I attempt to provide some of the opinions on both sides of the argument. Feel free to supply your own arguments.

It isn't a number

Can you do maths on it? If not, it isn't a number. It's a text string that happens to only contain numbers.

For example, asking "what is the average of these three serial numbers?" is a meaningless question. "What is this serial number divided by two?" again - that has an answer with no semantic content.

Counterpoint

What if the serial has some semantic content in it? For example even numbers were made in one factory, odds in another. Or prime numbers are demonstration units. It might be easier to grab information from a number type. But it's probably better to pull that data into separate fields.

It isn't guaranteed to always be a number

If this field was for the item's price, it's is always going to be a number. Sure, it might be a float or an int - but it'll be a number. Same if this was the item's height, width, or weight. Those values can only be numbers.

But there's nothing inherently "numbery" about a serial number. At any point, the boss could recommend adding Greek letters to it.

Given we can't do maths on it, and the "number" doesn't have any semantic content, there's no need to artificially restrict our database to a "number" type.

Counterpoint

The same is broadly true of any constraint. The maximum length could change. The validation rules might be updated. Modern databases need to be able to cope with changes in business requirements.

Technical Constraints

I don't know of any number type which stores leading zeros. Please enlighten me if I'm wrong. That means any data retrieved from the database has to be formatted before displaying it to the user.

That also means that searching the database requires data to be pre-formatted.

Counterpoint

All data should go through validation and sanitation before being displayed to the end user. Numbers aren't special in that regard. Similarly, data should be carefully checked before being searched for.

Numbers get misformatted

We all know how Microsoft Excel will look at any number and try to interpret it as a date. It also has a tendency to strip leading zeros. And some formatters will automatically add thousands separators to any number they see. Keeping the data as text reduces the risk of this happening.

Counterpoint

Excel will mangle anything that looks even vaguely like a number. Storing as text does not guarantee that it will be interpreted as text.

OK - I think that's the majority of the argument for not treating this as a number. What are the arguments on the other side?

Incorrect Data

If we allow text in this field, what happens if someone types a letter O rather than a number 0? Having this be an int prevents these errors creeping in. Yes, have checks at the front end, but this provides defence in depth.

Counterpoint

As above. Data should be checked by the application before submitting to the database. The database should be checking the data before storing it.

Searching and Sorting

Suppose we want to get all widgets with a serial number >= a specific ID. A number type is much better than string for those sorts of operations.

Counterpoint

This again assumes that the IDs have some semantic content. For example, 00012 was manufactured after 00008. This may or may not be the case. Such operations are best performed on a field like "Manufactured Date".

It's more efficient

It is quicker and computationally cheaper to store integers rather than text. Searching is faster, disk space requirements are lower.

Counterpoint

It isn't the 1970s. We're not paying per bit. Unless we're storing billions of rows, or working on constrained hardware, this isn't a practical concern for most users.

Human Usability

It looks like a number. A human, on encountering one of these IDs is going to assume it is a number. Anything which makes it harder for humans to understand is going to cause problems.

Counterpoint

Things like phone numbers look like integers, but they aren't. Momentary human confusion is preferable to mangled or imprecise data.

So now what?

I lean to the side that says that this is a string with specific constraints. Namely /^[0-9]{5}$/ (please supply your own regex meme).

When a user enters a new serial number, it should be checked that it meets the constraints. If it doesn't, refuse to submit it until the user has formatted it correctly. When submitted, the database should check against the constraints, and refuse to accept non-matching strings.

The speed of searching and sorting is not meaningfully degraded by storing as a string.

But, I'm very aware that I could be wrong. This is my strong opinion, but if you can supply a better argument, I will drop my weak grip on it.

Chaotic Good

Daniel Knell

@danielknell

Replying to @edent@edent 5x bigint fields, one for each digit.

---

About 4 years ago, I wrote about Visualising Twitter Conversations in 2D Space. Based on an idea by Lucy Pepper, I built a quick hack to show what Twitter threads actually looked like.

Well, lockdown finally got the best of me, and I finished the project!

TweeView.ml

Here are a few screenshots of "interesting" trees, and a little bit about how it works.

In this example, the user has a main thread that they are posting on: You can see that a few people reply to some of the earlier tweets in the thread.

In this example, the original user only posts once - but some other users have a pleasant chat in the thread. A common pattern is that there are lots of first-level replies - but very little discourse. That's not always the case - here's the Telegraph posting a story to Twitter.

While most people are happy to just comment, a few people have sustained conversations. It can get a little fanatic - I've rotated this sideways to make it slightly easier to view: Two people with very different political opinions have a civilised and productive discussion...

In this rare case, a 2nd- and 3rd-level comment attracts more attention than the initial Tweet.

And, just for fun(!) here's a ridiculously long thread:

Political Animal

@politic_animal

Things no-one else is crazy enough to be doing right now: final prep for a trip to answer one of the great unanswered questions: just how far can you travel by bus from London in 24hrs? I *think* I know on paper, but in practice…? Follow along from 3am as I slowly go mad. #bus24 pic.x.com/ga8hzywgp6

---

What does it look like?

To have a play, visit TweeView.ml and paste in a recent Twitter thread.

You can also look at these three pre-rendered demos:

• Small thread
• Complex thread
• Huge thread

Notes

• This is a demo.
• It is read-only. You can't send messages and it won't ask you to log in.
• Limitations in the Twitter API means it only works on Tweets in the last 7 days.
• This doesn't cope with threads "orphaned" after a parent Tweet has been deleted.
• The code is open source, and built on lots of open source components, GitHub.com/edent/TweeView.

Enjoy!

So, what's going on? As I've previously blogged about, Twitter has a new conversations API. That allows you to get Tweet replies in a tree-like structure.

A small conversation looks something like this:

You can see that the initial tweet gets a lot of attention - many first level replies. Then, some of those tweets spawn other, longer threads.

People occasionally delete their tweets. This means some branches of the conversation are orphaned. Which gives us this wonderful set of constellations.

In this example, the more retweets a tweet has - the larger its radius.

In this example, there are a chain of Tweets. While the first Tweet in the chain has the most replies and retweets - the rest of the thread is also heavily trafficked. Those Tweets on the right get plenty of love!

Finally, here's a thread of over 200 tweets from a popular account:

The first couple of tweets attract a lot of attention, and some parts of the conversation branch off into their own little trees.

You can play with these visualisations:

• Small 3D
• Complex 3D
• Ridiculous 3D
• Complex 4D
• Ridiculous 4D

A few points to note:

• This is a demo. It may break entirely or perform badly.
• The code is open source, and built on lots of open source components, GitHub.com/edent/TweeView.
• The Twitter conversation API only works on tweets sent in the last 7 days. So you can't use this on older threads.
• Large conversations take a long time to load and can be slow to render.

Enjoy!

Zachary Powell 🥑

@devwithzachary

A wild @gerardsans and @edent (winner of best t-shirt for sure) pic.x.com/Oc3KTPswvZ

---

Zachary Powell 🥑

@devwithzachary

Thank you very much @edent was great looking at how we could visualise tweets in more than 1D (4D twitter anyone?) #TapIntoTwitter pic.x.com/moMUQLe3t0

---

This is my paper from the Data Analytics module. I enjoyed it far more than the previous module.

This was my second assignment, and I was amazed to score 72%. In the English system 50% is a pass, 60% is a commendation, 70% is distinction. Nice!

A few disclaimers:

• I don't claim it to be brilliant. I am not very good at academic-style writing. I was marked down for over-reliance on bullet points.
• This isn't how I'd write a normal document for work - and the numbers have not been independently verified.
• This isn't the policy of my employer, nor does it represent their opinions. It has only been assessed from an academic point of view.
• It has not been peer reviewed, nor are the data guaranteed to be an accurate reflection of reality. Cite at your own peril.
• I've quickly converted this from Google Docs + Zotero into MarkDown. Who knows what weird formatting that'll introduce!
• All references are clickable - going straight to the source. Reference list is at the end.

And, once more, this is not official policy. It was not commissioned by anyone. It is an academic exercise. Adjust your expectations accordingly.

Abstract

We describe a method of visualising change of data over time using an animated TreeMap. This is used to display how government policies affect the type of files that government departments publish.

This research forms part of a MSc project to better analyse the data and metadata generated by the UK government. It will inform future storage space requirements and what incentives are needed to help the government meet its Open Government Partnership commitments.

This project was brought about following the Prime Minister and Cabinet's recent commitment to ensure that data are published in formats which are as open as possible (Johnson, 2021).

The use of time-series visualisations forms an essential part of our department's ability to monitor the effectiveness of our policies and guidance in an intuitive way.

The end result is a short video which shows the volume of files uploaded over time, and whether they meet our department's standards for openness.

1. Business Challenge Context

The UK Government publishes tens of thousands of documents per year. Each government department has responsibility for their own publishing. The Data Standards Authority (DSA) is responsible for ensuring that documents are published in an open format.

There is growing concern about the number of documents being published as PDF files (Williams, 2018).

The following questions have been identified as important to the organisation:

• How many different document formats are regularly used?
• How is the ratio between open:closed changing over time?
• Are the number of PDF files published increasing or decreasing?
• Which departments publish in non-standard formats?
• How can we visualise the data?

Answering these questions will involve analysing a large amount of data to produce a visualisation which can be used by the organisation to improve its business practices and quality of output.

The analysis is undertaken with the understanding that retaining the trust of our users and community is paramount (Véliz, 2020).

2. Data Analytics Principles

How key algorithms and models are applied in developing analytical solutions and how analytical solutions can deliver benefits to organisations.

Machine Learning (ML) provides the following key benefits:

• ML allows us to dedicate cheap and reliable computational resources to problems which would otherwise use expensive human resources of inconsistent quality.
• For example, ML can be used to extract text from documents and perform sentiment analysis to determine how they should be classified (Arundel, 2018).

There are two main models of ML:

• Supervised learning:
  • If data has already been gathered and classified by humans, it can be used in ML training.
  • A portion of the data is used as a training set - for the ML algorithm to process and derive rules.
  • A random sampling of the data is withheld from the training set. The ML process is run against this validation set to see if it can correctly predict their classifications.
• Unsupervised learning:
  • Uses unclassified data to discover inherent clustering patterns within the data.
  • There are multiple clustering algorithms, each with a different bias, and so choosing the correct one can be difficult (Xu and WunschII, 2005).

In both cases, ML seeks correlation between multiple variables:

• Pearson correlation coefficient assigns a value between -1 and +1 based on the relationship between two variables, divided by the sum of their standard deviations (Pearson, 1895)
• Linear Regression attempts to find a "best fit" linear relationship, usually via a sum of squared errors.
• With any results it is important for analysts to understand that correlation does not imply causation.

Another model is Big Data:

• Although facetiously referred to as "larger than can fit in excel" (Skomoroch, 2009), Big Data usually refers to the "Four Vs" of Velocity, Volume, Variety, and Veracity (Kepner et al., 2014).
• While Big Data can be helpful in gaining insights, complex approaches such as MapReduce (Dean and Ghemawat, 2008) are needed to efficiently analyse datasets using parallel processing.
• As a government department, we need to be mindful of the dangers to the public caused by Big Data (Zuboff, 2020).

All algorithms suffer from learning bias:

• Systemic bias is present when data are captured, when decisions are made about how to classify data, and when data are analysed (Noble, 2018).
• Failure to correct for this bias could be unlawful (Equality Act, 2010).
• Finally, we must recognise that a training dataset will always be inferior to the full data record. This is commonly known as the "no such thing as a free lunch" problem (Wolpert and Macready, 1997).

2. The information governance requirements that exist in the UK, and the relevant organisational and legislative data protection and data security standards that exist. The legal, social and ethical concerns involved in data management and analysis.

The UK Government is bound by several pieces of legislation. The following are the key areas which impact data management and analysis:

• GDPR is the main legislation covering the processing of personal data.
  • It also covers automated decision making and the right of data subjects to view, correct, and port their data (Data Protection Act, 2018).
  • This project does not use personal data and is thus exempt.
• Data created by public authorities is subject to laws concerning transparency and openness (Freedom of Information Act, 2000) - colloquially known as FOI.
  • The data and reports created from this dashboard will be subject to FOI.
  • By publishing the data and code in the open on GitHub, we believe that our obligations under FOI are satisfied under a s21 exemption (National Archives, 2019).
• Accessibility Requirements. All government websites must meet a minimum level of accessibility (CDDO, 2021).
  • If this were to be published, they would need to be tested against modern standards for features like alt text, colour contrast, and keyboard accessibility (W3C, 2018).
• Members of the Houses of Commons and Lords are able to ask Parliamentary Questions of our department (Cole, 1999).
  • The ability to quickly and accurately answer members' questions is part of our core business activity.
  • It is a civil servant's responsibility to allow Ministers to give accurate and truthful information (Cabinet Office, 2011)
• The Centre for Data Ethics and Innovation is a department which acts as a "watchdog" for ethical use of government data.
  • They consider "dashboard" style reports as having an important role in communicating data effectively (CDEI, 2021).
• The UK is a multilingual country. We have an obligation to consider whether the reports should be published in Welsh (Welsh Language Act, 1993)
  • A further exercise may need to be carried out to determine how many publications are wedi'i ysgrifennu yn y Gymraeg.

3. The properties of different data storage solutions, and the transmission, processing and analytics of data from an enterprise system perspective. This should include the platform choices available for designing and implementing solutions for data storage, processing and analytics in different data scenarios.

Governments around the world have adopted a Cloud First strategy (Busch et al., 2014), and the UK is no exception. Rather than traditional on-premises equipment or dedicated remote servers, we use a set of scalable resources which automatically adapt to our needs. Along with edge processing, caching, and associated features, this paradigm is commonly known as "Dew Computing" (Ray, 2018). This allows us to tightly control our costs, reduce our carbon footprint, and scale to meet demand. We can "shard" our data (that is, store it across multiple, redundant locations) (Koster, 2009). However, this requires careful management of transactions to ensure eventual consistency (Tewari and Adam, 1992).

The majority of our documents are stored as unstructured data in an Amazon S3 Bucket. Because of the lack of structured data, they are retrieved via a key-value store using NoSQL (Sadalage and Fowler, 2012). This reduces the overhead of creating and maintaining a schema, at the expense of the ability to construct deterministic queries.

Our enterprise storage needs generally follow the "Inmon Model" (Inmon, 2005) where multiple data publishers store their data in a "data warehouse" - then further processing takes place in separate datamarts.

Fig 01 Enterprise Data Warehouse diagram © (vlntn, 2009)

The ability to Extract, Transform, and Load (ETL) data allows users to construct their own queries rather than relying on predefined methods (Theodorou et al., 2017). This increases their utility and decreases our support costs.

Because of the lack of detailed metadata, a search index is generated using term frequency–inverse document frequency (TF-IDF) (Spärck Jones, 1972). TF-IDF is the most common method of building recommendation engines (Beel et al., 2016), and allows us to sort search results by relevance.

Basic metadata is stored in relational databases. This gives us the ability to use Structured Query Language to interrogate the database and retrieve information.

Manually exchanging data using outdated formats or standards is unreliable. During the COVID-19 crisis, contact-tracing data was lost due to the limitations of Microsoft's proprietary Excel format (Fetzer and Graeber, 2020). This has accelerated our adoption of APIs which should not suffer from such data loss. As per the National Data Strategy (Dowden, 2020), we now advocate an API first strategy to ensure that data can flow freely.

Storage and transfer of data aren't the only factors to consider. We also have a mandate to provide "Linked Data" (Shadbolt et al., 2012).

4. How relevant data hierarchies or taxonomies are identified and properly documented.

Taxonomies are useful tools for helping both humans and machines navigate data.

The taxonomy currently used in our data warehouse was identified through a process of ongoing user research involving stakeholders inside and outside the organisation.

Content stored on GOV.UK is subject to a sophisticated taxonomy which was intentionally designed to capture published information within a specific domain (GDS, 2019).

In order to meet the criteria for a well-defined taxonomy, the taxonomy is documented in a format which complies with ANSI/NISO Z39.19-2005 (National Information Standards Organization, 2010). The taxonomy documentation must also retain compatibility with other vocabularies used worldwide (ISO, 2011).

Taxonomies within the content area generally fall into multiple categories:

• Department - who published the content (e.g. Department for Education),
• Topic - what the content relates to (e.g. Starting a Business),
• Metadata - information about the content (e.g. filetype, date of publication, etc).

Documentation is stored in GitHub (GDS, 2021) in order to be easily discoverable and editable.

A supervised Machine Learning process was used to classify content which had not been tagged (Zachariou et al, 2018). This has led to an increase in correctly identified content.

The use of Convolutional Neural Networks to assist in Natural-Language Processing increases the accuracy of the identified data while reducing the human resources needed to keep the taxonomy up to date.

Because individual users and departments are free to create their own tagging structure, the overall effect is that of an unstructured "Folksonomy" (Vander Wal, 2004).

In the future, we may allow users of our service to create their own tags in a collaborative environment. We recognise that this may introduce challenges around diversity and inclusiveness (Lambiotte and Ausloos, 2006).

3. Product Design, Development & Evaluation

The application of data analysis principles. Include here the approach, the selected data, the fitted models and evaluations used in the development of your product.

Approach:

• In order to create a beneficial analytic visualisation, it is important to understand how the graphic will enable interpretation and comprehension of the underlying data (Kirk, 2021).
  • I carried out a brief research exercise to understand the organisational need.
  • Stakeholders wanted a way to dynamically visualise the change in proportion of open:closed document formats uploaded to the GOV.UK publishing platform.

Selected Data:

• Both Structured and Unstructured data were available.
  • The Structured data was in the metadata of the files - date of upload, name of uploading party, and Media Type (IANA, 2021).
  • The Unstructured data was the contents of documents. Further analysis may be taken on this to determine the conformance of the documents to their purported standard.
• I decided to create a time-series visualisation based on the following structured data:
  • Nominal data - the Media Type of the document,
  • Ratio-scale numeric data - the number of files uploaded,
  • Interval-scale numeric data - the date the file was uploaded.

Fitted Models:

• Representing data in an area, such as a Pie Chart, was conceptually understood by stakeholders. But there are numerous problems with people being able to correctly evaluate the area represented in charts - especially small slices (Cleveland, 1985)
• An attempt was made at "Data Sonification" (Kaper, Wiebel and Tipei, 1999). This converts the data into an audio wave so that changes and patterns can be discerned by ear, rather than by eye. The stakeholders considered the resultant "music" to be too experimental to be useful.

Evaluation:

An evaluation was undertaken using the Seven Hats of Visualisation Design (Kirk, 2021).

• Constraints
  • Time
    • Due to the recent changes in departmental priorities, it was uncertain whether newer, more complete, data could be obtained in time.
    • In order to quickly produce experimental results, I reused an existing dataset. Once the complete data were available, I was able to run my developed code against it.
  • Budget
    • The visualisation tools available (R and Python) were both cost-free.
    • The cost of obtaining and storing the data was met out of the existing analytics budget.
  • Technology
    • We retain enough cloud computing resources for both the storage and processing of this data.
    • Reports can be periodically run via a scheduled task manager like cron, or on demand.
    • Hosting and transcoding of video content is best suited to a dedicated resource like YouTube.
  • Politics
    • Any visualisation of data which identified individual departments could be interpreted as a rebuke from our department.
    • Our department's role sometimes involves having difficult conversations with other departments; however we strive to do this privately.
    • It was agreed that any public visualisation should anonymise departments as far as possible, and that publication would not occur without consultation.
• Deliverables
  • Presentation
    • While a static visualisation is common for changes over time, stakeholders suggested a dynamic presentation would be more engaging.
    • Animation requires specific consideration of accessibility needs.
  • Actionable items
    • Stakeholders wanted to know whether past policies had any effect on the publication rate of documents by media type.
    • Stakeholders wanted to see the scale of the problem so they could know how much resource to dedicate to it.
  • Proactive alerting
    • Any report would have to be run periodically.
    • Large changes will be immediately visible and can be used as the basis for further investigation.

Our department's mandate is to directly influence the future behaviour of people contributing to the data set. After assessing the various predictive models, stakeholders determined that the ability to predict future data using ML on current trends was not useful. We expect those behavioural patterns to change following our interventions.

Similarly, the ability to predict media type or openness based on filename or publisher was considered irrelevant due to the metadata being already available.

Pie Charts were felt to be an old-fashioned way of representing data - with their use in the UK Government first being popularised by Florence Nightingale during the mid-1800s (Anderson, 2011).

For these reasons, I decided to implement the TreeMap algorithm (Shneiderman, 1992).

Fig 02 A-Z Treemap with Common Child Offsets © (Johnson, 1993)

TreeMap provides several advantages over visualisations like Pie Charts:

• TreeMap allows users to more quickly understand large and/or complex data structures (Long et al., 2017).
• TreeMap supports sub-groups. That is, a section of the graph can contain sub-sections. This allows for more detail to be displayed.

Initial views of the TreeMap produced non-deterministic results which made comparing a time-series challenging.

Fig 03 Demonstration TreeMaps showing random ordering of results across time periods.

I investigated alternative approaches and discovered that using a squarifying algorithm (Bruls, Huizing and van Wijk, 2000) produced results which were more easily comparable.

Fig 04 Demonstration TreeMaps showing results in the same order across time periods.

Business Implications and Benefits

• The results of this analysis have been shared with the organisation. We now have a coherent understanding of the size of the problem, and its current trajectory.
• Individual departments will be notified about their results.
• Where departments are continuing to publish data in an unsuitable format, we are able to provide them with support.
• The report can now be run on a regular basis, and used to monitor ongoing compliance.

6. The application of concepts, tools and techniques for data visualisation, including how this provides a qualitative understanding of the information on which decisions can be based. Include here the visualisation aspects applicable to your product.

Our goal when producing visualisations is to be "Trustworthy, Accessible, Elegant" (Kirk, 2021).

Trustworthy:

• Accurate and definitive data was retrieved from the warehouse.
• Raw data was stored for others to verify our results.
• Open source scripts showed how the data are transformed into usable dataframes.
• Ensured the output is deterministic so that others can reproduce the results.

Accessible

• In this context, accessible means both meeting legal accessibility requirements (W3C, 2018) and available to those that need to see the data.
  • Accessibility:
    • Care was taken to meet minimum contrast guidelines between text and background.
    • Due to the high prevalence of colour blindness in the population (National Eye Institute, 2019), it was necessary to add labels to ensure that the visualisation was accessible.
    • The minimum font size was challenging because, by its nature, some segments of the diagram are small.
    • The frame-rate of the animation was set at a suitable frequency to ensure photosensitivity needs were met.
    • The final animation was provided as a series of still frames for those that required it.
  • Access
    • The animations were made available internally via our Wiki, email, Slack, and during presentations.
    • The data and graphs were published to GitHub under a permissive licence to encourage access and reuse.

Elegant

• As well as accessibility concerns (see above) there were æsthetic considerations. The default colouring provided by the TreeMap library was used.
• User research showed that viewers intuitively understood that area size was proportional to volume of published documents.

When testing with users, the TreeMap performed well compared to the common issues of visualising noted in literature (Cleveland, 1985):

• Over-emphasising small results was not a problem. Due to the visually smaller sizes of the area, and the proportionately smaller font size, insignificant results were ignored.
• Many charts use multiple similar colours. By restricting the TreeMap to four colours there was less distracting visual noise.
• While users were able to intuitively understand that size is proportionate to volume of documents, some shapes produced by the squarified algorithm were not always easily comparable.

Fig 05 In this example, the tall and thin shape (doc) has a similar area than the more square shape on its left (xls), but users felt it looked significantly smaller.

Due to the use of animation, I followed the example of the animated bubble maps demonstrated by Hans Rosling (Rosling, 2019).

The use of animation was crucial to showing both the scale of the data, and the momentum of change.

Clustering and grouping:

• Another advantage of TreeMap is the ability to subdivide data into groups.
• In this test image, there is no grouping, so it is not possible to see which Media Types are open and which are closed.

Fig 06 Uncoloured TreeMap.

• Once groupings were added, similar Media Types were congruent - which made assessing their relative volume easier:

Fig 07 Coloured TreeMap demonstrating grouping.

Alternate view on data:

• The data can be arranged in a multidimensional array and be considered as a "Data Cube" (Gray et al., 1996).
• This allows us to manipulate and slice the data in a variety of ways to more deeply examine relationships between data facets.
• The most requested view by stakeholders was the ability to cluster by department.
• Being able to "slice and dice" this data (Zaïane, 1999) gives us a more detailed view of the data.

Fig 08 Each black-bordered cell shows a single department, and the number and type of documents they published this year. Department names have been redacted for publication.

The final visualisation was a 30 second animation which demonstrated the rate of change of volume of uploads of different Media Types and their categories.

4. Personal Reflection

A reflective evaluation of the implications of conducting this investigation for your learning development on this programme.

I will evaluate my experience using Gibbs' Model of reflection (Gibbs, 1998)

Description of the experience:

• This was an excellent module which helped me learn useful concepts, tools and techniques.
• Data Science is seen by Number 10 as a hugely important civil service competency (Cummings, 2020). The ability to quickly gather, analyse, and interpret data is a key skill in the modern civil service.

Feelings and thoughts about the experience:

• This module introduced me to new algorithmic ideas, and provided me with valuable insights into how they can be effectively applied.
• I enjoyed sharing my expertise with classmates during our workshops, and explaining some of the vaugeries and limitations of the various languages we learned.
• I would have preferred to have gone into more depth on a single tool, rather than having a variety of tools and techniques to learn.

Evaluation of the experience, both good and bad:

• I had previous experience with both R and Python, but I had never used Azure or PowerBI.
• I discovered I have the ability to quickly apply learnings from other domains when I am confronted with a new piece of software or programming paradigm.
• I found some of the lessons a little too focussed on the syntax of tools, rather than understanding the underlying principles.

Analysis to make sense of the situation:

• This module has strengthened my belief that data science alone isn't the answer to government's problems.
• To tackle new and existing problems, we need expertise which goes beyond data science and which encompasses ethics, psychology, and social sciences (Shah, 2020)

Conclusion about what you learned and what you could have done differently:

• I underestimated the amount of time needed to get the precise data that I required, so I initially relied on an older data set. I should have been more explicit around timescales in my initial request.
• Visualisations are not well understood in our department, so I should have spent more time explaining their value to my team.

Action plan:

• Work with the existing cross-government R community to better understand how I can integrate R into our department's workflow.
• Introduce TreeMaps into more presentations.
• Improve my existing knowledge of Python and commit to blogging about my experience of this module.

Appendix: Code

This R code reads in a generated CSV and then produces the TreeMap images used in the animation:

library(gganimate)
library(treemapify)
library(plotly)

#   Read in the data
uploaded_files <- read.csv("report.csv", header=TRUE)

#   Get a vector of file extensions
mime_types <- unique(uploaded_files[c("Filetype")])
rownames(mime_types) = NULL

#   Get a vector of Organisations
organisations <- unique(uploaded_files[c("Organisation")])
rownames(organisations) = NULL

#   Get a vector of each week
file_dates <- unique(uploaded_files[c("Published.Date")])
rownames(file_dates) = NULL

#   Find broken dates
broken_dates <- subset(uploaded_files, Published.Date == "")

#   Remove rows with null dates
if (count(broken_dates)[,1] > 0 ) {
    uploaded_files <- uploaded_files[uploaded_files$Published.Date != "", ] 
}

#   Convert timestamps to Date objects
uploaded_files <- mutate(uploaded_files, Published.Date = as.Date(Published.Date))

#   Remove rows with too old dates
uploaded_files <- uploaded_files[as.Date(uploaded_files$Published.Date) >= as.Date("2013-01-01"), ] 

#   Add Week Column
uploaded_files["Week"] <- format(uploaded_files$Published.Date, format = "%Y-%W")

#   Get the file extensions - more accurate than MIME type
file_ext <- gsub("^.*\\.", "", uploaded_files$Filename)
file_ext <- sapply(file_ext, toupper)
uploaded_files$Filetype <- file_ext

#   Add category
uploaded_files["Category"] <- ""

uploaded_files$Category[uploaded_files$Filetype == "PDF"] <- "PDFs"

uploaded_files$Category[uploaded_files$Filetype == "DXF"] <- "Other"
uploaded_files$Category[uploaded_files$Filetype == "PS"]  <- "Other"
uploaded_files$Category[uploaded_files$Filetype == "RDF"] <- "Other"
uploaded_files$Category[uploaded_files$Filetype == "RTF"] <- "Other"
uploaded_files$Category[uploaded_files$Filetype == "XSD"] <- "Other"
uploaded_files$Category[uploaded_files$Filetype == "XML"] <- "Other"
uploaded_files$Category[uploaded_files$Filetype == "ZIP"] <- "Other"

uploaded_files$Category[uploaded_files$Filetype == "JPG"] <- "Image"
uploaded_files$Category[uploaded_files$Filetype == "EPS"] <- "Image"
uploaded_files$Category[uploaded_files$Filetype == "PNG"] <- "Image"
uploaded_files$Category[uploaded_files$Filetype == "GIF"] <- "Image"

uploaded_files$Category[uploaded_files$Filetype == "CSV"] <- "Open"
uploaded_files$Category[uploaded_files$Filetype == "ODP"] <- "Open"
uploaded_files$Category[uploaded_files$Filetype == "ODS"] <- "Open"
uploaded_files$Category[uploaded_files$Filetype == "ODT"] <- "Open"
uploaded_files$Category[uploaded_files$Filetype == "TXT"] <- "Open"

uploaded_files$Category[uploaded_files$Filetype == "DOC"]  <- "Closed"
uploaded_files$Category[uploaded_files$Filetype == "DOCX"] <- "Closed"
uploaded_files$Category[uploaded_files$Filetype == "DOT"]  <- "Closed"
uploaded_files$Category[uploaded_files$Filetype == "PPT"]  <- "Closed"
uploaded_files$Category[uploaded_files$Filetype == "PPTX"] <- "Closed"
uploaded_files$Category[uploaded_files$Filetype == "XLS"]  <- "Closed"
uploaded_files$Category[uploaded_files$Filetype == "XLSB"] <- "Closed"
uploaded_files$Category[uploaded_files$Filetype == "XLSM"] <- "Closed"
uploaded_files$Category[uploaded_files$Filetype == "XLSX"] <- "Closed"
uploaded_files$Category[uploaded_files$Filetype == "XLT"]  <- "Closed"

#   Vector of all filetypes
file_extensions <- unique(uploaded_files[c("Filetype")])[,1]

#   Vector of the filetype's category
file_categories <- data.frame(Category=character())
for(e in file_extensions) { 
    temp_row <-  (uploaded_files[uploaded_files$Filetype == e,]$Category[1])
    file_categories <- rbind(file_categories, temp_row)
}
colnames(file_categories)[1] = "Category"

#   Sort by date, then file type
uploaded_files <- uploaded_files[order(uploaded_files$Week, uploaded_files$Filetype),]

#   Weekly graph
weeks <- unique(uploaded_files$Week)

#   Create weekly summary
weekly_data <- data.frame(
    Week     =character(), 
    Filetype =character(), 
    Count    =integer(),
    Category =character(),
    stringsAsFactors=FALSE)

#   Loop through the weeks
for(week in weeks) {
    temp_data <- subset(uploaded_files, Week == week)


    # Loop through and add up all the previous uploads of this filetype
    for (ex in file_extensions) {
        #   How many of this file are there?
        file_count <- count( temp_data[temp_data$Filetype == ex,])


        #   What category is it in?
        cat <- subset(temp_data, Filetype == ex)$Category[1]


        #   Populate the row
        temp_row <- data.frame(Week = week, Filetype = ex, Count = file_count, Category = cat )


        #   Add the row to the existing data
        weekly_data <- rbind(weekly_data, temp_row)
    }
}

# Ensure the column names are right
colnames(weekly_data)[1] = "Week"
colnames(weekly_data)[2] = "Filetype"
colnames(weekly_data)[3] = "Count"
colnames(weekly_data)[4] = "Category"

# Ensure Count is an integer
weekly_data$Count <- as.integer(weekly_data$Count)

#   Generate the images

#  Keep track of the running total
running_total <- data.frame(
    file_extensions,
    file_categories,
    total = integer(length(file_extensions)), 
    stringsAsFactors=FALSE)

#   Loop through the weeks
for(week in weeks) {
    #   Data used for the output image
    image_data <- subset(weekly_data, Week == week)
    print(week)     #   Keep track of where we are


    # Loop through and add up all the previous uploads of this filetype
    for (ex in file_extensions) {
        current_count <- running_total[ running_total$file_extensions == ex, ]$total
        new_data <- image_data[ image_data$Filetype == ex, ]$Count
        new_total <- current_count + new_data
        running_total[ running_total$file_extensions == ex, ]$total <- new_total
    }


    #   Small images don't render - force the smallest ones to a valid size
    size <- sqrt(sum(running_total$total)) / 2
    if (size < 40) {
        size <- 40
    }

    #   Remove PDF (optional)
    # running_total <- running_total[running_total$file_extensions != "PDF", ]


                                 #  Optional layouts
    layout_style <- "squarified" #"fixed" "squarified" "scol" "srow"


    #   Colour scheme   Closed    Other     Open        Image      PDF
    fill_colours <- c("#f8766d","#00b0f6", "#00bf7d", "#a3a500", "#ff6bf3")


    #   Generate the TreeMap
    map <- ggplot(running_total, 
                  aes(area = total, 
                     label = paste( file_extensions, formatC(total, big.mark = ",") ,sep = "\n" ), 
                     subgroup = Category, fill=Category)) +
        geom_treemap(layout = layout_style, size = 2, color = "white") + #  Border of internal rectangles
        scale_fill_manual(values = fill_colours)+
        geom_treemap_text(colour = "white", place = "centre", grow = TRUE, min.size = 0.5, layout = layout_style) +
               geom_treemap_subgroup_border(colour = "white", size = 2, layout = layout_style) +
        ggtitle( paste("Total number of files uploaded to GOV.UK:", week, sep = "\n") )


    file_name <- paste("media/", week, ".png", sep = "")
    ggsave(file_name, 
           map, 
           width  = size, 
           height = size, 
           units  = "mm")
}

References

Anderson, R. J. Florence Nightingale: The Biostatistician (2011) CLOCKSS Archive. Molecular Interventions. Page: 63. DOI: https://doi.org/10.1124/mi.11.2.1

Arundel, Z. (2018) The Write Stuff: how we used AI to help us handle correspondence - Department for Transport digital. Available at: https://dftdigital.blog.gov.uk/2018/04/09/the-write-stuff-how-we-used-ai-to-help-us-handle-correspondence/ (Accessed: 16 June 2021).

Beel, Joeran & Gipp, Bela & Langer, Stefan & Breitinger, Corinna Research-paper recommender systems: a literature survey (2015) Springer Science and Business Media LLC. International Journal on Digital Libraries. Page: 305. DOI: https://doi.org/10.1007/s00799-015-0156-0

Bruls, Mark & Huizing, Kees & van Wijk, Jarke J. Squarified Treemaps (2021) Springer Science and Business Media LLC. Eurographics. DOI: https://doi.org/10.1007/978-3-7091-6783-0_4

Busch, P. et al. (2014) ‘A study of government cloud adoption: The Australian context’. Available at: https://opus.lib.uts.edu.au/handle/10453/121604 (Accessed: 6 June 2021).

Cabinet Office (2011) Drafting answers to parliamentary questions: guidance, GOV.UK. Available at: https://www.gov.uk/government/publications/drafting-answers-to-parliamentary-questions-guidance (Accessed: 6 June 2021).

CDDO (2021) Understanding accessibility requirements for public sector bodies, GOV.UK. Available at: https://www.gov.uk/guidance/accessibility-requirements-for-public-sector-websites-and-apps (Accessed: 6 June 2021).

CDEI (2021) Local government use of data during the pandemic, GOV.UK. Available at: https://www.gov.uk/government/publications/local-government-use-of-data-during-the-pandemic (Accessed: 6 June 2021).

Cleveland, W. S. (1985) The elements of graphing data. Monterey, Calif: Wadsworth Advanced Books and Software. doi: 10.5555/4084

Cole, Michael Accountability and quasi‐government: The role of parliamentary questions (1999) Informa UK Limited. The Journal of Legislative Studies. Page: 77. DOI: https://doi.org/10.1080/13572339908420584

Cummings, D. (2020) ‘“Two hands are a lot”’, Dominic Cummings’s Blog, 2 January. Available at: https://dominiccummings.com/2020/01/02/two-hands-are-a-lot-were-hiring-data-scientists-project-managers-policy-experts-assorted-weirdos/ (Accessed: 18 April 2021).

Data Protection Act (2018). Queen’s Printer of Acts of Parliament. Available at: https://www.legislation.gov.uk/ukpga/2018/12/contents/enacted (Accessed: 6 June 2021).

Dean, Jeffrey & Ghemawat, Sanjay MapReduce (2008) Association for Computing Machinery (ACM). Communications of the ACM. Page: 107. DOI: https://doi.org/10.1145/1327452.1327492

Dowden, O. (2020) National Data Strategy, GOV.UK. Available at: https://www.gov.uk/government/publications/uk-national-data-strategy/national-data-strategy (Accessed: 12 June 2021).

Equality Act (2010). Statute Law Database. Available at: https://www.legislation.gov.uk/ukpga/2010/15/contents (Accessed: 16 June 2021).

Fetzer, T. and Graeber, T. (2020) Does Contact Tracing Work? Quasi-Experimental Evidence from an Excel Error in England. SSRN Scholarly Paper ID 3753893. Rochester, NY: Social Science Research Network. Available at: https://papers.ssrn.com/abstract=3753893 (Accessed: 12 June 2021).

Freedom of Information Act (2000). Statute Law Database. Available at: https://www.legislation.gov.uk/ukpga/2000/36/contents (Accessed: 6 June 2021).

GDS (2019) GOV.UK Taxonomy principles, GOV.UK. Available at: https://www.gov.uk/government/publications/govuk-topic-taxonomy-principles/govuk-taxonomy-principles (Accessed: 6 June 2021).

GDS (2021) alphagov/govuk-developer-docs. Available at: https://github.com/alphagov/govuk-developer-docs/blob/faabf3ecceed0443db1d5243feecfd6d8ca4b0f8/source/manual/taxonomy.html.md (Accessed: 6 June 2021).

Gibbs, G. (1998) ‘Learning by Doing’, p. 134.

Gray, J. & Bosworth, A. & Lyaman, A. & Pirahesh, H. Data cube: a relational aggregation operator generalizing GROUP-BY, CROSS-TAB, and SUB-TOTALS (2002) Institute of Electrical and Electronics Engineers (IEEE). DOI: https://doi.org/10.1109/icde.1996.492099

IANA (2021) Media Types. Available at: https://www.iana.org/assignments/media-types/media-types.xhtml (Accessed: 12 June 2021).

Inmon, W. (2005) Building the Data Warehouse, 4th Edition | Wiley, Wiley.com. Available at: https://www.wiley.com/en-gb/Building+the+Data+Warehouse%2C+4th+Edition-p-9780764599446 (Accessed: 12 June 2021).

ISO (2011) ISO 25964-1:2011, ISO. Available at: https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/05/36/53657.html (Accessed: 6 June 2021).

Johnson, B. (2021) Declaration on Government Reform, GOV.UK. Available at: https://www.gov.uk/government/publications/declaration-on-government-reform (Accessed: 15 June 2021).

Johnson, B. S. (1993) ‘Title of Dissertation: Treemaps: Visualizing Hierarchical and Categorical Data’, p. 272.

Kaper, H.G. & Wiebel, E. & Tipei, S. Data sonification and sound visualization () Institute of Electrical and Electronics Engineers (IEEE). Computing in Science & Engineering. Page: 48. DOI: https://doi.org/10.1109/5992.774840

Kepner, Jeremy & Gadepally, Vijay & Michaleas, Pete & Schear, Nabil & Varia, Mayank & Yerukhimovich, Arkady & Cunningham, Robert K. Computing on masked data: a high performance method for improving big data veracity (2014) Institute of Electrical and Electronics Engineers (IEEE). DOI: https://doi.org/10.1109/hpec.2014.7040946

Kirk, A. (2021) Data Visualisation, SAGE Publications Ltd. Available at: https://uk.sagepub.com/en-gb/eur/data-visualisation/book266150 (Accessed: 6 June 2021).

Koster, R. (2009) Database “sharding” came from UO?, Raph’s Website. Available at: https://www.raphkoster.com/2009/01/08/database-sharding-came-from-uo/ (Accessed: 20 June 2021).

Lambiotte, Renaud & Ausloos, Marcel Collaborative Tagging as a Tripartite Network (2021) Springer Science and Business Media LLC. Lecture Notes in Computer Science. DOI: https://doi.org/10.1007/11758532_152

Long, Lim Kian & Hui, Lim Chien & Fook, Gim Yeong & Wan Zainon, Wan Mohd Nazmee A Study on the Effectiveness of Tree-Maps as Tree Visualization Techniques (2021) Elsevier BV. Procedia Computer Science. Page: 108. DOI: https://doi.org/10.1016/j.procs.2017.12.136

National Archives (2019) ‘Freedom of Information exemptions’, https://www.nationalarchives.gov.uk/documents/information-management/freedom-of-information-exemptions.pdf.

National Eye Institute (2019) Color Blindness | National Eye Institute. Available at: https://www.nei.nih.gov/learn-about-eye-health/eye-conditions-and-diseases/color-blindness (Accessed: 13 June 2021).

National Information Standards Organization (2010) ANSI/NISO Z39.19-2005 (R2010) Guidelines for the Construction, Format, and Management of Monolingual Controlled Vocabularies | NISO website. Available at: https://www.niso.org/publications/ansiniso-z3919-2005-r2010 (Accessed: 6 June 2021).

Noble, S. U. (2018) Algorithms of oppression: how search engines reinforce racism. Available at: https://www.degruyter.com/isbn/9781479833641 (Accessed: 16 June 2021).

Pearson, K VII. Note on regression and inheritance in the case of two parents (1895) The Royal Society. Proceedings of the Royal Society of London. Page: 240. DOI: https://doi.org/10.1098/rspl.1895.0041

Ray, Partha Pratim An Introduction to Dew Computing: Definition, Concept and Implications (2021) Institute of Electrical and Electronics Engineers (IEEE). IEEE Access. Page: 723. DOI: https://doi.org/10.1109/access.2017.2775042

Rosling, H. (2019) ‘World Health Chart | Gapminder’. Available at: https://www.gapminder.org/fw/world-health-chart/ (Accessed: 12 June 2021).

Sadalage, P. J. and Fowler, M. (2012) NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence. 1st edn. Addison-Wesley Professional.

Shadbolt, Nigel & O'Hara, Kieron & Berners-Lee, Tim & Gibbins, Nicholas & Glaser, Hugh & Hall, Wendy & schraefel, m.c. Linked Open Government Data: Lessons from Data.gov.uk (2012) Institute of Electrical and Electronics Engineers (IEEE). IEEE Intelligent Systems. Page: 16. DOI: https://doi.org/10.1109/mis.2012.23

Shah, Hetan Global problems need social science (2020) Springer Science and Business Media LLC. Nature. Page: 295. DOI: https://doi.org/10.1038/d41586-020-00064-x

Shneiderman, Ben Tree visualization with tree-maps (1992) Association for Computing Machinery (ACM). ACM Transactions on Graphics. Page: 92. DOI: https://doi.org/10.1145/102377.115768

Skomoroch, P. (2009) ‘@jakehofman was pondering a blog post on that, often people contact me about “big data” where big = slightly larger than can fit in excel :)’, @peteskomoroch, 6 March. Available at: https://twitter.com/peteskomoroch/status/1290703113 (Accessed: 23 June 2021).

SPARCK JONES, KAREN A STATISTICAL INTERPRETATION OF TERM SPECIFICITY AND ITS APPLICATION IN RETRIEVAL (1972) Emerald. Journal of Documentation. Page: 11. DOI: https://doi.org/10.1108/eb026526

Tewari, Rajiv & Adam, Nabil R Using semantic knowledge of transactions to improve recovery and availability of replicated data (1992) Elsevier BV. Information Systems. Page: 477. DOI: https://doi.org/10.1016/0306-4379(92)90027-k

Theodorou, Vasileios & Abelló, Alberto & Thiele, Maik & Lehner, Wolfgang Frequent patterns in ETL workflows: An empirical approach (2017) Elsevier BV. Data & Knowledge Engineering. Page: 1. DOI: https://doi.org/10.1016/j.datak.2017.08.004

Vander Wal, T. (2004) Folksonomy :: vanderwal.net. Available at: https://vanderwal.net/folksonomy.html (Accessed: 12 June 2021).

Véliz, C. (2020) Privacy is power: why and how you should take back control of your data.

vlntn, J. (2009) English:  Data Warehouse Feeding Data Marts. Available at: https://commons.wikimedia.org/wiki/File:Data_Warehouse_Feeding_Data_Mart.jpg (Accessed: 16 June 2021).

W3C (2018) Web Content Accessibility Guidelines (WCAG) 2.1. Available at: https://www.w3.org/TR/WCAG21/ (Accessed: 6 June 2021).

Welsh Language Act (1993). Statute Law Database. Available at: https://www.legislation.gov.uk/ukpga/1993/38/contents (Accessed: 6 June 2021).

Williams, N. (2018) Why GOV.UK content should be published in HTML and not PDF - Government Digital Service. Available at: https://gds.blog.gov.uk/2018/07/16/why-gov-uk-content-should-be-published-in-html-and-not-pdf/ (Accessed: 14 June 2021).

Wolpert, D.H. & Macready, W.G. No free lunch theorems for optimization (1997) Institute of Electrical and Electronics Engineers (IEEE). IEEE Transactions on Evolutionary Computation. Page: 67. DOI: https://doi.org/10.1109/4235.585893

Xu, R. & WunschII, D. Survey of Clustering Algorithms (2005) Institute of Electrical and Electronics Engineers (IEEE). IEEE Transactions on Neural Networks. Page: 645. DOI: https://doi.org/10.1109/tnn.2005.845141

Zachariou et al (2018) How we used deep learning to structure GOV.UK’s content - Data in government. Available at: https://dataingovernment.blog.gov.uk/2018/10/19/how-we-used-deep-learning-to-structure-gov-uks-content/ (Accessed: 6 June 2021).

Zaïane, O. (1999) Glossary of Data Mining Terms. Available at: https://webdocs.cs.ualberta.ca/~zaiane/courses/cmput690/glossary.html#D (Accessed: 13 June 2021).

Zuboff, S. (2020) The age of surveillance capitalism: the fight for a human future at the new frontier of power. First Trade Paperback Edition. New York: PublicAffairs.

Copyright and Copyleft

This document is 🄯 Terence Eden CC-BY-NC.

It may not be used or retained in electronic systems for the detection of plagiarism. No part of it may be used for commercial purposes without prior permission.

R code is under the MIT Licence.

This document contains public sector information licensed under the Open Government Licence v3.0.

Here's the best data deals that I could find. Stick them in your 2nd SIM slot, use them as broadband backup, or shove in a dongle and attach to a Raspberry Pi.

If you want the cheapest data - it's got to be EE. Less than a quarter of the price-per-GB than Three.

For the cheapest per-month equivalent - there's nothing between Three or Vodafone. Just pick whoever gives you best coverage.

Worth also checking whether the network you choose supports 5G on their pre-paid SIMS. Same for roaming. Some do, some don't - it's not always clear from their websites.

I'm not sure if I can physically use 120GB in a year given that I'm mostly WFH. So I think I might give Vodafone a go.

If you've spotted any better deals - please let me know in the comments below.

Terence Eden is on Mastodon

@edent

Is there any service which will let me "check in" to a beer? Because this Chocolate Tom I'm drinking is amazing.

---

The answers came in swiftly - Untappd was the app to use. So, a few minutes later:

Terence Eden is on Mastodon

@edent

I just earned the 'Newbie' badge on @untappd! http://untp.it/p3POA0

---

In the last decade, how much beer and cider have I drunk?

I've written before about how to extract your data from Untappd using their API.

(A few notes. I don't check in to every drink - I only tend to do so if it's a new beer. Some of these are only tasters of a beer - not a full pint. This is mostly an exercise in playing with R. Visit DrinkAware if you'd like to help manage your alcohol consumption.)

Quick Stats

• 985 check ins.
• 801 unique drinks
• 3.68 average rating
• 4.93 average ABV

Graphs

Is there a correlation between how strong a drink is, and how much I like it?

library(jsonlite)
beer_data <- read_json("untappd_data.json", simplifyVector = TRUE)

abv <- beer_data$beer$beer_abv
scr <- beer_data$rating_score

plot(abv, scr, main="ABV vs Score", xlab="ABV", ylab="Score")
abline(lm(scr~abv), col="red") # regression line (y~x)

Hmmm... There's some week positive correlation there. But it's a bit muddled. Let's turn that into a hexmap:

library(hexbin)
bin<-hexbin(abv, scr, xbins=10, xlab="ABV", ylab="Score")
plot(bin, main="Hexagonal Binning")

Aha! A bit easier to see. Most of the beers I drink are in the 4-5% ABV. And there is some correlation. But, mostly, I just like beer and cider. Hmmm... Which do I prefer?

Let's take a look at Cider first:

library(data.table)
beer_data <- read_json("untappd_data.json", simplifyVector = TRUE)
cider <- beer_data[grepl("Cider", beer_data$beer$beer_name), ]
cabv <- cider$beer$beer_abv
cscr <- cider$rating_score

How much do I like Cider vs Beer? Just beer (OK, also includes Mead and a few other not Ciders)

justbeer <- beer_data[!grepl("Cider", beer_data$beer$beer_name), ]
boxplot(cscr, ylab="Score", main="Cider Scores", ylim=c(0,5))

I like Cider a bit more than beer. Yup!

Let's plot that data on a map! It's a bit more complicated because the JSON is nested.

library(jsonlite)
beer_data <- read_json("untappd_data.json", simplifyVector = TRUE, flatten = TRUE)

venues_list <- beer_data$venue
venues <- as.data.frame(do.call(rbind, venues_list))
locations_list <- venues$location
locations <- as.data.frame(do.call(rbind, locations_list))
locations <-subset(locations, venue_state!="Everywhere")

Display them on an interactive map:

library(sf)
library(mapview)
locations_sf <- st_as_sf(locations, coords = c("lng", "lat"), crs = 4326)
mapview(locations_sf)

Let's zoom in on London:

Yup! Looks about right.

Well, that was a fun afternoon of noodling with R. If you'd like to play with the data, you can download a decade of my Untappd data in JSON format

So I switched to a £6/mo deal. That got me unlimited calls / texts and 2GB data. I figured that would be a sensible compromise. But, after a month, it became clear that I wasn't likely to make any use of that data. No going out for me!

So I switched again to a different provider. Same price - £6 - but less data. Again, after a couple of months, I realised that I wasn't using any data and barely any phone calls. So I switched to PAYG, loaded up a small bit of credit to see how much I'd use.

I also set up SIP. I paid a tenner for some calling credit. I can now call landlines at a cheaper rate than my mobile provider!

And, it has been... fine. In the last year, I've sent half a dozen texts. Mostly free spam reports. And a couple to a friend who doesn't use a smartphone. A little bit of data when I couldn't find WiFi when out-and-about. That's it.

Total spend in the last 12 months

July 2020: £6 contract. Sept 2020: £6 PAYG bundle Oct 2020: £6 PAYG bundle Nov 2020: £10 SIP Mar 2021: £10 PAYG credit

Thirty-eight quid a year. I remember when I used to spend that per month!

I've got about £3 credit left on my PAYG SIM and £6 on my SIP account. That should last me another couple of months.

As the world opens up again, I'm wondering how / if I'll change this. I don't need cellular voice / text - but I still want data when out of WiFi coverage.

My current provider, GiffGaff, charges 10p per MB, or £6/mo for 500MB, or £10 for 9GB. With other providers, it looks like the £6-£10 range gets about 2GB-15GB. Vectone offer 1GB for £3, and there is even a SIM which has 250MB free each month.

So I guess I now have some options:

1. Stick with PAYG data. And try not to use more than 60MB when out of the house. Probably impossible - even given the ubiquity of WiFi.
2. Use RWG's free 250MB per month. Might be tight.
3. Cough up a few quid per month and restrict myself to a couple of hundred MB. I think this should be doable - but might require extra spending if I get stuck somewhere boring.
4. Find a new provider with a more generous allowance and resign myself to paying a lot more than I'm used to.

So, gentle reader, which would you choose?

Recently, I've found another free provider of data, with an intriguing price-plan.

RWG Mobile are a Welsh MVNO who use EE as their backhaul. As well as the usual high-end plans, they also have this at the lower end.

Yup! A free SIM with 50 UK minutes / SMS included. Think about how much calling and texting you do each month. For me, the majority is over IP using Signal or Skype. I do less than an hour per month on hold to old-fashioned landline companies. My data usage is minimal at the moment - I'm on WiFi while I'm WFH, and most trains and shops have free WiFi too.

Could I survive on only 250MB? Probably! Their £1 per month (!!) price plan takes that up to half a gig. For comparison, my GiffGaff SIM costs £6 for the same amount of data (albeit unlimited calls).

I ordered a free SIM - it came with a letter in Welsh and English - popped the SIM into my 2nd slot and... it works! I could make calls to free phone numbers straight away, but had to email them to activate it for data.

I have crap EE coverage where I am, but it has proved useful when out and about. As a backup SIM, it's perfect. Even at £1 per month, it's probably worth having just in case you find yourself somewhere where only EE has coverage.

Please don't abuse this service - they're a small company trying something innovative.

But this really does open up mobile to so many more people. You can now get a cheap, no-name Android device for under £50 and a year of basic service for £12. Living in the future is amazing.

Generate the images

Getting the data is left as an exercise for the reader (sorry!). This loops through the data and generates a separate image for each TreeMap:

for(week in weeks) {
  weekly_data <- subset(file_data, Week == week)

  size <- sqrt(sum(weekly_data$Count)) / 2
  if (size < 40) {
    size <- 40
  }

  map <- ggplot(weekly_data, aes(area = Count, label = paste(Filetype,formatC(Count, big.mark=",") ,sep="\n"), subgroup = Category, fill=Category)) +
    geom_treemap(layout="fixed") +
    geom_treemap_text(colour = "white", place = "centre", grow = TRUE, layout="fixed")

  file_name <- paste("media/", weekly_data$Week[1], ".png", sep="")
  ggsave(file_name, map, width = size, height = size, units = "mm")
}

The width and height are proportionate the the square-root of the size of the data. Annoyingly, ggplot works in millimetres rather than pixels!

If images are too small, R throws an error of "Viewport has zero dimension(s)". So this sets a minimum size. This value was found using trial and error.

The layout is fixed, as per the documentation which keeps the order of the elements and their labels.

Resize and reorientate the images

Now I have a directory of images, each a different size. I want all of them to have the same size canvas and to be placed against the right-hand edge.

mogrify -gravity east -background white -extent 1750x1750 *.png 

That sets the "gravity" to the right - so the original image is centred vertically but is up against the right edge. The extent is the dimension of the new image.

Mogrify overwrites the original images.

Make a video

This is a lazy way to shove all the images into a video.

cat *.png | ffmpeg -f image2pipe -r 10 -vcodec png -i - -vcodec libx264 out.mp4

or

ffmpeg -framerate 8 -pattern_type glob -i '*.png' -c:v libx264 -r 30 -pix_fmt yuv420p out.mp4

Some websites will need further conversion as they have specific codec requirements.

That's it

It isn't the prettiest way to do things, but it seemed pretty effective. If you know of a more efficient - or more R-ish way to accomplish the same animation - please let me know.

Is the word "data" a plural? On a strict reading, yes. Datum is singular, data is its plural.

But humans are spongey meatbags who evolve language. And there will always be a tension between traditionalists and modernists.

So, I took a serious, scientific, and accurate Twitter poll.

Terence Eden is on Mastodon

@edent

Data:

#UKGC21

---

is: (231)
231
are: (72)
72

---

It's amazing how many people are wrong, eh?

What I want to understand, is why it has evolved to a singular?

Charlie expresses it perfectly:

Charlie

@CharlieEdmunds

Replying to @edent@edent IMO data is like rice
The rice/data is ready/dirty/tasty
If you want to talk about individual bits, you can talk about grains of rice or pieces of data

---

I understand what Charlie is saying, but I think I disagree with her. Think about cooking some chips. Would you say "the chips is ready"?

No. But chips are small individual "things". Just like rice.

What's something smaller than chips, but bigger than rice. Peas?

"How's dinner coming along?" "The peas is ready!" Nope!

Something between peas and rice? Sweetcorn?

"The sweetcorn is ready." Aha!

There seems to be some intuitive size related to when something is an individual thing, and when it is part of a whole.

If a dozen bees are flying towards you - they're a plural.

But a swarm of bees is a singular thing - despite being made of many small bees.

Data is a swarm. The individual datums are tiny compared to the mass of the dataset.

UK Network Three are giving away free SIMs which comes with 200MB per month of free data. Visit three.co.uk/datareward to register. My SIM was delivered a few days later.

Update - the SIM is now £10 but comes pre-loaded with 1GB for the first month. Then 200MB free for every subsequent month.

The first byte is always free

Obviously nothing in life is free. Three want you to use up your data and decide to pay for a bit more. Or pony up for some calls or texts. And, no doubt, Three will track your usage and location and sell it to advertising firms.

But, hey, you get a chunk of data out of it. So use with caution.

What can you use it for?

It appears to be a fully functional mobile broadband SIM. It works in phones and dongles. Three don't have a 2G network, but it works on their 3G and 4G networks. I don't have a 5G device to test it on.

200MB isn't a lot these days. Which seems ridiculous, but there we are. It is perfectly adequate for sending emails and some infrequent browsing. Turning on your browser's data saver mode might help. But WiFi is free and plentiful in most parts of the civilised world.

The best use, I think, is in automated systems and trackers which either don't use much data or need a reliable backup. I'm going to spend part of the new year experimenting with that.

But Three are crap?!!?!?

I know Three's coverage is rubbish. And they have a complete lack of a customer service ethos. And even if you could get online, the modern web is so bloated that you'll use up 200MB with 5 minutes of TikTok.

It reminds me of the old Woody Allen story:

There's an old joke... two elderly women are at a Catskill mountain resort, and one of 'em says, "Boy, the food at this place is really terrible." The other one says, "Yeah, I know; and such small portions."

The SIM I got from them just didn't work on their Data Reward plan. I contacted customer services several times, and each time they promised me it was working. But it never got the free data.

After wrangling with customer services for a bit, the free data did appear.

As I mentioned in an earlier post phone calls and texts are effectively free once you buy a small chunk of data. I'm now using VoIP / SIP for free calls, Signal for free messaging, and now Three for free data when I'm not on free WiFi.

Phone networks are now just a payment-plan vector for expensive mobile phones.

Without your permission, or even your awareness, tech companies are harvesting your location, your likes, your habits, your relationships, your fears, your medical issues, and sharing it amongst themselves, as well as with governments and a multitude of data vultures. They're not just selling your data. They're selling the power to influence you and decide for you. Even when you've explicitly asked them not to. And it's not just you. It's all your contacts too, all your fellow citizens. Privacy is as collective as it is personal.

This is an extremely timely and important book. Unlike the similarly themed "The Age of Surveillance Capitalism" this is completely readable by a non-academic. Utterly free of highfalutin verbiage. It is refreshingly plain. Véliz sets out her thesis and logically follows through all the arguments.

Véliz expertly takes the reader through a journey which is intended to radicalise them into taking privacy seriously. Your data is being used without your consent and it is having a disastrous impact on you and your community.

Again, unlike Zuboff's "Surveillance Capitalism", this contains practical actions an ordinary person can take. As well as political actions - it contains concrete steps. I'm happy to say they're things I've talked about:

• Use and Ad Blocker.
• Give fake details online.
• Use a password manager.
• Lie to public WiFi providers.
• Use a unique email for every service.

It is essential that you read this. Even if you think you understand the arguments for privacy, there will be stories in here which will shock and enrage you. It will give you the tools to defend yourself and sets out a decent plan for the future.

As Véliz says:

"We use fire doors to contain possible fires in our homes and buildings, and watertight compartments to limit possible flooding in ships. We need to create analogous separations in cyberspace."

Grab a copy, now.

After writing about how to use MoneyDashboard's unofficial API, the good folk at Moneyed told me about their officially supported API! So here's a quick review & howto guide.

Moneyed is a slightly strange service. I think it is designed for companies to give as a benefit to their employees. But you can sign up as an individual. The first month is free - but I don't see a way to tell how much subsequent months are. Although it is presented as an app for Android and iPhone, you can log in on the website.

It is a read-only account aggregator. This allows you to see all your credit cards, current accounts, and savings accounts in one place.

There are a good range of OpenBanking API accounts which you can add.

There's also investment accounts, credit cards, and pensions.

You can download your data as CSV - but that's not the exciting part!

In the settings screen, you can generate an API key. That gives you JSON access to your accounts and transactions.

Once done, there's a quick guide to downloading your data.

You need your API key sent as a header, and your account's unique ID.

curl --header "Authorization: Api-Key 123456" https://app.moneyed.co.uk/v1/capi/assets/abc123

Here's a sample JSON output from my credit card, it shows me buying groceries and getting a refund from eBay. At the bottom is the total balance on the card.

{
  "id": "ABC123",
  "name": "American Express",
  "transactions": [
    {
      "category": "GROCERIES",
      "label": "Morrisons Bradford",
      "pending": false,
      "timestamp": "2020-10-16T00:00:00",
      "value": {
        "amount": "-99.99",
        "currency": "GBP"
      }
    },
    {
      "category": "INCOME",
      "label": "Ebay O Luxembourg",
      "pending": false,
      "timestamp": "2020-10-19T00:00:00",
      "value": {
        "amount": "12.34",
        "currency": "GBP"
      }
    }
  ]
  "type": "CREDIT_CARD",
  "valuations": [
    {
      "date": "2020-10-24",
      "value": {
        "amount": "-123.45",
        "currency": "GBP"
      }
    }
  ]
}

That's not as detailed as the MoneyDashboard API - but it covers everything I need. It would be nice to have more accurate timestamps. At the moment, it only seems to give the last month of spending. But it's a beta project and should improve as time goes on.

You can sign up to Moneyed for one month free - no credit card details required.

Yesterday, I wrote up how to use the MoneyDashboard Classic API. Read that blog post first before reading this one.

MoneyDashboard have launched a new "Neon" service. The API is a bit more simple, but authentication is harder.

Here's a quick guide to the bits of the API that I found useful. I've lightly redacted some of the API responses for my privacy.

List of all supported institutions

MoneyDashboard only supports a limited number of OpenBanking Institutions. Here's a list:

https://neonapiprod.moneydashboard.com/v1/institutions

{
    "0": {
        "id": "44306c61-9fb9-4221-b4d9-f91cd711f665",
        "name": "AIB (NI)",
        "active": 1,
        "paymentsEnabled": false,
        "logo": "https://media.moneydashboard.com/logos/providers/firsttrust.jpg",
        "isAvailableFeatureFlagName": null,
        "primaryColour": "#7F2B7B"
    }
}

List of the User's Accounts

You can add multiple accounts to MoneyDashboard. Here's a list of everything you've added:

https://neonapiprod.moneydashboard.com/v1/accounts

{
    "0": {
        "cognitoId": "1234",
        "accountId": "4567",
        "providerId": "7891",
        "connectionsUserId": "8910",
        "accountType": "CREDIT_CARD",
        "accountNumber": null,
        "sortCode": null,
        "balance": "-123.45",
        "accountName": "Platinum Cashback Credit Card",
        "currency": "GBP",
        "description": null,
        "logo": "https://media.moneydashboard.com/logos/providers/amex.jpg",
        "providerName": "American Express",
        "primaryColour": "#016FD0",
        "created": "2020-09-13T12:34:56.533+00:00",
        "lastUpdateSuccess": null,
        "lastUpdateAttempt": null,
        "deactivated": null,
        "alias": "Platinum Cashback Credit Card",
        "lastRefreshStatus": 0,
        "paymentsEnabled": false,
        "isOffline": false,
        "tokenCreatedDate": "2020-09-13T12:34:56.533+00:00",
        "tokenRefreshDate": "2020-09-13T12:34:56.533+00:00",
        "tokenExpiryDate": null
    }
}

BABS - Balance After Bills

MoneyDashboard can predict what your balance is after bills:

https://neonapiprod.moneydashboard.com/v1/analytics/babs

{
    "babs": -1234.56,
    "predictedBalance": -2345.67,
    "unpaidSeries": 0,
    "dailyFlexSpend": 50.99,
    "daysRemaining": 19,
    "daysElapsed": 0,
    "predictedSpending": 123.45
}

Transactions

A full list of every transaction you've made - including tags:

https://neonapiprod.moneydashboard.com/v1/transactions/filter

{
    "6": {
        "id": "ABCDEFG",
        "created": "2020-10-03T00:00:00",
        "accountId": "4567",
        "customerId": "7891",
        "isPredicted": false,
        "providerTransactionId": null,
        "amount": {
            "amount": 168.35,
            "currency": "GBP"
        },
        "sourceAmount": {
            "amount": 168.35,
            "currency": "GBP"
        },
        "status": "Booked",
        "deactivated": null,
        "type": "Debit",
        "description": "MORRISONS               BRADFORD",
        "seriesId": null,
        "savedDate": "2020-10-13T12:20:23.176183",
        "merchant": "Morrisons Supermarket",
        "transactionBatchId": "ABC123",
        "excludeFromSpendCalculations": false,
        "originalTransactionDate": "2020-10-03T00:00:00",
        "originalTransactionDescription": "MORRISONS               BRADFORD",
        "ProprietaryProviderDetails": null,
        "categorisation": [
            {
                "id": 324978238,
                "certainty": 100,
                "source": "CategorisationService",
                "tag": "Supermarket",
                "level": 2,
                "created": "2020-10-13T12:20:33.678358"
            },
            {
                "id": 324978239,
                "certainty": 100,
                "source": "CategorisationService",
                "tag": "Groceries",
                "level": 1,
                "created": "2020-10-13T12:20:33.678356"
            }
        ],
        "bookedTransactionId": null,
        "merchantLogo": "https://media.moneydashboard.com/logos/merchants/morrisons_supermarket.png"
    }
}

Categories

Get your spending broken down by category. For example, how much do you spend on takeaways?

There are some weird JSON handling of floating point numbers in here. Beware!

https://neonapiprod.moneydashboard.com/v1/analytics/spend/category

{
    "1": {
        "categoryName": "Eating Out",
        "amount": 76.29666666666667,
        "transactionCount": 8,
        "transactions": [
            {
                "id": "ABCDEFG123793",
                "created": "2020-07-22T00:00:00",
                "accountId": "4567",

                "customerId": "7891",
                "isPredicted": false,
                "providerTransactionId": null,
                "amount": {
                    "amount": 22.47,
                    "currency": "GBP"
                },
                "sourceAmount": {
                    "amount": 22.47,
                    "currency": "GBP"
                },
                "status": "Booked",
                "type": "Debit",
                "description": "JUST EAT.CO.UK LTD      LONDON",
                "seriesId": null,
                "merchant": "Just Eat",
                "merchantLogo": "https://media.moneydashboard.com/logos/merchants/just_eat.png",
                "deactivated": null,
                "savedDate": "2020-10-13T12:20:23.176338",
                "ProprietaryProviderDetails": null,
                "categorisation": [
                    {
                        "id": 324978297,
                        "certainty": 100,
                        "source": "CategorisationService",
                        "tag": "Takeaway",
                        "level": 2,
                        "created": "2020-10-13T12:20:33.679618"
                    },
                    {
                        "id": 324978298,
                        "certainty": 100,
                        "source": "CategorisationService",
                        "tag": "Eating Out",
                        "level": 1,
                        "created": "2020-10-13T12:20:33.679617"
                    }
                ]
            }
        ],
        "cycleStartDate": "2020-06-01T00:00:00Z",
        "cycleEndDate": "2020-08-31T00:00:00Z"
    }
}

Authentication

OK, this is where it gets horrible and I get confused. MoneyDashboard uses Amazon Cognito. It does a complex authentication dance, passing along lots of different SRP_A tokens until, eventually, it gives you an IdToken. You can grab that by opening Developer Tools in your browsers. It will be a very long string.

You need to pass this as an x-auth header in your request, like so:

curl 'https://neonapiprod.moneydashboard.com/v1/accounts'\
 -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0'\
 -H 'Accept: */*'\
 -H 'Accept-Language: en-GB,en;q=0.5'\
 --compressed -H\
 'Referer: https://app.moneydashboard.com/'\
 -H 'x-auth: aBcDeFgHiJkLmNoPqRsTuVwXyZ1234567890'\
 -H 'Cache-Control: no-cache, no-store, must-revalidate'\
 -H 'Pragma: no-cache'\
 -H 'Expires: 0'\
 -H 'Origin: https://app.moneydashboard.com'\
 -H 'DNT: 1'\
 -H 'Connection: keep-alive'\
 -H 'TE: Trailers'

I don't know of any easy way to automated getting the token from your own username and password.

Good luck!