ODcamp – Terence Eden’s Blog

Open Data Man - how open is too open?

@edent — Sun, 08 Jun 2025 11:34:41 +0000

Open Data is dear to my heart. It is the idea that raw data should be public published, free of restrictions, and in a form that is most usable for others.

When people talk about Open Data, they usually mean "Data that has been paid for by tax-payers" - often data created or collected by Government agencies. But it also extends to the data used by researchers, scientists, economists and any other organisation which wants us to take them seriously. Some private companies also release Open Data - knowing that a level playing-field is good for ecosystem health.

What about individuals? Is it sensible, desirable, or useful for us to release our personal data as Open Data?

Last year, at Open Data Camp I ran a session about what data I release and what the implications are for my personal safety, security, and mental well-being.

First, for the hard-of-thinking, I am not suggesting you do anything you don't want. I'm also in no position to compel you to do anything. If you don't like my ideas, close the tab and write something better.

Right, still here? OK.

We all leave little trails of data with every online interaction. Yes, I know you use a VPN, and TOR, and Incognito mode - but those are all still part of a digital footprint. Here I'm mostly concerned with the intentional release of data.

Most of the data we intentionally release is highly curated. You post about your new job on LinkedIn - but you don't mention the 50 rejections leading up to it. Your online restaurant reviews make it sound like you only go to fancy joints with cool lighting and ethical menus - you don't post about the shameful 3am kebab from that dodgy van.

If you're using FourSquare or similar services, you're deliberately releasing a log of where you've been and at what time you were there. Even if you post a photo of a gig the day after it happened, it's trivial to locate you at that moment.

Some people are very open with their intimate medical data. I'm not. I haven't released a 3D model of my teeth - perhaps out of fear someone will use it to bite other people. Similarly, I haven't released my MRI scans just in case someone clones my brain.

So what about raw data?

I release my home's Solar Generation Statistics as raw & open data. They're useful enough to be cited in several academic papers.

I release my energy consumption (at 30 minute granularity) to The Living Lab so they can build up a model of how normal people use electricity and gas. I trust them not to abuse it, but I'm cognisant they can see when I'm away or when I'm cooking. Do I want you to know that as well?

The discussion we held was wide ranging. Some people want the world to know about them and their problems, others value their privacy. It's an individual choice. It is a discussion you should have with those around you - especially if they're leaking your data.

I am indebted to the wonderful Drawnalism rendering me in the style of da Vinci's Vitruvian Man.

You can read the full write-up of the session on the Open Data Camp blog.

Are APIs Elitist?

@edent — Tue, 20 Apr 2021 11:19:07 +0000

(This post written partly to tease my delightful colleague Charles, but also as a way of thinking about user needs.)

During a recent Open Data Café, one of the guests made an entirely reasonable point. An API only method of getting data is elitist.

I'd like to briefly discuss that idea.

Downloading a CSV, for example, is simple. CLICK HERE. Done! The file starts transferring to your machine. Obtaining the data can be done by anyone who can read basic instructions and follow a link.

Getting data via an API? That's a lot tougher. Register for an API key, thoroughly read the documentation, compose a query, futz about on the command line, figure out whether the API key is needed in the header or somewhere else, install some weird program to cryptographically sign your request, do an OAuth dance, sacrifice a chicken.

*sigh*

Look, I love APIs. But the developer experience is often poor. Even proficient devs get frustrated when trying to download a simple set of data.

Put simply, you need an order of magnitude more education in order to interact with an API than you do to click "DOWNLOAD".

An API-only method of getting data is elitist. Or, at the very least, exposes privilege. Do you have enough time to read through all the poorly written API docs and examples? Have you had a modern computer science education which explained the different HTTP verbs and idempotency? Do you know your REST from your SOAP from your GraphQL? Are you able to mentally translate the examples in Ruby into the language of your choice?

But...

APIs are necessary. As we move into a world where computers communicate for us, we need to enable data to flow without human intervention. We need APIs. But we also need to meet human users' needs as well.

A link which says "Click Here To Download This Month's Data" should be powered by an API. It shouldn't be a human wasting time collating a spreadsheet and then manually uploading it to the CMS. Behind that link should be a /getData?range=month&order=latest&key=123.... API call.

So, yes, I agree with the statement. An API only solution is probably elitist. For some services, it may make sense, but if you want non-technical users to easily access your data - you need a direct access method.

Further discussion on Twitter:

OpenBenches - some stats

@edent — Mon, 26 Nov 2018 12:30:25 +0000

For the last year-and-a-bit, Liz and I have been running OpenBenches.org. An open data website dedicated to memorial benches.

Here are some rough and ready numbers about how it has gone so far.

9,870 Benches

At the time of writing, we're a little shy of 10,000 benches. As you can see, we have photos from all around the world.

9,000 UK Benches

The majority of our benches are in the UK. Memorial benches seems like a peculiarly Anglosphere habit. I've spoken to people from all sorts of different countries, and they really are rare outside the English-speaking world.

110 Registered Users

That's over a hundred people signing in with Twitter. We've recently added Auth0 support, so people can sign in with Facebook and GitHub.

We also added a leaderboard so that our mate Jenny could see how many photos she added in Edinburgh.

We still have lots of anonymous users.

21,840 Photographs

Most benches have two photos - one of the inscription, and then another of either the whole bench or a view from the bench.

We also have panoramic shots like this:

66GB of Photos

All that media is pretty weighty. Modern camera phones take high-res photos, and a typical shot is around 4MB.

At the moment, our hosting provider is holding up. Huge shout out to Krystal Hosting - use discount code EDENT for £5 off.

I am looking at using Cloudinary to speed up resizing and delivery. But at close to $100 a month, that's too much for a small hobby project.

Hundreds of Moo Cards

Get 25% off your first Moo Card order!

261 Commits, and 8 GitHub contributors

The whole site is open source on GitHub - and it has been lovely using the issues tab to discuss where the site should go next.

Lots of Fun

But, most importantly, it has been a lot of fun. It's wonderful seeing people from around the planet contributing. Thank you to everyone who has submitted a bench, shared our links, or generally been supportive of this daft little endeavour.

Here are some of the benches which made us smile:

And, finally, Liz and her mum in New Zealand - sat on the bench dedicated to Liz's grandparents.

A love-letter to OpenDataCamp #odcamp

@edent — Mon, 12 Nov 2018 12:15:12 +0000

Last week I took an all-too-short visit to Aberdeen for Open Data Camp 6. A two-day unconference looking at Open Data from an international perspective. What works, what doesn't, how we can do better, and what exciting things are coming up.

I was mostly there for work - but managed to spend some time talking about our personal project OpenBenches. Here's a brief run down of what I loved, and what could have gone slightly better.

Love

Scotland! It is so nice to attend an event outside of London! Kudos to the team for not just falling back on yet-another Central London conference venue. Also meant I got to try the Scots delicacy known as Butteries.

Diversity of voices. People from all over the world came - although curiously few from Scottish government bodies. Lots of students and people new to the Open Data world. Great to hear so many new and different voices.

Quality of discussion was high - as was attendance over both days. Usually there's a substantial drop-off on the 2nd day. I suspect that's partly because lots of people were away from home.

Lots of people were interested in the OpenBenches story. We discussed how it was set up, organised, and run. Is there a future for it? Will we need funding?

Room for improvement

Some of the conversation was... forthright. That's a polite way of saying that some people were over-eager to voice their displeasure at other attendee's opinions.

It is understandable that data formats arouse such passions - but it's quite intimidating if you're new to the scene, and especially off-putting if you don't know the history behind the people, their relationship, and their arguments. My wife has already written about this.

Surprisingly few women turned up. Something the team are looking into.

And, as ever, too many good conversations happening at once.

Where next

Open Data Camp is marvellous because it is always somewhere interesting - and that's the best way to attract new people to the cause.

If you're interested in hosting it - visit https://www.odcamp.uk/odcamp-hosting-guidance/

Extracting your data from Untappd

@edent — Sat, 03 Nov 2018 07:54:07 +0000

I rate every pint I taste using the Untappd app. Think of it like TripAdvisor for lager, stout, cider, bitter, and all manner of other beery goodness. Seriously, I've reviewed over 600 different drinks

Recently, I decided to see if I could self-host my beer check-ins. The first step - extracting my own data from Untappd.

Pay To Play

If you become an Untappd Supporter for US$5 per month, you can extract your data in CSV or JSON. I feel like I ought to be able to GDPR that and get it for free but, hey, a couple of quid doesn't seem like a bad way to support an app.

Here's a brief extract:

beer_name,brewery_name,beer_type,beer_abv,beer_ibu,comment,venue_name,venue_city,venue_state,venue_country,venue_lat,venue_lng,rating_score,created_at,checkin_url,beer_url,brewery_url,brewery_country,brewery_city,brewery_state,flavor_profiles,purchase_venue,serving_type,checkin_id,bid,brewery_id,photo_url
"Devon Mist (Cloudy Sparkling Cider)","Sandford Orchards","Cider - Other",4.5,0,"Inoffensive. Found in Aldi.",,,,,,,2.5,"2018-09-29 18:14:59",https://untappd.com/c/657661791,https://untappd.com/beer/163090,https://untappd.com/brewery/14080,England,Crediton,Devon,,,Bottle,657661791,163090,14080,https://untappd.s3.amazonaws.com/photo/2018_09_29/46269e6ba9db8ebc6c66efeddc28edd9_c_657661791_raw.jpeg
"Älska Passion Fruit Apple Cider","Swedish Cider Company ","Cider - Other Fruit",4,0,"Basically an alcopop. Bit sticky.",,,,,,,2,"2018-10-01 19:33:33",https://untappd.com/c/658763031,https://untappd.com/beer/2023147,https://untappd.com/brewery/107942,Sweden,Karlstad,,,,Bottle,658763031,2023147,107942,https://untappd.s3.amazonaws.com/photo/2018_10_01/1a193187913f2429c325c44fe3537b35_c_658763031_raw.jpeg
"War Horse","White Horse Brewery","Extra Special / Strong Bitter",4.9,0,"Gorgeous and light. Summery delicious.","Oxford Playhouse",Oxford,Oxfordshire,"United Kingdom",51.755,-1.26088,4,"2018-10-05 16:51:58",https://untappd.com/c/659797322,https://untappd.com/beer/240251,https://untappd.com/brewery/2678,England,Stanford-in-the-Vale,Oxfordshire,,,,659797322,240251,2678,

Fun fact, the data export didn't originally give a photo URL. So I asked nicely and now it is included for everyone!

But that's not all the data they hold about the check-in.

API

Untappd has a brilliant API which returns much more detail than the CSV export. And you don't even have to pay to use it.

Take a look at this check-in:

"checkin_id": 658763031,
"created_at": "Mon, 01 Oct 2018 18:33:33 +0000",
"checkin_comment": "Basically an alcopop. Bit sticky.",
"rating_score": 2,
"user": {
   "uid": 46296,
   "user_name": "edent",
   "first_name": "Terence",
   "last_name": "E.",
   "location": "UK",
   "is_supporter": 1,
   "url": "https:\/\/shkspr.mobi\/blog",
   "bio": "Long haired geek.",
   "relationship": null,
   "user_avatar": "https:\/\/untappd.akamaized.net\/profile\/5d928465e29fec4375a2990ca2767852_thumb.jpg",
   "is_private": 0,
   "contact": {
      "twitter": "edent",
      "foursquare": "56367",
      "facebook": "609836217"
   }
},
"beer": {
   "bid": 2023147,
   "beer_name": "Älska Passion Fruit Apple Cider",
   "beer_label": "https:\/\/untappd.akamaized.net\/site\/assets\/images\/temp\/badge-beer-default.png",
   "beer_style": "Cider - Other Fruit",
   "beer_slug": "swedish-cider-company-alska-passion-fruit-apple-cider",
   "beer_abv": 4,
   "beer_active": 1,
   "has_had": false
},
"brewery": {
   "brewery_id": 107942,
   "brewery_name": "Swedish Cider Company ",
   "brewery_slug": "swedish-cider-company",
   "brewery_page_url": "\/w\/swedish-cider-company\/107942",
   "brewery_type": "Cidery",
   "brewery_label": "https:\/\/untappd.akamaized.net\/site\/brewery_logos\/brewery-107942_ca21d.jpeg",
   "country_name": "Sweden",
   "contact": {
      "twitter": "",
      "facebook": "",
      "instagram": "",
      "url": "http:\/\/www.alskacider.com"
   },
   "location": {
      "brewery_city": "Karlstad",
      "brewery_state": "",
      "lat": 59.4109,
      "lng": 13.4854
   },
   "brewery_active": 1
},
"venue": [],
"comments": {
   "total_count": 0,
   "count": 0,
   "items": []
},
"toasts": {
   "total_count": 0,
   "count": 0,
   "auth_toast": false,
   "items": []
},
"media": {
   "count": 1,
   "items": [{
      "photo_id": 195061947,
      "photo": {
         "photo_img_sm": "https:\/\/untappd.akamaized.net\/photo\/2018_10_01\/1a193187913f2429c325c44fe3537b35_c_658763031_200x200.jpeg",
         "photo_img_md": "https:\/\/untappd.akamaized.net\/photo\/2018_10_01\/1a193187913f2429c325c44fe3537b35_c_658763031_640x640.jpeg",
         "photo_img_lg": "https:\/\/untappd.akamaized.net\/photo\/2018_10_01\/1a193187913f2429c325c44fe3537b35_c_658763031_1280x1280.jpeg",
         "photo_img_og": "https:\/\/untappd.akamaized.net\/photo\/2018_10_01\/1a193187913f2429c325c44fe3537b35_c_658763031_raw.jpeg"
      }
   }]
},
"source": {
   "app_name": "Untappd for iPhone - (V2)",
   "app_website": "http:\/\/untpd.it\/iphoneapp"
},
"badges": {
   "retro_status": false,
   "count": 1,
   "items": [{
      "badge_id": 2980,
      "user_badge_id": 401989193,
      "badge_name": "Johnny Appleseed  (Level 34)",
      "badge_description": "Grab a pot and put it on your head because you're a modern day Johnny Appleseed. That's 170 different ciders. Try 5 more to unlock Level 35.",
      "created_at": "Mon, 01 Oct 2018 18:33:33 +0000",
      "badge_image": {
         "sm": "https:\/\/untappd.akamaized.net\/badges\/bdg_appleseed_sm.jpg",
         "md": "https:\/\/untappd.akamaized.net\/badges\/bdg_appleseed_md.jpg",
         "lg": "https:\/\/untappd.akamaized.net\/badges\/bdg_appleseed_lg.jpg"
      }
   }]
}

Badges! Beer labels! Locations! Venues! Images! Data!

By default, the API only returns 25 check-ins per request. But it does come with a pagination option:

"pagination": {
   "since_url": "https:\/\/api.untappd.com\/v4\/user\/checkins\/edent?min_id=659797322",
   "next_url": "https:\/\/api.untappd.com\/v4\/user\/checkins\/edent?max_id=602463014",
   "max_id": 602463014
},

Python Power!

Here's a quick script which will download all your beer-data and create a single JSON file to use. You will need to register an API key and generate your own access token.

import json
import requests

username     = "abc"
client_id    = "123"
client_secret= "456"
access_token = "789"

user = requests.get("https://api.untappd.com/v4/user/info/"+username+"?client_id="+client_id+"&client_secret="+client_secret)
user_data = user.json()
total_checkins = user_data["response"]["user"]["stats"]["total_checkins"]

print("Total Checkins = " + str(total_checkins))

checkins = requests.get("https://api.untappd.com/v4/user/checkins/?access_token="+access_token)
checkins_data = checkins.json()
full_data = checkins_data["response"]["checkins"]["items"]

while (len(full_data) < total_checkins):
    max_id   = checkins_data["response"]["pagination"]["max_id"]
    next_url = "https://api.untappd.com/v4/user/checkins/?max_id=" + str(max_id) + "&access_token="+access_token
    print("Downloading " + next_url)
    checkins = requests.get(next_url)
    checkins_data = checkins.json()

    full_data.extend(checkins_data["response"]["checkins"]["items"])

with open("untappd_data.json", "w") as f:
    json.dump(full_data, f)

That will save a file with all your data. Happy drinking!

Crafting A Perfect FOI Request

@edent — Tue, 24 Feb 2015 11:55:18 +0000

One of the sessions I lead at Open Data Camp was "Open Data Horror Stories" - we talked about some of the disasters we've had when requesting or using Open Data.

This lead on to a session ran by Gaia Marcus on how to create the "perfect" FOI request.

This is not an exhaustive guide to making FOI requests. I strongly suggest that you read WhatDoTheyKnow's recommendations and FOI Man's Top Ten Tips.

To be clear, this isn't going to guarantee that you'll receive an answer - but it should ensure that the answer you get back is useful and conforms to best practices.

Data Format

If you want to interact with the data, you will need it to be supplied in an open and machine readable format. Not PDF!

It makes sense to be explicit as to which format you require the data to be supplied. If, for example, you want geographic data you might want it in KML format. Be aware that most FOI officers are not specialists and will almost certainly supply data in whatever the default format is that their systems export.

The suggested wording of your request should be something like :

Please can you supply these data in an open format such as a CSV file. Section 3 of The Secretary of State's Code of Practice (datasets) issued under Section 45 of the Freedom of Information Act gives more details on acceptable data formats https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/243468/9780108512636.pdf.
For the avoidance of doubt, please do not supply the data in PDF or Microsoft Word format.

Licence

You will often receive data which you cannot make use of due to a restrictive licence. Generally speaking, data provided from a public authority should be under the Open Government Licence. Add this line to the end of your FOI :

I assume that the data provided will be licensed under the Open Government Licence (OGL) as per the guidance from the ICO (https://ico.org.uk/media/for-organisations/documents/1151/datasets-foi-guidance.pdf).

Online Publication

Ideally, all FOI data should live online. We shouldn't need to make specific requests to access it. We also want it to be regularly updated. Try including this line in your request :

Please can you publish the data on your website as per the ICO's model publication scheme (https://ico.org.uk/media2/for-organisations/documents/1153/model-publication-scheme.pdf) and supply me with the URL. I would appreciate it if you keep the data updated, with an indication of when and what changes are made. If you are unable to publish online, please send the me the file(s) by return.

Be Polite

Remember, FOI officers are human! They have a stressful and difficult job. Wrapping up your request in boilerplate legal language and being aggressive in your wording is unlikely to win you any friends.

FOI teams can reject "vexatious requests". Be polite, respectful, and as helpful as possible. Don't bombard a department with dozens of similar requests.

These are tips for getting better data - if you have any more, please leave a comment below!

Almost Open Data

@edent — Mon, 23 Feb 2015 08:37:11 +0000

(Inspired by a session at Open Data Camp called "Open Data Horror Stories")

I was having an ~~argument~~ spirited discussion with my friend John the other day.

He was sympathising with Tony Blair's regrets over the introduction of the Freedom of Information Act. Apparently, FOI requests put too great a strain on departments and government bodies. Not only is there a huge cost of compliance, but there's a an institutional fear of "dangerous" FOIs - requests which take too much time, are costly to process, or could embarrass someone powerful.

Good.

I want institutions to fear FOI requests. I want them to tremble and quake each time WhatDoTheyKnow.com sends them an FOI email.

Each FOI request is a reminder that a department isn't doing their job properly. It shows that the people who pay for the information (the public) have not been easily able to access the information. In today's hyper-connected world, that is a tragedy.

At the moment, an FOI Officer's job is to search for information which hasn't yet been made public and then publish it. I'm greatly over simplifying here - the work of FOI staff is vitally important, but at the moment they are fulfilling a stop-gap function.

The public should have immediate, read-only, direct access to all official sources of information. The data should be well structured, reliable, and comprehensive. It doesn't need to be simplified - but consideration should be given to how non-experts will use and interpret the data.

That's the minimum requirement for Open Data. Giving the public the same level of access as officials.

Let me give you an example...

Hail Hydra!

I wanted to know the location of every fire hydrant in the UK. Just because⁰.

People have asked for this before and have been fobbed off with bizarre excuses, including :

Fire Hydrants are a national security issue!
Their location is commercially sensitive!
We don't have a comprehensive list!

All utter nonsense. A determined group of individuals could crowd-source the data for a restricted geographical area without much fuss. It's merely a case of walking around and looking!

I sent off an FOI request to my local fire service to see what they'd say. I was pleasantly surprised when they replied well within the legal time limit and gave me the information I asked for.

Nearly.

Death To The PDF!

Adobe have created many abominable file formats. The hideously insecure Flash, the needlessly complicated Photoshop, and the tragically misused PDF.

Look, PDF is a perfectly fine way to represent a paper document in a digital format - but it should never be used for the transfer of data. It's a complex file which deliberately obfuscates data and makes it hard to perform any meaningful actions on its contents.

So, of course, that's how my FOI data were delivered to me - in a sodding PDF.

But, oh! Gentle reader, it gets so much worse.

There are many tools which will extract text from a PDF and render them as pure data. These tools are born out of necessity and - while they make a valiant effort to be correct - often fall short. They perfectly demonstrate why PDF is not a robust method of data transfer.

Sadly, the data in my PDF were not able to be extracted. The data in my PDF were encoded as images.

Page after page of JPG. Each one trapping the data as a picture, rather than machine readable text. 15 bloody MBs of it!

This is Open Data in name only. A sham. A cruel trick played by vengeful gods determined to frustrate mere mortals! Wickedness perpetuated out of demented malice serving only to enrage and disempower!

Or was it? I sent a polite email asking for the data in .CSV format, and it was quickly provided.

Which was nice.

In truth, there was nothing malicious about the way the data was sent to me. A busy FOI Officer hit the big "Export" button and accepted the default file format. If I had wanted a specific format, I should have been explicit in my request. Of course, I have no way of knowing what those defaults are - and the officer has no way of knowing what format suits me best.

My friend Andy Mabbett sensibly adds this note to all his FOI requests :

I would like this data, please, in a format which I can edit, such as a CSV file, or one which I can enter into a mapping service, such as a GPX/KML file.
My preference would be for you to publish this data on your website and to provide the relevant URL(s); and to keep it updated, with an indication of when and what changes are made. Otherwise, please send the file(s) by return.

It's that final paragraph on which I want to concentrate.

Solving (Some) Open Data Frustrations

The data I want is stored on a database known as Hydra (which is quite different from Marvel's Hydra!)

Give the public access to Hydra.

Seriously. Do it. These data aren't secret. There's no need to hide them away other than "well, that's just how we've always done it."

To be clear, I'm not suggesting that any Tom, Dick, or Harry should be allowed to edit the data (I don't think the UK Civil Service is quite ready for the Wikipediaisation of their world) - but it solves so many problems by just providing direct access.

There are existing models for this throughout the UK - the Department for Education provides free and open access to EduBase - a database of all the schools in the country.

I'm suggesting that every single database held by the state should be available to everyone.

Working With The Data

Here is a sample of the data stored in Hydra.

mains_size ngr_easting ngr_northing address_property     address_street   address_town
100        458455000   222364000    OPP. BICESTER BRIDAL MANORSFIELD ROAD BICESTER

Converting the Eastings and Northings into Latitude and Longitude is relatively simple and means we can plot them on a map.

Using Google Streetview's API we can locate the hydrant marker via its co-ordinates.

We can also use the human-readable string "Opposite Bicester Bridal" to confirm the location of the hydrant itself.

With this mix of open data and proprietary services, we can create new services. Would the Fire Brigade benefit from street level photos? Could we crowdsource photos of each hydrant? How well maintained they are? Can I provide a competing service to the water companies?

That's the beauty of open data - it opens up endless possibilities.

Visualising The Data

By using the quite remarkable CartoDB we can plot all of these points on an easy to use map.

By panning and zooming, we can easily see if the house we're about to buy / rent is well served by hydrants. Useful!

OK, I wanted the data not "just because" but for quite a specific reason. When insuring a property, it would be helpful to know how far away the nearest fire hydrant is. ↩︎