When people talk about Open Data, they usually mean "Data that has been paid for by tax-payers" - often data created or collected by Government agencies. But it also extends to the data used by researchers, scientists, economists and any other organisation which wants us to take them seriously. Some private companies also release Open Data - knowing that a level playing-field is good for ecosystem health.

What about individuals? Is it sensible, desirable, or useful for us to release our personal data as Open Data?

Last year, at Open Data Camp I ran a session about what data I release and what the implications are for my personal safety, security, and mental well-being.

First, for the hard-of-thinking, I am not suggesting you do anything you don't want. I'm also in no position to compel you to do anything. If you don't like my ideas, close the tab and write something better.

Right, still here? OK.

We all leave little trails of data with every online interaction. Yes, I know you use a VPN, and TOR, and Incognito mode - but those are all still part of a digital footprint. Here I'm mostly concerned with the intentional release of data.

Most of the data we intentionally release is highly curated. You post about your new job on LinkedIn - but you don't mention the 50 rejections leading up to it. Your online restaurant reviews make it sound like you only go to fancy joints with cool lighting and ethical menus - you don't post about the shameful 3am kebab from that dodgy van.

If you're using FourSquare or similar services, you're deliberately releasing a log of where you've been and at what time you were there. Even if you post a photo of a gig the day after it happened, it's trivial to locate you at that moment.

Some people are very open with their intimate medical data. I'm not. I haven't released a 3D model of my teeth - perhaps out of fear someone will use it to bite other people. Similarly, I haven't released my MRI scans just in case someone clones my brain.

So what about raw data?

I release my home's Solar Generation Statistics as raw & open data. They're useful enough to be cited in several academic papers.

I release my energy consumption (at 30 minute granularity) to The Living Lab so they can build up a model of how normal people use electricity and gas. I trust them not to abuse it, but I'm cognisant they can see when I'm away or when I'm cooking. Do I want you to know that as well?

The discussion we held was wide ranging. Some people want the world to know about them and their problems, others value their privacy. It's an individual choice. It is a discussion you should have with those around you - especially if they're leaking your data.

I am indebted to the wonderful Drawnalism rendering me in the style of da Vinci's Vitruvian Man.

You can read the full write-up of the session on the Open Data Camp blog.

Most hobby sites and side projects don't cost a lot to run. Lots of services have generous free tiers to (ab)use, and they can pay well in "exposure". But OpenBenches is reaching a tipping point where it is slowly overwhelming us.

We've now got nearly 300GB of photos - which means our storage and bandwidth costs are on the high side. Yes, we could losslessly transcode them all (which takes up compute resources) or store them in Glacier (which involves transit costs) and get a better CDN (as opposed to using free tiers). Similarly, we do a lot of forward and reverse geo-coding, OCR, map drawing, and other little bits and pieces.

Basically, assuming we value our time at zero, OpenBenches costs us about £250 per year.

Now, that's not extravagant, but it's also not nothing. So, how do we make it cost neutral to us?

Having a paywall is the antithesis of open data.

Placing adverts next to memorials and expressions of grief feels grim, so we've decided against advertising.

We don't think that a VC is going to invest a million bucks into this. But, hey, if someone out there wants to make us an offer...

That leaves us with merchandising and sponsorship.

Merch!

We've opened a shop on Spreadshirt.

Spreadshirt is pretty easy to use. Upload a graphic, select your products, launch. Each t-shirt brings us in about £4 after VAT. So far we've sold half a dozen things. Early days, but promising. It's a fairly low effort way to (hopefully) get in a trickle of cash.

Sponsorship

Perhaps you have enough t-shirts from random websites? Wouldn't you prefer a shout out on the website and unending gratitude?

We've set up GitHub sponsors but, infuriatingly, it only accepts donations in USD - which means hefty foreign-exchange fees. Luckily, Open Collective accepts payment in GBP which is a lot easier to manage.

Between the two of them, we've had about £175 so far this year. Again, a promising start.

Other plans

I'm not (para)social enough to do a Patreon. I don't want a job of delivering monthly newsletters, livestreaming my coding, or keeping the hype train running. I don't want 1,000 True Fans. In short, we're not trying to make a profit out of this.

We just want to keep the site running for the foreseeable future and, hopefully, pay a bit forward to the brilliant Open Source projects we're built on.

For now we'll stick with those and see how the year plays out. If you have any clever ideas for how to make the site more self-sustaining, please let us know.

Here's how those three years look, according to our smart meter.

Quick maths!

The difference between the generation and the export is our solar consumption: 4,604kWh over 3 years.

So we directly use about 38% of the solar we generate. The remaining 62% gets sold back to the grid. I thought our consumption would be higher; we work from home during daylight hours and we have a modest 2kWh battery. But it is exactly in line with what we were seeing after one year.

Our solar consumption and grid import is the total electricity that we used: 12,125kWh over 3 years.

The average UK home uses around 3,800kWh of electricity per year. We're a bit higher than that. Despite all our energy efficiency measures, we use about 4,000kWh per year.

That means, on average, solar produced 99% of our electricity needs across three years!

You can read more about our solar installation if you'd like to understand the number of panels, details of the inverter, and other technical gubbins.

Money Money Money

People always ask how much the panels cost. That's a difficult question to answer. Every house is different. We have a 3 storey house, which meant extra costs for scaffolding. Our roof was in decent shape, so didn't need reinforcement - and we sold the tiles which were removed. The panels we wanted weren't in stock, so we got a slight discount. Etc etc etc. Basically, go get a couple of quotes and see how much it will cost in your specific circumstances.

The next question is about how much money it has saved us. Again, that's slightly complicated. Energy prices have been "interesting" in the UK. At an average price of 30p/kWh, the panels have saved us about £1,400 over the three years.

But! We also sell the excess electricity. We get paid a variable amount depending on demand - usually between 15p to 30p per kWh. By my calculations, we've sold about £1,500 worth of electricity over 3 years.

Let's call it a total saving of £1,000 per year. If electricity prices go higher, we'll save more. If they drop, we'll save less.

Open Data

Every day, a robot uploads our solar generation data to GitLab where it is freely available for research purposes. The solar stats from our previous home are also available.

☀️

Solar "just works". Yes, even in dreary, rainy, cloudy, foggy UK. Yes, even if your roof doesn't face the optimal direction. Yes, there is a large up-front cost. But even with the current technology, solar is a completely viable way to generate your own electricity and save money.

I remember having a blazing row high-spirited discussion with someone about the way I thought about society. In a fit of rage an attempt to provide clarity, I tried using a metaphor:

"Didn't your mother ever tell you to share your toys?" I asked.

He looked at me, baffled.

"No..." he said, cautiously. "We each had enough toys."

That is the strawman crux of the issue. Those of us who grew up with limited resources know that sharing is caring. We've been taught from an early age that the only way to fulfilment is to share our possessions.

We were raised on a diet of Stone Soup. But other people were not so lucky.

To them, lending a toy just means losing a toy. There is nothing the borrower could reciprocally lend you that you don't already have. There is literally no advantage to sharing. If you really want something, just buy it.

At school, you probably went through the fad of collecting stickers, Pogs, cards, or some other merchandise. The playground was full of swappers. Everyone engaging in the social activity of trade. That's another fertile training ground for understanding the limits of friendship the pleasure of working together for a common goal.

Except for that one kid whose dad just bought them a full set.

How do we embed the positive value of sharing at all levels of society?

There's sometimes a pernicious fear that lending a toy might result in someone else profiting from it.

That's the question at the heart of the Open Data debate. What if someone finds a way to monetise our data - and doesn't share the revenue?

I think - and it's only a think - that's the wrong way to approach things. Most data sits locked away in a vault. It isn't used to generate insights and it certainly isn't making money. It's the same with our toys. Most are at the bottom of the toybox, generating pleasure for no one.

When we open up data - or share our toys - we create opportunities for someone else to derive use from something we have no use for. We give up exclusivity and, in return, discover new ways that we can all have more fun.

I'm a civil servant (this is a personal blog) so I can't comment on the politics behind the idea. But I think it is fair to say that, over the years, it fell out of fashion. Open Data kept being published but very few people looked. Sure, there was the occasional news story where a journalist discovered that the Ministry for Paperclips had accidentally purchased a nuclear submarine on the corporate credit card. But crowdsourcing using the Internet was a very early-2000s idea and the mooted "army" never really materialised.

I thought it was, sadly, a dead concept. Until yesterday:

Gender Pay Gap Bot

@PayGapApp

In this organisation, women's median hourly pay is 13.5% lower than men's.

IBM

@IBM

👩🏻‍💻🧑🏾‍💼👩🏼‍🔬

In celebration of #InternationalWomensDay, take a look at some of the timeless contributions and technological advancements developed by women of IBM. ⬇️ pic.x.com/zZvbuDdUXH

---

---

The UK compels all companies with 250 or more employees to publish data on their gender pay gap. Both public organisations and private companies have to do this.

And so, Francesca Lawson and Ali Fensom built a Twitter bot. Whenever a corporate Twitter account Tweets about International Women's Day - the bot quotes the Tweet with that organisation's gender pay gap.

The results have been beautiful!

Helen

@heldutson

I have never seen any single account wreak more havoc on branded social content than the Gender Pay Gap Bot

---

There is an entire thread of accounts who were so ashamed of being called out that they deleted their Tweets:

madeline odent

@oldenoughtosay

a thread of all the employers who deleted their #IWD2022 tweets as a result of getting called out by the @PayGapApp: because i love mess

---

There are a few (and only a few) organisations which have achieved parity.

Gender Pay Gap Bot

@PayGapApp

In this organisation, men's and women's median hourly pay is equal.

St John Ambulance England

@stjohnambulance

St John's brilliant Women's Network. 💚

Happy International Women's Day. #BreakTheBias #IWD2022

---

---

And even fewer where the gap is negative.

Gender Pay Gap Bot

@PayGapApp

In this organisation, women's median hourly pay is 2.7% higher than men's.

London Fire Brigade

@LondonFire

#IWD2022 Meet Tiarna: “People are surprised when I say I'm a firefighter. I find it hard sometimes being the only woman on my Watch & for a time I was the only black person at my station. But to make a change you need to be present & represent” orlo.uk/FKHpD pic.x.com/00LhU93y6N

---

---

You can argue about whether the data are accurate or meaningful. And you can argue whether a small gap in either direction is statistically significant. There's also a good thread about the impact of furlough on these statistics. It's also debatable as to whether this sort of action will result in meaningful change.

But what you can't argue with, is that this is the epitome of the "armchair auditor" concept. And I'm extremely glad it is back.

I've also created a JSON graph of all of London's rail services - including DLR, Trams, C2C, ThamesLink etc.

Demo

• Play with it yourself
• Grab the code from GitLab
• I've also created a version with every London station and train line.

Limitations

• This is a quick weekend hack - don't expect polished code or performance!
• "Overlapping" lines don't show. So where a route is served by multiple lines, you'll only see one of them.
• Zoom is a bit shonky.
• Relies on an older version of D3 (V5).

Background

Drawing the Tube map as a graph is a popular pastime. See these prior arts:

• "Force-directed graph of tube lines" by Martin Zaltz Austwick (2011)
• "Graph Drawing by Weighted Constraint Relaxation" (2017)
• "What would the London Tube Map look like if Data Scientists designed it?" by Keith McNulty (2018)
• "Going Underground: Graphing and Pathfinding London Tube Lines" by Joe Depeau (2019)
• And many others.

I forked some code from Massimo Santini.

Chris Bell maintains an excellent CSV of all London stations and their connections

The official colours for each line can be found in TfL's colour branding guide and in hex versions at Oliver O'Brien's site.

What's next

Possible ideas - feel free to contribute on GitLab.

• Experiment with different layout algorithms.
• Weight links according to speed, distance, business, or some other metric.
• Better colour options for non-TfL lines.
• More efficient code.

So, the NHS releasing the number of times a doctors' surgery has prescribed Paracetamoxyfrusebendroneomycin is probably OK - unless it is only ever recommended for one-legged Welshmen over the age of 60, and you're the only person in the catchment area who meets that description.

I've found a semi-open data set which make me slightly nervous.

In the UK, you can change your gas and electricity supplier. There are hundreds of different gas suppliers. Which one supplies your home?

Ofgem, the energy regulator, tells people to use the FindMySupplier.energy website to find their supplier.

Type in your postcode and house number, tick a couple of boxes, and promise that you are who you say you are:

And then you can see your gas supplier and Meter Point Administration Number.

This isn't, technically, open data. The licencing terms are restrictive, there's no complete dataset available, and no public API. But it is open for anyone to view. There's nothing stopping you typing in a neighbour's address, or a stranger's address.

Is this a problem?

I'm not sure if my commercial relationship with a supplier is personal data and should be protected. It feels personal.

There's a risk someone could use it in a phishing attempt. Telephoning me and claiming to be my supplier could be a good way to get me to hand over money or other information.

If I work for Energy Supplier X but get my gas from their rival, that could be embarrassing.

Could someone call my energy supplier and claim to be me? Would they be able to socially engineer any more data out of them?

There are lots of niche suppliers. I'm not aware of any which are, for example, marketed exclusively at LGBT+ customers. But I'm sure someone more nefarious than me could find something that a customer might be uncomfortable being revealed.

Is this too remote a risk? All I can say is that I find it kinda creepy that anyone could look up my details, and I kinda worry about how they could use that information against me.

Why is this available?

There's a good reason why this dataset is available in the sort-of open. When you move house, you may not know who the energy supplier is. Finding out the supplier means you can quickly get set up and be billed correctly.

Similarly, the UK has an open database for Car Tax and MOTs (car safety certificates). Type in any car's registration plate, and you can see if its tax is up-to-date and what it failed its last MOT for. Again, useful if you're buying a vehicle. But also handy if you're a nosy neighbour.

In Norway, you can look up anyone's salary and see how much tax they paid. Sounds like fun! But is it a win for open data? Perhaps, but because it encouraged snooping and other unhealthy behaviour, people now have to log in using their national ID number. You can't be anonymous while you search.

As the Norwegian website says:

You can also check if someone has searched for you. If you search for anyone, they can see that you did.

In the UK, we don't have a National ID scheme. So we can't have something like that. Perhaps it would put off illegitimate users of your gas data? Or perhaps it would be intimidating to see just who'd looked up your details?

Open Data is hard. It's hard to strike a balance between useful and creepy. I don't know where the balance is with this data set.

Much rarer is seeing "... & daughters".

But, of course, the plural of anecdote is not data!

The UK register of businesses - Companies House - has a pretty good search engine.

Doing a search for AND SON returns 220,000 results. We use the singular because that should also match the plural.

Instinctively, how many "AND DAUGHTER" businesses do you think they are? Fewer? By how much?

A search for AND DAUGHTER returns 206,000 results!

At first glance, they look similar. Yay gender equality! But there's a problem. Both searches also return dissolved companies.

Additionally, "AND SON" also matches "ANDSON" - which distorts the results. How can we get all the live companies in the system? The search doesn't offer any filters.

Companies House offers a nifty search API. But, sometimes, for tasks like this, we just want a honking big CSV. It's 2.2 GB. You're not opening that in Excel. It's PANDAs TIME!

This PDF contains details of all the currently trading businesses in the UK. Let's open it up!

# Import the library
import pandas as pd

# Read only the first column into a dataframe
df = pd.read_csv("BasicCompanyDataAsOneFile-2020-07-01.csv", usecols=["CompanyName"])

What are we looking for? We can't just search for "SONS" and that will bring back things like "HENDERSONS". Some businesses use "AND SONS" others use "& SONS". Some have "&SONS". We also have to account for the singular.

This quick-and-dirty regex will attempt to find any of the above, without also getting "AND SONGS", for example. Do tell me if there's a better way.

(?:AND |\&)\s?SON([S?]|[\s])

In Pandas terms, that's:

df[df['CompanyName'].str.contains('(?:AND |\&)\s?SON([S?]|[\s])')]

Which prints out:

                                              CompanyName
258                                  & SON STUDIO LIMITED
307                         &SONS TRADING COMPANY LIMITED
467      (BOWEN AND SONS) BAS MECHANICAL SERVICES LIMITED
19492                       1ST CLASS REMOVALS & SONS LTD
28096                                     24LEX & SON LTD
...                                                   ...
4667235                                   ZOLEE & SON LTD
4668636                                    ZORAN&SONS LTD
4668689                             ZORBAS & SONS LIMITED
4671244         ZYBERI & SONS CAPITAL INVESTMENTS LIMITED
4671350                      ZYGMUNT CURRY & SONS LIMITED

[17950 rows x 1 columns]

Or, you can run len() on the output to get the count.

Running the same for DAUGHTER returns:

                                   CompanyName
246                         & DAUGHTER LIMITED
86594          A STEWART AND DAUGHTERS LIMITED
98526                  A.R & DAUGHTERS LIMITED
100064        A.W.F FLETCHER AND DAUGHTERS LLP
179094                   AFI AND DAUGHTERS LTD
...                                        ...
4568242         WILSON SON & DAUGHTERS LIMITED
4582108  WIZDOM: BY OSAGIE & DAUGHTERS LIMITED
4583025        WK LUMSDEN AND DAUGHTER LIMITED
4649690           Z.J. KUBANEK & DAUGHTERS LTD
4669405                     ZS & DAUGHTERS LTD

[320 rows x 1 columns]

Oh. There are about 56x as many "AND SON" businesses as there are "AND DAUGHTER" businesses. Of course, these data don't tell us anything about the size of the businesses or how successful they are. It doesn't tell us how many companies are named "sons and daughters". And a dozen other little data issues.

But, I think the trend is clear. Over time, approximately the same number of "& SONS" businesses and "& DAUGHTER" businesses have been registered. But far more DAUGHTERs have been dissolved.

Why is that?

• Colantuono, Giuseppe & Kor, Ah-Lian & Pattinson, Colin & Gorse, Christopher PV with multiple storage as function of geolocation (2018) page: 217-232. Elsevier BV. Solar Energy. https://doi.org/10.1016/j.solener.2018.03.020
• Shukhobodskiy, Alexander Alexandrovich & Colantuono, Giuseppe RED WoLF: Combining a battery and thermal energy reservoirs as a hybrid storage system (2020) page: 115209. Elsevier BV. Applied Energy. https://doi.org/10.1016/j.apenergy.2020.115209

Someone privately asked if I had more data - so I'm happy to release the full dataset. About five-and-a-half years' of readings, from 2013-12-28 to 2019-06-23.

Download .ZIP file (7MB)

The format is of the CSV is simple. ISO 8601 date, Watts being generated, Cumulative power generation.

Some days and data are missing due to power cuts and Internet outages.

The location of the panels was Oxford, UK (51.725275,-1.232889). They were South-facing. More technical details can be supplied if necessary.

These data are licensed to you as CC BY-SA. If you use them in an academic paper, you are expected to publish as open access.

Enjoy!

This blog post will show you why I've gathered the data, let you download the full dataset, and explain what I learned from it.

Why do this?

There's a long-standing plan to use waste-heat from a nearby supermarket to provide communal heat to our neighbourhood.

A low temperature heat main would connect directly to the chillers at the superstore via a heat recovery unit and circulate ~25° hot water around the neighbourhood. The homes will symbiotically draw on the waste heat produced by the supermarket.

A heat pump would be installed in each household, as a replacement to their existing gas boiler.

The properties will use the existing radiators to run their heating longer but at a lower temperature in order to deliver the same level of thermal comfort.

Bioregional's Rose Hill renewable energy feasibility study

In order to assess whether this is feasible, we need to understand heating demand.

Graph it up!

This graph shows February 14th 2017 - 2019. The green graph line is the temperature inside my house. The blue graph line is the external temperature as measured by a local weather sensor in Oxford. The vertical lines represent when the heating was on - with yellow, orange, and red representing low-demand, medium-demand, and high-demand.

• Fairly obviously, heating demand is highest in winter - but it surprised me just how late it starts.
• Internal temperature doesn't vary as much as external.
• Even in the depth of winter, with the heating off, the house didn't drop bellow 10°C. That's pretty good insulation!

Zoom!

Let's zoom in on December 2018 (tap for bigger):

We were away over the holidays, so the heating was set to stay off - unless the internal reading was under 10°C. Luckily, that didn't happen!

You can also see how rapidly the house cools when the external temperature drops.

Conversely, the rise in external temperature from 5 to 10 barely raised the internal temperature.

Enhance!

Let's zoom in on a couple of days in December 2018.

Here you can see the heating demand. The theory is that Tado can use the weather forecast to see how much heat it needs to generate. I can't easily assess whether it works in practice, but there are a few instances where the heating cuts off before the house reaches the target temperature. Presumably because it know the environment will provide the rest of the thermal energy.

Get The Data!

There are two datasets available. Both are being released under CC BY-SA 4.0.

The first is a CSV file of heating demand (190KB).

The second is the complete Tado output (2.5MB Zip, 25MB unzipped). It is a series of daily JSON files which contain:

• Internal temperature
• External temperature
• Internal humidity
• Whether anyone was at home
• What the thermostat was set to
• If the heating was on
• Heating demand
• When the hot water was on

Feel free to use this data for something interesting, or to make beautiful graphs. I make no claims to its accuracy or completeness.

But there's a sting in the tail.

The majority of the datasets are under the Open Government Licence (OGL). That's basically Creative Commons Attribution (CC-BY 4.0).

Eight of the datasets are licenced under the slightly different Glasgow Open Government Licence (GOGL). 2024 Update: The licence can no longer be found.

Again, it is a fairly standard CC-BY style licence. But with an "interesting" addendum. It warns users:

You must not use any of the Information for an illegal or immoral purpose

Is this condition compatible with Open Data?

Where are your laws based?

What's legal in my country may not be legal in yours. That's always the tricky thing with the Internet - a laptop in Beijing can download a dataset from Glasgow and use it on computers in Montana - three very different jurisdictions. Luckily for us, the GOGL states:

This licence is governed by the laws of Scotland unless otherwise specified by the Information Provider.

There is a full list of Scottish Legislation available for your reading pleasure. How confident are you that you aren't in breach of some archaic piece of Scots law? Would your dastardly plans leave you with a Not Proven verdict?

I'm struggling to think what illegal thing one could do with these data. Even if I used the Glasgow House Sales (1991-2013) data to commit a series of complex frauds - what's the punishment for failing to comply with this clause?

if you fail to comply with them the rights granted to you under this licence, or any similar licence granted by the Licensor, will end automatically.

The Creative Commons licence has been tested in court several times - but I couldn't find any examples where the works themselves had been used for anything illegal.

Whose morals?

Personally, I am a moral relativist. There are no gods which define a perfect way to behave. Our morals clearly change over time - whether it is around homosexuality, divorce, self-defence, privacy, or copyright. And those morals are in constant flux around the world.

What constitutes an "immoral purpose" when it comes to Open Data?

I suppose I could take the Rail Station Facilities data, find out which stations are unstaffed, and use it to create a map of which stations can be graffiti'd without staff noticing. Is that immoral enough?

I struggle to think of anything which is immoral rather than illegal. Perhaps I could use CCTV Locations Operated by Community Safety Glasgow to cheat on my wife without being caught on camera. Would that breach the licence?

Who gets to decide what usage is immoral?

Is this Open Data?

I don't know. The wording seems vague and unenforceable. Once you have released Open Data it is open. You can impose some conditions on how it is used, sure. But they need to be precise and easily understandable across the world.

What do you think?

Continue the conversation on Twitter:

Terence Eden is on Mastodon

@edent

Here's an interesting conundrum.
Is @GlasgowCC's Open Data License...
A) Legally enforceable?
B) Compatible with Open Data?
C) A pragmatic and sensible restriction?

open.glasgow.gov.uk/ckansupport/op…

#OpenData #ImmoralPurposes pic.x.com/zxibvqiidy

---

Or leave a comment below.

Here are some rough and ready numbers about how it has gone so far.

9,870 Benches

At the time of writing, we're a little shy of 10,000 benches. As you can see, we have photos from all around the world.

9,000 UK Benches

The majority of our benches are in the UK. Memorial benches seems like a peculiarly Anglosphere habit. I've spoken to people from all sorts of different countries, and they really are rare outside the English-speaking world.

110 Registered Users

That's over a hundred people signing in with Twitter. We've recently added Auth0 support, so people can sign in with Facebook and GitHub.

We also added a leaderboard so that our mate Jenny could see how many photos she added in Edinburgh.

We still have lots of anonymous users.

21,840 Photographs

Most benches have two photos - one of the inscription, and then another of either the whole bench or a view from the bench.

We also have panoramic shots like this:

66GB of Photos

All that media is pretty weighty. Modern camera phones take high-res photos, and a typical shot is around 4MB.

At the moment, our hosting provider is holding up. Huge shout out to Krystal Hosting - use discount code EDENT for £5 off.

I am looking at using Cloudinary to speed up resizing and delivery. But at close to $100 a month, that's too much for a small hobby project.

Hundreds of Moo Cards

Terence Eden is on Mastodon

@edent

Printed out loads of @openbenches moo cards in preparation for Open Data Camp in Aberdeen! Made with ♥ out of #OpenData

Free tickets available tomorrow.
#ODCamp6 #odcam pic.x.com/dVnUQt8Odn

---

Get 25% off your first Moo Card order!

261 Commits, and 8 GitHub contributors

The whole site is open source on GitHub - and it has been lovely using the issues tab to discuss where the site should go next.

Lots of Fun

But, most importantly, it has been a lot of fun. It's wonderful seeing people from around the planet contributing. Thank you to everyone who has submitted a bench, shared our links, or generally been supportive of this daft little endeavour.

Here are some of the benches which made us smile:

And, finally, Liz and her mum in New Zealand - sat on the bench dedicated to Liz's grandparents.

I was mostly there for work - but managed to spend some time talking about our personal project OpenBenches. Here's a brief run down of what I loved, and what could have gone slightly better.

Love

Scotland! It is so nice to attend an event outside of London! Kudos to the team for not just falling back on yet-another Central London conference venue. Also meant I got to try the Scots delicacy known as Butteries.

Terence Eden is on Mastodon

@edent

Tried butteries for the first time. Love them! #odcamp

---

Diversity of voices. People from all over the world came - although curiously few from Scottish government bodies. Lots of students and people new to the Open Data world. Great to hear so many new and different voices.

Quality of discussion was high - as was attendance over both days. Usually there's a substantial drop-off on the 2nd day. I suspect that's partly because lots of people were away from home.

Lots of people were interested in the OpenBenches story. We discussed how it was set up, organised, and run. Is there a future for it? Will we need funding?

Ian Watt

@watty62

Having heard about @openbenches from @edent and @summerbethat #odcamp this weekend, I have just added my first four #Aberdeen benches, including this great one:

openbenches.org/bench/9621/openbenches.org
The Cloche Boys - Nov 3 1953 In Penance For A Das
The Cloche Boys - Nov 3 1953 In Penance For A Das

---

Beck Strickland

@BeckStrickland

Ajudication please @edent & @summerbeth
Bench in a pub, yes or no? #openbenches #odcamp pic.x.com/3MGDmAnss3

---

Room for improvement

Some of the conversation was... forthright. That's a polite way of saying that some people were over-eager to voice their displeasure at other attendee's opinions.

It is understandable that data formats arouse such passions - but it's quite intimidating if you're new to the scene, and especially off-putting if you don't know the history behind the people, their relationship, and their arguments. My wife has already written about this.

Surprisingly few women turned up. Something the team are looking into.

And, as ever, too many good conversations happening at once.

Liz Eden

@summerbeth

The problem with multi - track unconferences like #odcamp is missing great sessions due to clashes. I could have used Hermione Granger's Time Turner yesterday.

---

Where next

Open Data Camp is marvellous because it is always somewhere interesting - and that's the best way to attract new people to the cause.

If you're interested in hosting it - visit https://www.odcamp.uk/odcamp-hosting-guidance/

Is today recycling day? Or green waste day? Or humble landfill day? Should my food caddy go out? Gosh! Who can keep track? I mean, obviously I could look at my calendar. Or sign up for free SMS alert. Or use push notifications in their app. But where's the fun in that?

"Alexa..."

{
  "uid": "https://example.com/bins/",
  "updateDate": "2017-07-04T00:00:00+01:00",
  "titleText": "Bins",
  "mainText": "Today's bin collection is Green Refuse Bins and Food Caddies",
  "redirectionUrl": "https://www.oxford.gov.uk/homepage/154/find_your_bin_collection_day"
}

https://apps.cloud9technologies.com/oxford/citizenmobile/openapi/addresses?postcode=OX1%203HZ

https://apps.cloud9technologies.com/oxford/citizenmobile/openapi/wastecollections/200004675068

<?php

$wasteURL = "https://apps.cloud9technologies.com/oxford/citizenmobile/openapi/wastecollections/";
$uprn = "";

//  Get the data
$data = file_get_contents($wasteURL . $uprn);

//  Data is in JSON
$json = json_decode($data, true);

//  Strip out unneeded elements
unset($json["WasteCollectionDates"]["UPRN"]);

//  An array to hold the collection information
$collections = array ();

//  Loop through the array
//  Store the data as timestamp=>bins
foreach ($json["WasteCollectionDates"] as $key => $value) {
    $details =  $value["ContainerDescription"];
    $date = strtotime($value["CollectionDate"]);
    if ($collections[$date] != null)
    {
        //  Multiple bins can be collected each day
        $details = " and " . $details;
    }

    //  Ignore empty elements
    if ($date != null)
    {
        $collections[$date] .= $details;
    }
}

//  Date order the array
ksort($collections);
//  Place pointer at the start of array
reset($collections);

//  The array key is a UNIX timestamp
$first_date = key($collections);
$first_bins = $collections[$first_date];

//  Is the next bin collection today, tomorrow, or further in the future?
//  https://stackoverflow.com/a/25623230/1127699
$today = new DateTime(); // This object represents current date/time

$today->setTime( 0, 0, 0 ); // reset time part, to prevent partial comparison

//  Get the timezone as the bins API doesn't return it
$timezone = $today->getTimezone();

//  What is the date of the next bin collection?
$match_date = DateTime::createFromFormat("U", $first_date);
//  Set the timezone based on the local time
$match_date->setTimezone($timezone);
$match_date->setTime( 0, 0, 0 ); // reset time part, to prevent partial comparison

$diff = $today->diff( $match_date );
$diffDays = (integer)$diff->format( "%R%a" ); // Extract days count in interval

$headline = "";

switch( $diffDays ) {
    case 0:
        $headline = "Today's bin collection is " . $first_bins;
        break;
    case +1:
        $headline = "Tomorrow's bin collection is " . $first_bins;
        break;
    default:
        $headline = "The next bin collection is " . $first_bins . " on ". $match_date->format('l');
}

//  Set the correct header for JSON data
header('Content-Type: application/json');
?>{
  "uid": "https://example.com/bins/",
  "updateDate": "<?php echo $today->format(DateTime::ATOM); ?>",
  "titleText": "Bins",
  "mainText": "<?php echo $headline; ?>",
  "redirectionUrl": "https://www.oxford.gov.uk/homepage/154/find_your_bin_collection_day"
}

I've tried previously to create an interactive Alexa skill - it did not go well. This time, I thought I'd create an easier skill - a Flash Briefing. You ask Alexa for your daily news report and it reads out items that you've configured - news, weather, traffic, that sort of thing.

For a basic skill, all you need is a tiny scrap of JSON. In this case:

{
  "uid": "https://example.com/flood/index.php",
  "updateDate": "2017-07-02T09:30:00Z",
  "titleText": "UK Flood Forecast",
  "mainText": "The overall flood risk is VERY LOW for the next five days. Minor surface water flooding impacts are possible, but not expected, on Thursday.",
  "redirectionUrl": "https://flood-warning-information.service.gov.uk/"
}

If you can dynamically generate something like that, and host it on a website, you can make a skill.

Following the Amazon process is slightly tedious - especially when it gives you errors like this.

Pro Tip - tell your users why and where something has failed. They can't be expected to magically guess what the problem is.

You can beta test a skill before it goes live. The submission and review process was painless - took about 12 hours on a weekend.

Voice Interfaces For Government

(My thoughts - not my employer's)

This is an unofficial service that I've built using the Open Data produced by my friends at the environment agency.

In an ideal world, you wouldn't need to create specific skills like this. Nor would you have to create a specially formatted file for every voice assistant out there. Websites would be full of semantic markup and intelligent assistants would be easily able to work out when the council offices were open, which bus routes were delayed, and what the flood risk was.

We're not there yet.

There is a huge industry push towards voice interfaces. Is it incumbent on Government to create services like this, or is it enough to create APIs and let others build things?

There's a long-standing moratorium on the UK Government creating mobile apps - we can't waste money developing for a dozen different platforms, keeping them updated, reacting to changing requirements from store owners.

Is the same true for voice interfaces? Siri, Cortana, Alexa, whatever Google's one is called, Bixby, and a dozen more round the corner. Each needing a different configuration to get them to vocalise your news. Each needing you to sign up to yet-another developer agreement.

My job is looking at open standards. We're too early in this industry to converge on one-standard-to-rule-them all. But if Voice is The Future™, we all need to think about how we make it as easy to deploy interactive speech services as possible.

Voice feels like it could be something - but only if it is open to all.

You can install the UK's flood forecast on Amazon's Alexa.

As part of a minor act of disobedience, I decided to send two Freedom of Information requests. The first to the Home Secretary asking for her email metadata. The second to the Police and Crime Commissioner for Surrey asking for her web browsing history. After all, if they can spy on me, why can't I spy on how they're using their taxpayer-funded computers?

Unsurprisingly, the Home Office rejected my request saying that it was "vexatious" and it was a "scattergun approach and seems solely designed for the purpose of ‘fishing’ for information without any idea of what might be revealed."

Well duh! Of course, in so doing, they rather eloquently prove my point - trawling vast troves of data with no idea what you're looking for is a dangerous pursuit.

What was more surprising is that Surrey Police approved my request!

(I have to declare that I know some of the people within Surrey Police - having worked alongside them at various hackdays and BarCamps. That said, I doubt that influenced their decision to publish.)

I started out by making an FoI request on Twitter (a perfectly valid way to make a request).

Terence Eden is on Mastodon

@edent

Replying to @DameLynneOwens@CCLynneOwens please can I have a list of your Internet history from your work computer and phone. Consider this an #FoI request. Thanks.

---

After a few delays...

Terence Eden is on Mastodon

@edent

Replying to @SurreyPolice@SurreyPolice the last 12 months for preference. Many thanks.

---

Surrey Police

@SurreyPolice

Replying to @edent@edent Sorry for the delay. We're still processing your request & expect to send a final response within 2 weeks. Will update if any change.

---

I received the answer...

Terence Eden is on Mastodon

@edent

Replying to @DameLynneOwens@CCLynneOwens please can I have a list of your Internet history from your work computer and phone. Consider this an #FoI request. Thanks.

---

Surrey Police

@SurreyPolice

Replying to @edent@edent We apologise for the delay in responding to your request. You can now view the response on our website: surrey.police.uk/About-Us/Your-…

---

You can download the entire report as a PDF. This link no longer works and isn't in the Internet Archive.

A few points to note...

• Fair play, I wasn't expecting them to release this! It does somewhat undermine my argument that the state keeps secret information it desires from its citizens.
• It's a PDF - so not quite open data - but good enough.
• I'm not convinced that the data are entirely accurate. Doesn't seem like a lot of usage - and a suspicious lack of tracking data is reported (although that may well be hidden by their automated reporting tools).
• It's possible to draw some (incorrect) conclusions from the data.

Terence Eden is on Mastodon

@edent

Replying to @SurreyPoliceBlimey! Didn't expect @SurreyPolice to answer this FoI for @NCA_LynneOwens' browsing data.
x.com/surreypolice/s… pic.x.com/memipzuevp

---

Jacob Aron

@jjaron

Replying to @edent@edent @SurreyPolice @NCA_LynneOwens Full list includes a primary school website, from which I can infer a fair bit, should have redacted?

---

@jjaron @edent @SurreyPolice I visited it to be interviewed for child's project - so no inference needed. I was looking for directions Tweet from NCA_LynneOwens

The Government want a lot more data that this - and will give it to potentially hundreds of organisations. The risk of your late night browsing sessions leaking remains high.

If you're interested in joining the fight against the Surveillance State, please become a supporter of The Open Rights Group.

While some data holders are wise, others are foolish. Some strengthen our belief in the promise of open data, while others leave us reeling with incredulity. I hereby present a tale of light and darkness, which will bring you equal measures of hope and despair.

Earlier this year, I decided to embark on an open-data-quest. I wanted to find all the fire hydrants in my local area and map them. Where where the areas of poor coverage? Should I buy a house on this road or that? Would insurance companies adjust premiums based on the presence of hydrants?

I could just wander around the streets of Oxford and mark every one I came across on a map - but that is a tiresome and error-prone process. Instead, I turned to "What Do They Know" and submitted a Freedom of Information request to my local Fire and Rescue service.

A month later and I had the data! I was ecstatic! A beautiful .CSV filled with locations and descriptions. I crafted interactive maps, mashed the data up with other sources, brilliant!

My good friend Andy Mabbett thought he'd try to do the same thing with a different local Fire & Rescue service. And that's where the trouble started.

Derbyshire Fire and Rescue Service initially refused to supply the information on the grounds that "disclosure of the location of many or all hydrants in the County would prejudice the commercial interests of the water companies and Derbyshire Fire & Rescue Service."

That is, to put it politely, bunkum. All hydrants are prominently displayed in public - so what's can be commercially sensitive about them? They also tried the old "national security" canard:

The water network is part of the critical national infrastructure and we are only provided access to the hydrant information to fulfil our statutory role of fire-fighting.

...

The water network could be perceived to be a potential target. It is part of the critical national infrastructure and Derbyshire Fire & Rescue Service is only provided access to the hydrant information to carry out its statutory role of fire-fighting. Furthermore, the information is received as part of an agreement with water utility companies and we are not permitted to release the information.

Andy appealed and, after intervention from the ICO, the data were released.

the Centre for the Protection of National Infrastructure (CPNI) are of the view that the national security argument to claim the exemption is not strong enough at this time in terms of level of risk and impact on the Critical National Infrastructure.

Hurrah!

Emboldened with his success, Andy tried his luck with the West Midlands Fire Service. An identical request was made - and it was refused. Why? National Security

Yes, that's right, the terrorists might disrupt hydrants in the West Midlands (which they can easily see) or they might commercially compete with the local water company. Errr... what?

Andy is in the middle of an appeal to the ICO. It's hard to see how the West Midlands can refuse this request when others are releasing the data.

And that's part of the problem with Open Data in the UK. Too many public services are outsourced to commercial providers who have no incentive to open up the data for the good of the country. The pervasive fear that someone, somewhere might behave maliciously acts as a massive brake on the progress of open data. Public authorities have to recognise that these data can provide massive public benefits with little real chance of serious harm.

We also have to contend with the fragmented nature of decision making in the UK which leaves us in a confusing and contradictory state. Perhaps, ICO decisions in favour of releasing data need to be binding on all related public bodies?

For now, sadly, we have to make the same arguments time and again in order to promote open data. Onwards!

This lead on to a session ran by Gaia Marcus on how to create the "perfect" FOI request.

This is not an exhaustive guide to making FOI requests. I strongly suggest that you read WhatDoTheyKnow's recommendations and FOI Man's Top Ten Tips.

To be clear, this isn't going to guarantee that you'll receive an answer - but it should ensure that the answer you get back is useful and conforms to best practices.

Data Format

If you want to interact with the data, you will need it to be supplied in an open and machine readable format. Not PDF!

It makes sense to be explicit as to which format you require the data to be supplied. If, for example, you want geographic data you might want it in KML format. Be aware that most FOI officers are not specialists and will almost certainly supply data in whatever the default format is that their systems export.

The suggested wording of your request should be something like :

Please can you supply these data in an open format such as a CSV file. Section 3 of The Secretary of State's Code of Practice (datasets) issued under Section 45 of the Freedom of Information Act gives more details on acceptable data formats https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/243468/9780108512636.pdf.

For the avoidance of doubt, please do not supply the data in PDF or Microsoft Word format.

Licence

You will often receive data which you cannot make use of due to a restrictive licence. Generally speaking, data provided from a public authority should be under the Open Government Licence. Add this line to the end of your FOI :

I assume that the data provided will be licensed under the Open Government Licence (OGL) as per the guidance from the ICO (https://ico.org.uk/media/for-organisations/documents/1151/datasets-foi-guidance.pdf).

Online Publication

Ideally, all FOI data should live online. We shouldn't need to make specific requests to access it. We also want it to be regularly updated. Try including this line in your request :

Please can you publish the data on your website as per the ICO's model publication scheme (https://ico.org.uk/media2/for-organisations/documents/1153/model-publication-scheme.pdf) and supply me with the URL. I would appreciate it if you keep the data updated, with an indication of when and what changes are made. If you are unable to publish online, please send the me the file(s) by return.

Be Polite

Remember, FOI officers are human! They have a stressful and difficult job. Wrapping up your request in boilerplate legal language and being aggressive in your wording is unlikely to win you any friends.

FOI teams can reject "vexatious requests". Be polite, respectful, and as helpful as possible. Don't bombard a department with dozens of similar requests.

These are tips for getting better data - if you have any more, please leave a comment below!

I was having an argument spirited discussion with my friend John the other day.

He was sympathising with Tony Blair's regrets over the introduction of the Freedom of Information Act. Apparently, FOI requests put too great a strain on departments and government bodies. Not only is there a huge cost of compliance, but there's a an institutional fear of "dangerous" FOIs - requests which take too much time, are costly to process, or could embarrass someone powerful.

Good.

I want institutions to fear FOI requests. I want them to tremble and quake each time WhatDoTheyKnow.com sends them an FOI email.

Each FOI request is a reminder that a department isn't doing their job properly. It shows that the people who pay for the information (the public) have not been easily able to access the information. In today's hyper-connected world, that is a tragedy.

At the moment, an FOI Officer's job is to search for information which hasn't yet been made public and then publish it. I'm greatly over simplifying here - the work of FOI staff is vitally important, but at the moment they are fulfilling a stop-gap function.

The public should have immediate, read-only, direct access to all official sources of information. The data should be well structured, reliable, and comprehensive. It doesn't need to be simplified - but consideration should be given to how non-experts will use and interpret the data.

That's the minimum requirement for Open Data. Giving the public the same level of access as officials.

Let me give you an example...

Hail Hydra!

I wanted to know the location of every fire hydrant in the UK. Just because⁰.

People have asked for this before and have been fobbed off with bizarre excuses, including :

• Fire Hydrants are a national security issue!
• Their location is commercially sensitive!
• We don't have a comprehensive list!

All utter nonsense. A determined group of individuals could crowd-source the data for a restricted geographical area without much fuss. It's merely a case of walking around and looking!

I sent off an FOI request to my local fire service to see what they'd say. I was pleasantly surprised when they replied well within the legal time limit and gave me the information I asked for.

Nearly.

Death To The PDF!

Adobe have created many abominable file formats. The hideously insecure Flash, the needlessly complicated Photoshop, and the tragically misused PDF.

Look, PDF is a perfectly fine way to represent a paper document in a digital format - but it should never be used for the transfer of data. It's a complex file which deliberately obfuscates data and makes it hard to perform any meaningful actions on its contents.

So, of course, that's how my FOI data were delivered to me - in a sodding PDF.

But, oh! Gentle reader, it gets so much worse.

There are many tools which will extract text from a PDF and render them as pure data. These tools are born out of necessity and - while they make a valiant effort to be correct - often fall short. They perfectly demonstrate why PDF is not a robust method of data transfer.

Sadly, the data in my PDF were not able to be extracted. The data in my PDF were encoded as images.

Page after page of JPG. Each one trapping the data as a picture, rather than machine readable text. 15 bloody MBs of it!

This is Open Data in name only. A sham. A cruel trick played by vengeful gods determined to frustrate mere mortals! Wickedness perpetuated out of demented malice serving only to enrage and disempower!

Or was it? I sent a polite email asking for the data in .CSV format, and it was quickly provided.

Which was nice.

In truth, there was nothing malicious about the way the data was sent to me. A busy FOI Officer hit the big "Export" button and accepted the default file format. If I had wanted a specific format, I should have been explicit in my request. Of course, I have no way of knowing what those defaults are - and the officer has no way of knowing what format suits me best.

My friend Andy Mabbett sensibly adds this note to all his FOI requests :

I would like this data, please, in a format which I can edit, such as a CSV file, or one which I can enter into a mapping service, such as a GPX/KML file.

My preference would be for you to publish this data on your website and to provide the relevant URL(s); and to keep it updated, with an indication of when and what changes are made. Otherwise, please send the file(s) by return.

It's that final paragraph on which I want to concentrate.

Solving (Some) Open Data Frustrations

The data I want is stored on a database known as Hydra (which is quite different from Marvel's Hydra!)

Give the public access to Hydra.

Seriously. Do it. These data aren't secret. There's no need to hide them away other than "well, that's just how we've always done it."

To be clear, I'm not suggesting that any Tom, Dick, or Harry should be allowed to edit the data (I don't think the UK Civil Service is quite ready for the Wikipediaisation of their world) - but it solves so many problems by just providing direct access.

There are existing models for this throughout the UK - the Department for Education provides free and open access to EduBase - a database of all the schools in the country.

I'm suggesting that every single database held by the state should be available to everyone.

Working With The Data

Here is a sample of the data stored in Hydra.

mains_size ngr_easting ngr_northing address_property     address_street   address_town
100        458455000   222364000    OPP. BICESTER BRIDAL MANORSFIELD ROAD BICESTER

Converting the Eastings and Northings into Latitude and Longitude is relatively simple and means we can plot them on a map.

Using Google Streetview's API we can locate the hydrant marker via its co-ordinates.

We can also use the human-readable string "Opposite Bicester Bridal" to confirm the location of the hydrant itself.

With this mix of open data and proprietary services, we can create new services. Would the Fire Brigade benefit from street level photos? Could we crowdsource photos of each hydrant? How well maintained they are? Can I provide a competing service to the water companies?

That's the beauty of open data - it opens up endless possibilities.

Visualising The Data

By using the quite remarkable CartoDB we can plot all of these points on an easy to use map.

By panning and zooming, we can easily see if the house we're about to buy / rent is well served by hydrants. Useful!

By coincidence, the panels were switched on during the Winter Solstice - the shortest day of the year. So, how have they performed?

Tracking Solar Performance - The Easy Way

The solar panels have a generation meter. It tracks every kWh the panels produce. As of today, it stands at 4,061kWh. To put that into perspective, the average electricity usage per household in 2013 was 4,192 kWh - and that is dropping.

Based on our previous usage, we estimate our normal consumption to be ~3,300 kWh per year. That's about 9 kWh per day.

Without a doubt, solar panels have completely offset out energy usage and have made Oxford just a little bit greener.

Using the Fronius Solar.Web monitoring service, we can see how the panels have performed over the year.

You can see the tail end of last December on the left. There is a slight discrepancy between what the meter records and what the API spits out.

Tracking Solar Performance - The Hacker's Way

A solar system consists of several parts. The panels, the wiring, the fuses, and the Inverter. The Inverter is a big box which changes the DC electricity from the panels into delicious AC. I specifically chose an Inverter with an API.

That allowed me to write a script which would graph the panels' output. What to do with that output? Why! Tweet it of course!

Edent's Solar Panels

@Edent_Solar
Automated

Getting there. @edent's solar panels generated ~6.79kWh, earning £1.20
FAQ: shkspr.mobi/blog/solar-gra…
#edentSolar pic.x.com/s4mqd6zomm

---

Once a day, at sunset, @Edent_Solar tweets a graph showing how well, or poorly, the panels have done that day. Do please feel free to follow it.

Tracking Solar Performance - The Silly Way

Ok, so that gives us daily monitoring. Isn't there a way to get a fine grained view of how the panels are performing minute-to-minute? Of course!

Solar Realtime Edent

@OxfordSolarLive

168 watts. pic.x.com/vcxwb0ggid

---

Yup, every minute of every day, a cheap webcam takes a photo of the sky above our house, grabs data from the solar panels, and tweets it. If you decide to follow @OxfordSolarLive, you are a muppet.

Tracking Solar Performance - Open Data

For the last few months, I've been contributing to PVoutputs - a global community of solar producers. Every 5 minutes, my panels report their generation for the world to see. This means everyone can see my solar stats by minute, hour, day, week, month, or year.

What Do We Do With The Energy?

We don't have a digital electricity meter - so it's rather hard to tell how much energy we're actually using. During the day, we've got the usual domestic appliances running - plus a couple of servers. We've got a Solar iBoost which heats our hot water with the excess electricity which we haven't been able to use. Sun heated showers are the best!

Over the last 12 months, we've diverted 962 kWh into the hot water tank.

That's roughly a quarter of what we've generated - it's cut our gas bills in summer to virtually nil. There's no API for the iBoost, sadly.

FAQs

I do have an FAQ about the panels, but I'll quickly recap.

• They cost £8,000 from the fine people at Sims Solar.
• In the last 365 days, the UK Government have paid us about £730 for our energy generation.
• It has cut our energy bills during summer to close to zero. During winter, our electricity use is also reduced. We also reduce our gas usage by heating water with an iBoost.
• The panels will pay for themselves in under 10 years. The payment scheme is in place for 20 years. The panels and associated bits are guaranteed for 10-25 years.
• No, we don't have electricity during a power-cut. That's to prevent equipment damage, or electrocuting anyone working on the power lines.
• There's no way to store the energy if we're not using it. So it flows back into the grid and is used by other houses on our street.

Want to know more? Drop a message in the comments :-)

Open Data

In the spirit of open data, I hereby give you access to about a year's worth of unfiltered data from my panels. Every minute during daylight, they've faithfully stuck the data into a .csv file. Should you wish to do something useful with it, please do so!

The format is simple. ISO 8601 date, Watts being generated, Cumulative power generation.

Enjoy!

2020 update! You can also download the full five year dataset.

These data have been cited in:

• Colantuono, Giuseppe & Kor, Ah-Lian & Pattinson, Colin & Gorse, Christopher PV with multiple storage as function of geolocation (2018) page: 217-232. Elsevier BV. Solar Energy. https://doi.org/10.1016/j.solener.2018.03.020

• Shukhobodskiy, Alexander Alexandrovich & Colantuono, Giuseppe RED WoLF: Combining a battery and thermal energy reservoirs as a hybrid storage system (2020) page: 115209. Elsevier BV. Applied Energy. https://doi.org/10.1016/j.apenergy.2020.115209

• Lange, Jarren Hilton, Realisation of AD hoc renewable rural power systems with decentralised active power dispatch techniques. (2019)

• Shukhobodskiy, Alexander Alexandrovich & Zaitcev, Aleksandr & Pogarskaia, Tatiana & Colantuono, Giuseppe RED WoLF hybrid storage system: Comparison of CO2 and price targets (2021) Elsevier BV. Journal of Cleaner Production. Page: 128926. DOI: https://doi.org/10.1016/j.jclepro.2021.128926

• Wiesheu, Michael & Rutešić, Luka & Shukhobodskiy, Alexander Alexandrovich & Pogarskaia, Tatiana & Zaitcev, Aleksandr & Colantuono, Giuseppe RED WoLF hybrid storage system: Adaptation of algorithm and analysis of performance in residential dwellings (2021) Elsevier BV. Renewable Energy. Page: 1036. DOI: https://doi.org/10.1016/j.renene.2021.07.032

• Shukhobodskiy, Alexander Alexandrovich & Zaitcev, Aleksandr & Colantuono, Giuseppe RED WoLF hybrid energy storage system: Algorithm case study and green competition between storage heaters and heat pump (2023) Elsevier BV. doi:10.1016/j.egyai.2023.100287