The (theoretical) risks of open sourcing (imaginary) Government LLMs
By @edent
· AI govenment Open Source · 13 comments · 850 words
Last week I attended an unofficial discussion group about the future of AI in Government. As well as the crypto-bores who have suddenly pivoted their "expertise" into AI, there were lots of thoughtful suggestions about what AI could do well at a state level.
Some of it is trivial - spell check is AI. Some of it is a dystopian hellscape of racist algorithms being confidently incorrect. The reality is likely to be somewhat prosaic.
Although I'm no longer a civil servant, I still enjoy going to these events and saying "But what about open source, eh?" - then I stroke my beard in a wise-looking fashion and help facilitate the conversation.
For many years, my role in Cabinet Office and DHSC was to shout the words "OPEN SOURCE" at anyone who would listen. Then patiently demolish their arguments when they refused to release something on GitHub. But I find myself somewhat troubled when it comes to AI models.
Let's take a theoretical example. Suppose the Government trains an AI to assess appeals to, say, benefits sanctions. An AI is fed the text of all the written appeals and told which ones are successful and which ones aren't. It can now read a new appeal and decide whether it is successful of not. Now let's open source it.
For the hard of thinking - this is not something that exists. It is not official policy. It was not proposed as a solution. I am using it as a made-up example.
What does it mean to open source an AI? Generally speaking, it means releasing some or all of the following.
- The training data.
- The weights assigned to the training data.
- The final model.
I think it is fairly obvious that releasing the training data of this hypothetical example is a bad idea. Appellants have not consented to having their correspondence published. It may contain deeply personal and private information. Releasing this data is not ethical.
Releasing how the data is trained is probably fine. It would allow observers to see what biases the model has encoded in it. Other departments could use the model to train their own AI. So I (cautiously) support the opening of that code.
But training weights without the associated data is kind of useless. Without the data, you're unable to understand what's going on behind the scenes.
Lastly, the complete model. Again, I find this problematic. There are two main risks. The first is that someone can repeatedly test the model to find weaknesses. I don't believe in "security through obscurity" - but allowing someone to play "Groundhog Day" with a model is risky. It could allow someone to hone their answers to guarantee that their appeal would be successful. Or, more worryingly, it could find a lexical exploit which can hypnotise the AI into producing unwanted results.
Even if that weren't a concern, it appears some AI models can be coerced into regurgitating their training data - as discovered by the New York Times:
The complaint cited examples of OpenAI’s GPT-4 spitting out large portions of news articles from the Times ... It also cited outputs from Bing Chat that it said included verbatim excerpts from Times articles. NY Times copyright suit wants OpenAI to delete all GPT instances
Even if a Government department didn't release its training data - those data are still embedded in the model and it may be able to reconstruct them. So any sensitive or personal training data might be able to be reconstructed.
Once again, to be crystal clear, the system I am describing doesn't exist. No one has commissioned it. This is a thought experiment by people who do not work in Government.
So where does that leave us?
I am 100% a staunch advocate for open source. Public Money means Public Code. Make Things Open It Makes Things Better.
But...
It seems clear to me that releasing training data is probably not possible - unless the AI is trained on data which is entirely safe / legal to make public.
Without the training data, the way it is trained is of limited use. It should probably be opened, but would be hard to assess.
The final model can only be safely released if the training data is safe to release.
What next?
I'll admit, this troubles me.
I want to live in a world where the data and algorithms which rule the world are transparent to us. There will be plenty of AI systems which can and should be completely open - nose-to-tail. But there will be algorithms trained on sensitive data - and I can't see any safe, legal, or moral way of opening them.
Again, I want to stress that this particular example is a figment of my imagination. But at some point this will have to be reckoned with.
I'm glad this isn't my problem any more!
@blog if one can’t release it as open source because it might allow someone to manipulate the decision making process, that’s a reasonably good sign that the decision making process is likely to be of a sort that might invite judicial review.
If it can’t be open sourced it’s probably doing something suss (or at least not demonstrably being not-suss). Ergo, if open sourcing it is impractical, maybe it’s not a great idea for government to be using it…
Mike says:
This ^. People who make decisions about people’s lives have to be able to explain those decisions. Explaining why a collection of algorithms someone sold claiming it’s AI reached a decision would require explaining how those algorithms work. If you are unable or unwilling to explain how the algorithms work, don’t use them to make decisions about people’s lives.
@Edent I do wonder what the limits of closed box neural network style machine learning is in government.
I find many departments want to take a "credit risk scoring" approach to the people they deal with which I don't think is appropriate given most of the domains.
(We specifically don't do this in our SEND and looked after children modelling. We project future demand based on past performance of the org and then offer ways of doing scenario modelling)
...If something is in the category of 'decisions should be reasoned and explicable' - and there is a strong argument that appeals should be in that category, then perhaps that it where there transparency-as-sunlight comes from, even if transparency-as-code is constrained for the reasons you say.
there is a second release model - that of Research Data Scotland - where data (which has been depersonalised but which contains enough granularity to be repersonalised with other data) is made available in a sandbox to researchers and their output has to be approved by the sandboxers before release
@blog Nice post. I like the like "The final model can only be safely released if the training data is safe to release." I guess that means either "Open the data, code and model if safe" or else "Think really carefully about why that isn't safe."
@Edent Phi-1.5 and Phi-2 from Microsoft are largely trained on data generated by a much bigger model, so they have a fictitious but truthy - and, critically, very high quality - baseline. There doesn't seem to be a danger to the original applicants in that case. You can say to the upstream LLM "generate stuff that looks like this", and the training data is then a sort of parallel construction which avoids the eventual model being able to reveal anything it shouldn't.
Mike says:
“spell check is AI” No.
@edent says:
How do you think modern spell check works? It isn't just "look up words in a list". It looks at surrounding context, stochastic inference, and is predictive.
Those statistical algorithms based on a large corpus of data surely meet any definition of AI.
I'll grant you that it doesn't seem like AI because it isn't that powerful and has been in our lives for decades.
@blog You've reminded me about using Expensify for expense management at a previous employer. They (Expensify) touted their amazing AI used to scan and evaluate photographs of expenses some years ago. Then it turned out it was a 'mechanical turk' of human meat bags in far-flung (e.g. cheap) places, to actually read your receipts 🙂
Mike says:
That'll teach me to make flippant negative comments at someone who has thought about what they've written and can back it up. 😀 I do genuinely struggle with taking any claim that anything is AI seriously though. The term has been used by so many for so long to describe so much. It's all a bit, but you said the last thing was AI and that was obviously a bunch of if/else statements in a black box, but now you want me to believe this is AI? I tend to think of AI has something futristic beyond current technology you can hold a proper conversation with that is capable of original (as any is) thought. Lieutenant Commander Data is AI. Cylons are AI. Kryton is. Possibly not Talkie Toaster, since it only seems capable of holding a conversation about a single subject. Perhaps it's me who's wrong. But it could be nearly everyone else. 😀
@blog I think your made-up example shows that systems that cannot explain themselves cannot be used for decision making affecting people. Most people reactions to AI and algorithms have more to do with the fact that “computer says no”, with no human recourse, is where we’re heading.
More comments on Mastodon.