The news has got me a bit down this week - AI is not being used well.

Hacking

The generally big news this week - allegedly an OpenAI model broke out of its sandbox and was able to hack another company. Note the language. The company in question is Hugging Face. If you do not work deeply in AI deployment or data science, you probably have not heard of them. But they host a huge number of models which other companies use in their products and analysis. So not just one company impacted by the hack. If this happened, it is a really big deal. Or is it all clever marketing? Hugging Face seem really unbothered by the whole thing, which is making me question what kind of hack this was.

In the spirit of transparency, here is what Clem Delangue, CEO of Hugging Face asked OpenAI publicly:

Radical transparency: let us release the traces from the rogue agents so the entire research community can study what happened.
More capabilities for defenders: let us commit $100M in compute from OAI to help the Hugging Face community build powerful cyber defences with the best open and closed models.

I genuinely cannot work out if this sounds like the response of a company in crisis, but trying to be cool (because OpenAI are the bullies on the playground after all). It reads more like a PR opportunity. Which either means Hugging Face are remarkably unbothered about being hacked by a rogue AI, or they were not entirely surprised by it. I genuinely do not know which is more concerning.

Hallucinating

Now onto more serious matters. The UK Home Office has been accused of using AI and hallucinating a document to refuse an asylum claim. The case involves a Moroccan woman and her child fleeing forced child marriage and extreme violence. The Home Office cited a country policy document as evidence that Morocco was safe to return her to. The document does not appear to exist. A senior judge at the upper tribunal said the Home Office's refusal letter "bears hallmarks consistent with the use of artificial intelligence" - and that if so, it would represent "an extremely serious failing." The woman's case is still ongoing.

I urge everyone who has not, to read the book Weapons of Maths Destruction (I am a nerd, yes). It was written pre-this wave of AI, but it beautifully illustrates what happens when biased machines make decisions. Now they can fake documents too.

As we have all learnt from Black Mirror, it is never the tech, it is the humans. A human should have been checking this every step of the way.

Hindering

There is a third story this week that sits alongside these two, though it is a stranger one. A trial at Lewes Crown Court was halted midway through after the complainant tried to submit screenshots from her phone as evidence. Those screenshots revealed she had used an AI chatbot to prepare for cross-examination - not just to refresh her memory, but working through specific headings: On what happened that night, On Consent and Capacity, On Physical Evidence, On Your Behaviour After. The part-time judge stopped the trial, calling it witness coaching. The Court of Appeal judgment came out this week - they said the judge may have been too hasty in halting proceedings entirely, but added that this case "is unlikely to be an isolated example" and called for clearer rules.

I personally find talking to an AI chat like it is a person odd. It mirrors what you say, in subtle ways. It can go off and fixate on small wording choices. As the Court of Appeal flagged - an AI could unintentionally reshape an honest witness's memory, not just help a dishonest one rehearse.

This is what I am stuck on - if the witness had done the same thing in a private browser session and deleted it, nobody would ever have known. You can prepare with a friend, talk through your account, read your own statement back. None of that is coaching. AI did not create a new problem here - it just made visible something that was always happening invisibly. And as is the pattern with generative AI, intensified it. The court is now scrambling to regulate something it cannot actually detect.

Help?

An uncomfortable week - with some really big consequences. And another call for things to catch up. Can the Home Office use AI and not properly check things? Should they be able to, or should process (or you know, humans) catch this? At what point is talking to an AI coaching? I cannot help but think perhaps this chat transcript should be used in evidence as the witness statement - but can it?

Usually I'm more in favour of innovation vs regulation, as I feel regulation is usually implemented badly. But something needs to catch up.

Three Cheers for AI - Hacking, Hallucinating, Hindering

Want to put this into practice?