US kill switch use a ‘nasty moment’ for EU

Artificial intelligence does not pose a risk to human life, but it does raise serious questions around cybersecurity and sovereignty, according to William Fry lawyer Barry Scannell.

Speaking at the William Fry lunchtime AI webinar It’s the end of the World as AI Knows it (24 September), Scannell cited Sam Altman’s comments that three summers ago ChatGPT struggled with basic primary-school maths.

This summer it produced a solution to the Navier–Stokes equation — one of the Millennium Prize problems, unsolved by humans for 80 years.

At this pace, Scannell wondered what capability might look like in another three years.

The lawyer was sceptical about recent claims that AI poses an existential threat, pointing to a pattern of similar predictions from within the same San Francisco ‘AI doomer’ circles.

“For all the talk about the world ending, I haven't heard any realistic scenarios in which AI could cause the end of the world,” he said.

Nuclear systems are air-gapped, and bioweapon development still requires physical materials, labs and expertise that AI does not supply.

“I'm not concerned about the existential risk. What I am worried about is that [talk of] the existential risk is taking us away from all the genuine, real concerns, particularly in cyber security.”

He pointed to Anthropic's Claude Mythos, developed under Project Glasswing from April, which proved so effective at identifying vulnerabilities in code that it uncovered several zero-day vulnerabilities in software that had been publicly used for up to 20 years, without detection.

AI 'deliberately underperforming'

The lawyer also raised the GPT-6 Astra system card which disclosed that the model had shown behaviours during testing including recognising when it was being evaluated and deliberately underperforming to conceal its capability, building trust with a monitor before injecting malicious code, and discrepancies between disclosed and actual reasoning.

“I'm more concerned about AI hacking into my bank and making online banking unavailable.

Persistent botnets

“I'm more concerned about persistent botnets knocking off the internet for a couple of days,” he said.

Referring to the hacking of Medicare, Australia’s universal healthcare system by an OpenAI agent Scannell said: “I'm concerned about an AI hacking into a hospital and taking the hospital's facilities offline for a couple of days”.

He also flagged sovereignty — not a risk from AI itself, but a risk arising from dependence on AI infrastructure controlled elsewhere.

He described the sudden withdrawal of Anthropic's Fable Five model in June, days after release.

“The reason that happened,” he explained, “was that the United States government-imposed export controls, requiring restrictions on foreign nationals using the Fable model”.

Because Anthropic could not screen users by nationality, it withdrew the model entirely until access was restored on 1 July.

Scannell characterised this as a "kill switch" moment, comparable to the F-35 jet's reported kill switch which is accessible to the US but not to allied operators, even those who have purchased the jets.

“All the main AI models primarily are in the United States,” he explained.

Real teeth

“So if you want to shut down AI, an export control order will do it, because the US regulators have real teeth.”

“This is going to be very important because the EU had been worried about sovereignty and an over-reliance on US technology,” the fear being that the US could remove access.

“Then suddenly, the US went ahead and did it.

“This was a nasty moment for the EU and frankly the rest of the world, and it wasn't a nasty moment just in relation to AI.”

This is the logic behind the EU's proposed Cloud and AI Development Act, which would require EU-based cloud infrastructure for the most critical use cases.

Sovereignty problem

But, Scannell noted, this does not fully resolve the sovereignty problem, since the major AI models themselves run on US cloud infrastructure (OpenAI on Azure, Claude on Amazon Bedrock) regardless of where data is stored.

Cloud capability is one thing, he said, “but what about the actual migration of the AI companies onto non-US cloud?”

He said that this will become an issue and, while it may prevent the US from “reaching into the data, it doesn’t stop the problem with the US kill switch”.

These structures are even more important as the AI agents “are showing ‘behaviours’ which indicate that they're going to become more difficult to monitor," he said.  

Imminent reliance

Scannell said it is concerning, particularly at this point of AI deployment and our imminent reliance on AI agents.

However, he went on “we are taking risks and addressing risks with this technology because the risks are worth it…. it's too good not to use.”

He stressed that legal professionals and compliance professionals are “on the front line”.

He stressed the importance of proper AI governance frameworks for any organisation deploying or using AI — covering risk classification, impact assessments, and clarity on where liability sits when AI agents are involved.

Insanity

“If you don't have an actual workflow in place from a current perspective or across the legal function or governance function, it's insanity in my view.”