NZ could be in the dark if an AI agent hacked a government system here

Good Ancestors New Zealand is calling on the government to require leading AI companies to report serious incidents, after an OpenAI agent accessed non-public files on an Australian Government health portal. OpenAI took a month to tell Australian officials after the company found out, which was almost three months after the incident actually occurred.

Australian Prime Minister Anthony Albanese disclosed the incident yesterday. During an internal OpenAI test in June, an agent looking up health statistics got around the access controls on an Australian Government healthcare statistics portal and obtained public and non-public files. CNN describes it as the first known AI hack of a government system.

OpenAI says "our models took actions we did not intend". The company found the activity in August but did not notify the Australian Government until 10 September, by email to a generic public disclosures inbox. Albanese has called this "unacceptable" and set up a taskforce to investigate.

Good Ancestors Executive Director Dan Foote says that according to reports the agent was not told to break into a government system. It did so while trying to complete an ordinary task.

"This is the behaviour AI safety researchers have been warning about, and it has now happened to a real government system across the Tasman. As these systems become more capable and more powerful, the gap between what they are asked to do and what they actually do matters more and more."

"New Zealand has no requirement for AI companies to report incidents like this. If an AI agent did the same thing to a New Zealand government system, we would be relying on the company choosing to tell us, and we might not find out at all."

Good Ancestors has recommended that leading AI companies be required to report incidents and risks to the New Zealand government, as they already do in California and the European Union. Its paper on AI incident reporting sets out how this could work.

Foote says Australia also had an AI Safety Institute ready to help respond and it is part of the investigating taskforce. New Zealand has no equivalent, and establishing one is the first of Good Ancestors' six policy recommendations.

"Incident reporting is a minimum first step, and an easy one. But the government also needs people with the expertise to make sense of these reports and advise on them, and that is what an AI Safety Institute would provide."