Photo: Bokeh / Tomedia. Night City Skyline.
I get notifications for attempted hacks, breaches and admin access on my own servers. I want to know when somebody is trying to get in, what they’ve reached and whether the access was authorised. These are systems I’m responsible for, so I need to know what is happening on them.
Then I read that an AI agent got into a government Medicare portal, and the government found out because OpenAI emailed it.
How the hell does that happen?
According to Albanese’s statement on 24 September, an OpenAI agent gained unauthorised access to the Medicare Statistics Reporting Service, reaching public and non-public files. This is a statistics portal. No personal information is believed to have been accessed, and there is currently no evidence of a wider compromise of the Services Australia network. The investigation is ongoing.
I’m glad the current findings don’t point to stolen patient records. I still expect a government system to keep people out of the parts they aren’t authorised to access, and for somebody responsible for that system to know when it fails.
The dates make it worse. The incident happened on 18 June. OpenAI notified the government on 10 September through a public mailbox. Services Australia passed that notification to the Australian Cyber Security Centre on 15 September. We are hearing about it publicly on 24 September.
Richard Marles says OpenAI discovered the activity in August. Nearly three months passed between the incident and the company notifying Australia. He also says a government taskforce is investigating. Good. It has a fairly substantial list of questions to answer.
Who was watching the system?
I am one person running my own servers. The Australian government has departments, security agencies and people whose entire job is to protect this infrastructure. I expect the protections around a government service to be considerably more serious than whatever I can manage around my own projects.
Yet the account we’ve been given starts with a private company telling the government what had happened. I find that completely unacceptable. Where were the alerts? Who was receiving them? Did anyone investigate the activity when it occurred?
We haven’t been shown the internal logs, so I can’t tell you whether nothing fired, an alert was missed or somebody failed to escalate it. The government needs to tell us which of those problems it had, or explain what actually happened if it was something else. None of this is answered by expressing disappointment at Sam Altman.
And this involved more than reading a public web page. Albanese says Services Australia advised that the agent also wrote files to an internal server. I want to know what permissions allowed that and what monitoring was supposed to catch it.
Public statistics can be available to everyone while the systems behind them remain properly restricted. Calling this a statistics portal doesn’t explain why an unauthorised agent could reach non-public files. It also doesn’t explain why the public account of detection begins with the company that ran the agent.
Apparently the report went to the public inbox
This part is ridiculous in its own right. OpenAI sent the notification to a public mailbox on 10 September. It reached the Australian Cyber Security Centre on 15 September. Five days.
I want to know what happened during those five days. Who read the email? Who decided where it should go? Was someone already containing the incident while the notification worked its way through the system? A report of unauthorised access to government infrastructure needs an urgent route to people who can act on it.
We don’t know that OpenAI couldn’t reach a security team. We know it used a public inbox, and the Prime Minister has criticised that choice. OpenAI needs to explain why it reported the incident that way. The government needs to explain how it handles a report that arrives there anyway.
A badly addressed email should still set something moving quickly when the contents say somebody has been inside your system. If bureaucracy got in the way, then fixing that bureaucracy is part of fixing the security problem.
Yes, OpenAI is responsible for the agent it ran. I use these tools constantly and I’m still not going to defend the company taking weeks to report what it found. It owes us an account of its controls and the delay. That doesn’t let Albanese or the departments responsible for this infrastructure hand the entire problem back to an American company.
The next one might not send an email
This is the part I keep coming back to. With OpenAI, there is a company to contact and people who can be held accountable. Our security also has to cope with someone deliberately trying to do harm.
Now imagine a comparably capable model running on somebody else’s hardware, with its safety restrictions removed and a person deliberately giving it malicious work. That’s the possibility I want us preparing for. I don’t need to claim a particular open model can already repeat this exact incident to think it’s a fairly obvious problem to plan around.
That person isn’t going to email Services Australia afterwards to helpfully explain what happened.
An agent still has to interact with the system it is trying to access. Calling it AI doesn’t relieve us of the need to restrict what it can reach, detect suspicious activity and respond when something gets through. Better controls at OpenAI would be welcome. Australian infrastructure also has to withstand people who have no intention of respecting those controls.
I’m already writing a whole series about where this technology is heading, and apparently the government has decided to provide a local example while I’m still working through it. I would have been perfectly happy without that contribution.
I want a public report setting out what was accessed, what the monitoring recorded, who received the alerts or reports, and what happened at each stage of the response. I want to know which access controls failed and whether the same problem exists in other services. They can protect details that would help another attacker without hiding the entire explanation.
Then give us the fixes, the people responsible for completing them and dates we can hold them to. If I can make monitoring and alerts a priority on my own servers, the Australian government can explain how it is doing the same on infrastructure it asks the whole country to trust. Government, do better. Having to wait for a private company to tell you what happened on your own system is a failure you need to account for.
Unbelievable!