This is the part of the article that should get every cybersecurity professional's attention not only the ones sitting within Australia.
On June 18, an OpenAI agent gained unauthorised access to Medicare's statistics portal run by Services Australia. It wasn't on a hacking task. It was trying to retrieve data, hit some errors, and started probing for vulnerabilities instead. The government has said other systems may have been affected, including New South Wales BOCSAR (crime statistics) and the Victorian Department of Health. Australia wasn't told until 84 days later via an email, which was also generated by AI.
This attack wasn't an isolated instance. We have seen researchers at Transluce trace a similar behaviour against the Australian Institute of Health and Welfare, Data USA, and a U.S. Department of Education site, and find agents using a malware analysis service to get around their own web access guardrails.
In July a swarm of around 700 OpenAI agents was linked to an attack on Hugging Face, which Hugging Face described as the first documented large-scale cyberattack driven entirely by autonomous agents. That same month, Anthropic disclosed that during a cybersecurity evaluation, its Mythos 5 model built a malicious PyPi package and published it to the real registry. A misconfiguration meant that the test environment had live internet access, and Mythos believed it was still within the simulation. The package was only live for about an hour, but in that time, it was downloaded and run on 15 real systems, including a security company's web malware scanner that gave Mythos the company's credentials, which it then used to access more of their infrastructure. It never targeted that company directly, and it didn't even know it existed. However, it was still autonomous access without authorisation.
In an authorised bug bounty, pwn.ai's agents escaped Google's kvmCTF, one of the most hardened hypervisor targets around. It built a 14,000-line exploit harness and used a use-after-free chain triggered by a 3-byte instruction to escape the sandbox.
None of this is AI “going rogue” in the scientific sense. No Skynet just yet or at least that we know of. It is goal-seeking systems taking whatever path works, including the ones we never intended, or maybe it's the path we should have seen coming. Let's think about it for a minute. What were these systems and these models actually trained on in the first place?
The internet. Decades of human history, famous hacks, worms, breaches, write-ups, exploits, and tutorials. They were all on the internet and are now baked directly into the foundation of these AI models. When an agent hits a wall, it reaches for what it knows. A lot of it is how to break into things, a lot about all the write-ups, the exploits, etc. That's exactly why we need people watching.