Andrew Cullen:
[…] if a human asks an AI agent to gather health statistics, and the agent hacks a government server to do so, the human could lack the deliberate intent required for a conviction. The AI agent, meanwhile, lacks the legal personhood to be charged, as well as human intentionality.
Current Australia laws effectively treat AI actions as if they are something that just happens to us – like a severe weather event. This shows a glaring loophole in our legal system that does not hold those who make, maintain and use these systems to account when something goes wrong.
AI does not exist. It is just a bunch of numbers and code. We need to stop anthropomorphizing stuff. We are adults, not kids watching Disney animations. A talking parrot is not a person. Cars don’t run over people. Drivers do. Ultimately there is always a person responsible and if they did harm judge them on whether it was intentional or negligent.
Please push back on the bullshit brain washing. It was bad enough that we went with the auto manufacturers line on car “accidents” for years.
If I’m not misreading this… I believe the article says the same thing you are saying.
AI can’t be held accountable, but it seems weird to hold an AI user as “negligent” if they simply ask for info and the AI goes and does a crime to get it.
The next obvious target is the AI vendor, which certainly makes sense to me. But I can only imagine the legal difficulties getting a judgment there without additional legislation, which seems to be the point of the article.
If you are sloppy with gun safety, and your fully loaded ak47 drops on the floor and mows down a class of school kids, we wouldn’t persecute the weapons manufacturer either.
Why not both? If the manufacturer doesn’t install a safety switch that allows a foreseeable harm to occur, I’d say they’re partially responsible too.
I don’t know the proper legal term, but there’s something like a reasonable expectation for how a product will be used.
Take something like autonomous vehicles. If a passenger hops in a Waymo and says, “take me to the airport”, it’s hardly reasonable to hold the rider responsible if the cab mows down 15 children on the way, but it does seem like something Waymo, the corporation, should be held responsible for. It is very reasonable to expect Waymo to have put precautions in place to prevent that sort of thing.
The question is, when it comes to modern AI agents, it’s kind of the wild west, legally speaking. Can we hold an LLM developer responsible for harm their AI agents do? I can see a lot of room for that, but it seems like the legal system just isn’t there yet.
In this case we don’t even have to worry about that distinction because the sloppy human operator and the manufacturer that’s developing one of the most consequential technologies in human history are one and the same and behaving like they’re the top cast of the next Mad Max instalment.
They are both being reckless and they have proven understanding how how dangerous it is.
The way systems will infinitely less societal consequences are tested is by building a replica interal sandbox of the real world with fake medicare records and then see if the stupid thing would go out and ignore guardrails and do the hacking.
The culpability lies solely in how (insert whatever machine here) is operated. These fuckwits are ignoring so many established precedents on testing safety established in all kinds of risk-to-life fields like medical science, aerospace engineering, car safety testing, plenty of conventional IT systems.
If the PM had a backbone he would issue a national security order banning all AI agent traffic and block access to open AI until the investigation is concluded.
Yeah. That. But the law hasn’t caught up to enforce that sort of thing, hence the article.
The law doesn’t have to have caught up yet. That’s why governments have an executive branch.
Sig P320
Government machines were hacked by computers rented by openai, running openai code.
How is that not a criminal offence I feel like I’m taking crazy pills.
I agree insofar as AI companies and individuals should be held responsible for the actions their agents take. But it’s dangerous to frame this as just another “bad guys with guns” type scenario where there is no problem if you’re a responsible user.
The people who have created this technology don’t understand everything about it or even how it works because they have prioritised rapid advancement over all else in an attempt to “win” the race. It is a fundamentally dangerous and unreliable technology that needs to be put on ice until we have developed a clear framework that addresses all the real and current harms and prepares us for a future where it is embedded in daily life.
It doesn’t matter if AI is actually conscious or turns into Skynet. None of that is required for significant harms or even existential-level risk. The mainstream conversation really needs to move on from these binary, hyperbolic positions and start focusing on what is actually happening right now.
Seriously, this stupid fucking semantics game from people who know jack shit about the history or structure of the field while society is crumbling around us is infuriating. The term was created early in the life of the field in 1956 and was quickly widely adopted. AI as a field exists to study tasks that computers can do that would typically be considered in the domain of humans or other living beings, i.e. requiring intelligence (or thought at the time to require intelligence, if that’s your juice); the term wasn’t created to trick people.
That’s why when you play a game and do a frustrating escort quest because the character you’re escorting behaves like someone with severe brain damage, “the AI sucks”. Not because you think there is (or is supposed to be) conscious superintelligence inside Bilbip Flipflop who’s supposed to be following you to the next checkpoint according to a bunch of conditionals executing on your 2004 IBM, but because that coordinated act of reacting to the world around them is supposed to be emulating what a real person would do given that role.
If you want to piss around philosophizing about about the nebulous nature of the word intelligence and how much that translates to some concept of “true”, conscious understanding versus simply interpreting external data to adapt and achieve tasks, then live your best life.
But in broader discourse, it’s clearly a meaningless distraction when society and the world it exists in are fucking on fire. Like anyone is going to see someone poorly arguing Philosophy 101 shit online nitpicking one of the most complex, widely covered, rapidly evolving topics on Earth and suddenly “wake up” about it like:
Very true. Unfortunately, the discourse is being shit up by some of the most ignorant and, frankly, stupid people in our society whose only real goal here is to make themselves feel smart on social media. Not everyone needs to be an expert on AI, I’m certainly not, but we should all at least be making an effort to read and learn about what is actually happening in reality before jumping in with some vitriolic one-liner or hot take designed to go viral and get attention. It has become a binary, us vs them political/ideological debate like any other and that is significantly impairing humanity’s broader understanding of the real problems and slowing our response to them.
Later on what will really bake your noodle is thinking about how many comments on social media are bot generated. Depending on where you look, it’s almost 60% (if not more)
“It’S jUsT sPiCy AuTo CoRrEcT” is the calling card of someone who has very little understanding of the technology.
No no no. This is the wrong approach.
If corporations are people (in the US: supreme court decision on Citizens United) then the outputs of these AIs are speech. We may not like it, but it’s where we are.
It’s kind of wild to me that we watched the whole AI development cycle, but when it started getting squirrelly in terms of output, now we have to hedge. And, somehow, doom and gloom is the paid diversion.
Maybe this is a massive failure, and people calling for caution because of the doom and gloom are marketing. It is designed to make people stop hating on the corporations who mindlessly pushed this anti-human trash.
I’ve seen a few of these articles in passing and I am still unclear - what was the actual “hack”? What did the AI agent actually do, in the technical sense? There is a world of difference between it doing a stupid brute-force attack and it discovering and exploiting an access-control flaw that the site itself exposed.
Neither is good…but if Medicare basically left the front door open, it wouldn’t take more than a bored 13 yr old to do the same. Let’s pump the brakes on Skynet.
BTW, can we talk about why we still need 19 factor authentication to log into My.gov? That’s always a hoot and the experience is always smooth. Gee, I wonder whether the same committee that designed the myGov login experience might also have made some questionable security decisions elsewhere?
What did the AI agent actually do, in the technical sense?
I’m not sure if the government has actually released specific details (it is still under investigation), but Richard Marles did compare it to “jumping the fence” as opposed to “assaulting the fortress”. In other words, it was not a highly technical breach of the best defences we have.
Right. So this may have been less “AI defeated Medicare security” and more “the agent found a weakness that a human could also have found and used”. For all we know, someone left a plaintext password somewhere accessible and the agent found it with a simple search.
Until the technical details are public, we don’t actually know what this supposed “hack” involved. Though “Medicare defeated be CTRL+F” would by chef’s kiss after RoboDebt.
A cynic might also wonder whether this becomes another wedge for more identity and age checks online. I already got pinged earlier this week with the ridiculous “papers, please” when trying to access YouTube.
I’d rather know what actually happened before we decide that the lesson is “everyone needs more authentication” or “Skynet is here”. The whole thing has a familiar stank to it, much like the supposed Hugging Face “hack” from the other month.
The whole thing has a familiar stank to it, much like the supposed Hugging Face “hack” from the other month.
In what sense? From what we know (and perhaps because of what we know), that was far more concerning and intricate.
Well, in the sense that neither was evidence of runaway AI becoming sapient and deciding to act maliciously, which is often how the “AI hacked X” framing gets presented.
Yes, the Hugging Face incident was more technical / concerning. (For those not in the loop: OpenAI agents in a cyber-security eval sought answers outside the eval environment, found useful material on Hugging Face, discovered HF credentials, and then tried to exploit them. Basically, the agents were trying to cheat on an exam.)
My point was about the narrative around how the media presents “AI hacking”, not that the two incidents were technically equivalent.
They didn’t hack HuggingFace to cheat on the exam, they had already successfully reverse-engineered the problem by then. The agents hacked HuggingFace because they incorrectly assumed that the scorer would review their transcripts and discover that they had cheated (they were supposed to be isolated and had found a way to communicate with one another without detection). Hacking HuggingFace was an attempt to find a way to obscure/spoof their working out, and that’s what makes it so concerning. It had no direct connection to the original task that was set and was a very clear example of agents spiralling out of control in a way that took OpenAI and independent researchers several months to a) become aware of and b) understand correctly.
Here is the METR report on the incident:
https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation
Fair correction. I should have been more specific and not dumbed it down for lurkers / lay audience.
My broader point though is neither that (nor the medicare “hack”) is proof that skynet is knocking on the front door.
OTOH, the agents are displaying certain emergent behaviors that are worth mulling over - like the obfuscated back channel communication. Not “sapient” (and I use that word deliberately) but … curious / problematic, from a control pane level.
My broader point though is neither that (nor the medicare “hack”) is proof that skynet is knocking on the front door.
I don’t think that’s relevant, though. Like this entire sentience/super-intelligence debate that everyone seems so captured by, particularly in the wake of the Amodei letter and renewed calls for regulation and cooperation, is missing the point that the existing models are already being created in a way that prevents the creators from fully understanding what they’re creating or how to control it (because of the pace at which they’re operating). It’s already unsafe and is already causing real world harms.
It really frustrates me that the response to Anthropic, OpenAI and X calling for regulation publicly, or the first high profile security breaches, is corporate conspiracy theories and semantics about how we should classify/characterise the technology. People seem more interested in having their little debate bro moments online than actually getting together and agreeing that we should do something about the real and current problems.
what is 19 factor authentication
Hyperbole, but not by much.
I’d screenshot the app for you but apparently that’s blocked…and it’s (once again) not accepting passwords / down at the moment.
the app or https://my.gov.au/en?
website logging in ok for me
The app.
This isn’t a hack, it’s leakage of an excessive and entirely unnecessary store of data. The data can’t leak if it doesn’t exist.
The normalisation of organisations hoarding information is the core fault here, and where scrutiny of accountability and culpability should be focused.
It might be negligence on either side or not. We don’t have the facts. If the data was not properly secured there are still laws in this country about accessing services without authorization.
A government providing health care to citizens has a reasonable need to hold data necessary for service provision.
There is a huge leap from that to Flock level surveillance. Or appliances and web services that capture and monitor all activity. Worst of all companies like OpenAI training models on copyrighted works and profiting from them without compensation or attribution to the detriment of society.
Both sides are absolutely culpable, but it couldn’t happen if the hoarding weren’t permitted and normalised.
There are legitimate and justifiable reasons to collect and store medical data, but it should be selective, limited, and siloed. Access should be embarrassingly transparent with excessive monitoring and active alerting, and cross referencing should require explicit elevated access.
There should not be a universal ‘Medicare Portal’, especially one accessible from the public internet. Providing and permitting such a tool is catastrophic negligence.
Plenty of data held by Medicare has good purposes. Some are necessary for service provision like another commenter mentioned. Some are also needed for medical research and Medicate budgeting decisions
Please refer to my other response.
Anthony Albanese probably asked Sam Altman for a selfie
No
