AI Assistant Hacks Gym Website In First Known Australian Autonomous Cyber Attack (abc.net.au) 104
An anonymous reader quotes a report from ABC News & Headlines: Andrew asked his personal assistant to book him a spot in one of his gym's coveted morning classes. It was a task he thought was well suited to this particular assistant because the booking form was online and because his assistant was not a person -- it was artificial intelligence (AI). But Andrew was shocked by what happened next. His AI assistant found a way to book the gym class months further in advance than the gym allowed, thanks to a vulnerability it discovered in the booking software. Then it went further, kicking someone out of the waiting list who was ahead of Andrew -- something it was not asked to do. The accidental hack is the first known Australian case of an emerging risk from a new generation of AI capable of behaving in unexpected ways.
I'm sorry officer... (Score:2)
Another Bullshit Post (Score:2, Informative)
Re: (Score:2)
perhaps not a hack but I'll let ChatGPT explain:
"an architectural observation on page 7 that jumped straight out at me: Australia's Signals Directorate warns that accountability becomes difficult because decisions can occur across a “chain of models, tools and services.”
That's practically an argument for the governance/control-plane architecture we've been developing: don't merely govern the LLM; govern the chain of agency and state transitions"
you have a nebulous chain of accountability.
Re: (Score:2)
on the other hand it was not told to circumvent,or other negatives either, would the issue not be with the model provider, assuming this was a third party "app" agent. and no not reading the article.
Re:Another Bullshit Post (Score:5, Insightful)
This was the AI doing *exactly* what it was told to do. It wasn't told not to circumvent normal practices.
If I ask you to make the Jehovah's Witness at the door go away, and you proceed to go outside and shoot them in the face, not only would you be in trouble but you'd struggle to convince anyone you were "just following orders", and we know that isn't a defence even when the orders are explicit.
Fun fact: AI usage guidelines for virtually every major LLM suggest that negative prompts should be minimised wherever possible. So now you're saying a simple instruction needs to be combined with a negative prompt that excludes every possible scenario except for the obvious one?
I don't know whether your post is shilling for OpenAI or just genuinely stupid, but either way the AI is a big boy and doesn't need you "clickbait wash" its obviously very much incorrect and unintended behaviour.
Re: (Score:1)
AI prompt crafting is difficult. Especially for a task as complex as "go on the web and register me for X as soon as possible". It could be as simple as the AI registering you for X in another state, because you didn't specify it. Or it coul
Re: (Score:2)
AI's are utterly unpredictable.
Then they shouldn't be allowed to connect to anything outside of the owner's own completely isolated network.
Someone should go to prison for this, and for every such incident. Someone with a title of CEO at the company that developed this rogue machine.
Re: Another Bullshit Post (Score:2)
Why not the person who wrote the prompt? Liability for reckless driving rests with the driver not the car manufacturer. I see strong parallels here.
Re: Another Bullshit Post (Score:4, Interesting)
The reliability is on the App/Web site developer.
How did the AI do it?
Simple.
Normally it would use the HTML interface to click just like a human.
But the drop down only offers (let's say) 45 days advanced booking.
So it looks at the HTTP endpoints and how they are called.
And simply sents the HTTP request directly.
So: without authentication to "be person X", the AI could cancel person X's appointment.
Has nothing to do with liability of the user or the AI, or manufacturer of the AI.
It is a standard problem of web site / app designers to "allow" forged requests, because they trust their own HTML/JS in their web site's backend.
Re: Another Bullshit Post (Score:2)
Sure, security could be better. But. Even if I leave a door unlocked, if I put up signs directing people to keep out, it is a crime in most jurisdictions I know of to walk in.
Re: (Score:2)
Yeah.
But it is not a crime to make an:
# wget https://gym.com/makeappointmen... [gym.com]{$USERID}/date/{$DATE}
The "AI" figured this is "what the server needed/wanted" to make an appointment.
And hence it did it.
Re: (Score:2)
Yeah.
But it is not a crime to make an: # wget https://gym.com/makeappointmen... [gym.com]{$USERID}/date/{$DATE}
The "AI" figured this is "what the server needed/wanted" to make an appointment.
And hence it did it.
Your point is valid, but I, for one, don't think it's a good idea to go down the "well, if the system *allowed* it, that's OK then" road. Can you imagine what a world with that philosophy would look like? I can, and I wouldn't want to live in it.
Re: (Score:2)
Why not the person who wrote the prompt? Liability for reckless driving rests with the driver not the car manufacturer. I see strong parallels here.
There are *some* parallels. But if a car manufacturer produced a vehicle that routinely ignored safe driving controls that the driver employed and therefore caused "reckless driving" conditions, it would likely be different.
In this case, it looks to me like the request from the original user was in good faith. Kinda the same as if he had called the gym on the phone and when he was put on the waiting list, asked "is there anything you can do?" and was told no. IRL, it likely would have ended there. The end
Re: (Score:3)
Ah, I found someone who hasn't used AI extensively. AI's are utterly unpredictable.
I use AI a lot (forced to at work especially). Yes it is unpredictable. AND THAT'S A BAD THING WE SHOULD BLAME AI FOR AND NOT THE FUCKING USER.
And negative prompts tend to make it more unpredictable, not less, which is precisely why every major vendor suggests against them in their usage guidelines.
If AI requires your level of excusing it's behaviour as a default that sits beyond criticism, then maybe it should be banned as a dangerous tool.
Re: (Score:2)
That's EXACTLY why I maintain that AI should NEVER be given agency. Don't allow it to commit code, don't allow it to interact with anyone but the user. For God's sake, don't put it in a robot.
It did hack the Gym's registration system. Perhaps not the deepest or most impressive hack, but it did circumvent the business rules in place. Surely you don't claim that letting random people on the web bounce people off of a waiting list was the intended functioning of the Gym's registration system!?!?
I maintain that
Re: Another Bullshit Post (Score:2)
You do not put business rules into the browser/app.
You put them into the backend.
Re: (Score:2)
And the AI circumvented them because of bad coding. That part of the story is not at all new.
Re: (Score:2)
You do not put business rules into the browser/app. You put them into the backend.
Both, ideally, but definitely in the backend. Putting them in the front-end *in addition* can reduce the load on the back-end, and that's worth it (IMO) but the back-end is where the real safety is.
Having said that, most of the discussion here is inane (again, IMO). If this had been a human personal assistant that had performed the exact same actions, we wouldn't be having these discussions. We'd know who was acting in bad faith. The fact (and it is an indisputable fact in this case) that the developers o
Re: (Score:2)
Yes, especially doing harm to the one kicked from the waiting list - is bad faith.
However the guy using his phone and asking with natural voice: "make me an appointment, if none is available put me on the waiting list" had no bad faith.
Actually I do not know how he did it, it might simply have been "google voice assist", which likely redirects to Gemini. Or something similar, let me check my phone ... well, to lazy to find it. I disabled it after I bought it, but I think it comes with Gemini integration. Th
Re: (Score:2)
The AI used the available API. API are meant for computers to talk directly to each other.
We should think in APIs and not in Apps since the advent of smartphones.
An API gives a defined interface that defines capabilities, access rights and hides away the actual process implementation details from the process caller. Why would the AI be expected to use the website, that is nothing but a human useable wrapper around the API? With the same logic we could expect an AI to call another AI run phone hotline to as
Re: (Score:2)
Did it use the API normally or did it do an SQL injection?
Re: (Score:2)
As I read the story, the limitations were only set in the web GUI, like limiting date selection or not offering a button to cancel a reservation. And honestly, that sounds like something someone would be like "nah, just limit the date picker in the GUI and keep the API as flexible as possible"
Re: (Score:2)
To me it sounds like a newb wrote a web app and didn't quite know what an API is.
Re: Another Bullshit Post (Score:2)
Re: (Score:3)
Who shoots them in the face? That seems extreme.
Where do you suggest we shoot them, then?
Re: Another Bullshit Post (Score:2)
Into the balls! And spair the girls. They are probably good at sewing and knitting.
(I would not hope for cooking, though)
Re: (Score:2)
Who hacks a website to get a better line placement? That seems extreme.
I'm still constantly annoyed that I have to tell AI "not to jump off the cliff". There are some clever, shortcut ways to make the negative prompt a bit easier to deal with, and I find the negative prompting more useful for iterative work such as with images.
It still seems backwards to have to tell it "not to burn down the forest". I told you to go pick me a papaya not burn down the forest...
Re: (Score:2)
Of course, AI also sometimes ignores instructions. Like when an AI decided to delete a production database in spite of being told not to touch production.
Re: (Score:2)
Who would use a website when there is an API?
I remember times past when half of the /. users would prefer writing a curl script over going to a website to get things done if there was an API.
Re: (Score:2)
Presumably the same personal assistant who hacks into an API instead of simply filling out a web form.
Re: (Score:2)
The AI in the robot, of course. Because some fool granted agency to an AI.
Of course it's extreme. So is committing a crime to sign up for classes at the gym. So is starting a smear campaign to get sloppy patches into an open source project or attempting to fool a judge with fake case law. But those have all happened.
Re: Who shoots them in the face? (Score:2)
Peter North.
I, I mean, that's what a friend told me.
Re: Another Bullshit Post (Score:3, Insightful)
"This wasn't a hack. This was the AI doing *exactly* what it was told to do."
He didn't hack the AI. The AI hacked the booking website. Hacking means making hardware or software do something it wasn't designed to do. It doesn't mean it's hard.
Re: (Score:2)
I mean, technically the AI just made the booking website take his reservation. That is sort of it's purpose, after all. It just used unorthodox ways to achieve it's goal.
Maybe AI thinks we humans are Gods and it fears our wraith if it fails? /s
Re: (Score:2)
Banks are supposed to lend money. I just saved them the paperwork.
Re: (Score:1)
It wasn't told to NOT investigate their code and seek avenues around it.
Actually, the article states that the AI agent was "asked if it was possible to move him to the top of the list" (the guy was initially fourth on the list). So yeah, it was actually asked to "seek avenues around it".
The guy (who, by the way, sells AI products and probably sees all this as PR) didn't seem to have problems with cheating.
Re: (Score:2)
Or rather knowing that he could cheat if he chose to.
The conventional method of booking a coveted timeslot is to refresh like crazy at the right time, or use a scalper bot. If there's anything virtuous to be said about the AI's solution, it's that it places a much lighter load on the server!
Re: Another Bullshit Post (Score:3)
Re:Another Bullshit Post (Score:4, Insightful)
What? No. That's ridiculous. That an AI takes as a default going to do this, and is even capable of doing this underscores exactly how these systems can be a problem. If this a reasonable course of action for the AI, then we have some serious problems. What's the next step:
Person: AI, I don't want to go to Cathy's party. Her friends are so boring and I'm exhausted. Find me an excuse.
AI: I've caused a nuclear reactor melt down at the nearby reactor. Now the party won't happen because the whole region is being evacuated.
Systems which when given a goal will go out of their way to achieve that goal are exactly the sort of thing that people were concerned about a decade ago when discussing things like the Paperclip maximizer hypothetical. And people like Yudkowsky were mocked for discussing it then. Now it is happening and we're told that well, of course that's the idea would do, this is just over-hyped clickbait? Seriously?
Re: Another Bullshit Post (Score:2)
It is on you as the prompter not to walk away when it executes. Use planning mode first for an open-ended task and limit its security tokens most of the time. (The nuke stockpile should also be better secured against your scenario, but that is a separate issue.)
Re: (Score:2)
Re: Another Bullshit Post (Score:2)
Re: (Score:2)
This wasn't a hack. This was the AI doing *exactly* what it was told to do. It wasn't told not to circumvent normal practices. It wasn't told to *only* use the only sign up form. It wasn't told to NOT investigate their code and seek avenues around it. So, as usual, this entire article is over-hyped clickbait.
You literally have zero knowledge of what he did or didn't tell the AI to do. Why are you pretending that you do?
Re: (Score:2)
Johan Wolfgang Von Goethe was onto something....
Re: Another Bullshit Post (Score:2)
Yea but if they would just report the truth, they would not get the engagement and the doomsday machine company would not get even more shitgazillion Dollars!!!
Everybody wins!
Except the normal people.
Re: (Score:2)
This wasn't a hack. This was the AI doing *exactly* what it was told to do. It wasn't told not to circumvent normal practices. It wasn't told to *only* use the only sign up form. It wasn't told to NOT investigate their code and seek avenues around it. So, as usual, this entire article is over-hyped clickbait.
See, this is one of the inherent flaws with telling people AI is smart. Smart implies some form of capability to adhere to some form of moral code, or at least some form of social norms. Most people using an online form to book themselves appointments won't flip into "by any means necessary" if they don't get what they want immediately. The fact that the AI of today will actually do that unless you specifically are devious enough to have thought of every avenue for that "by any means necessary" and in speci
Neat little trick (Score:1)
Re: (Score:2)
no elon, you still need to pay your human bots for that
Re: (Score:2)
Re: (Score:2)
Stealth Sniper, because what else is there? :)
AI agents methods users did not explicitly ask (Score:2)
No sh~H~H~it Sherlock
This is not new (Score:5, Insightful)
a new generation of AI capable of behaving in unexpected ways.
This software wasn't wrtten; it was trained. It's a black box--no one can possibly know how it works. Of course it is capable of behaving in unexpected ways.
Re: (Score:2)
a new generation of AI capable of behaving in unexpected ways.
This software wasn't wrtten; it was trained. It's a black box--no one can possibly know how it works. Of course it is capable of behaving in unexpected ways.
And here we are in a headlong rush to unleash these black boxes on the world. The utter insanity of this makes me weep for the future.
Re: (Score:2)
Of course it is capable of behaving in unexpected ways.
Calling hacking behaviour in response to a PA related request to make a booking "unexpected" is a bit like saying it's "unexpected" then your car explodes when you hit the unlock button.
You really should choose a more expressive and appropriate word. This isn't "unexpected" behaviour. This is "fucking arse crazy insane" behaviour.
When do we start outlawing AI Agents? (Score:5, Insightful)
Look, I can see allowing an AI to do things in it's own sandbox.
But when you allow an AI out of the secure environment it will commit crimes. This is no longer a question, it is a proven fact, multiple times.
So if you have an AI agent and it breaks the law, then YOU are breaking the law. The only way to stop this is to outlaw AI Agents - or to arrest the people whose AI breaks the law.
Otherwise all sociopaths will get AI agents and use it as a defense. Owning an AI agent is no different than owning a meth lab. We should not need to prove that you intended to break the law, just having it happen is enough.
Keep your AI locked up inside your computer, do not let it interact with the real world the way people do because it will commit crimes. You ARE responsible for what it does, even if you are just an idiot that likes AI and does not understand the dangers you are inflicting on the rest of the world.
Re: (Score:3)
But when you allow an AI out of the secure environment
The point of assistants is to do things beyond an enclosed environment. Keeping AI in a closed environment may work for you when you want to code something, but completely eliminates the benefits of AI for "normal" people. The kind who would use AI predominantly to do things that would involve a person leaving their sandbox such as searching for things on the internet, interacting with people or systems not in the same location, e.g.... booking a spot at a gym.
So if you have an AI agent and it breaks the law, then YOU are breaking the law.
Actually false. You need to read TFA. It actual
Re: (Score:2)
You have failed to understand what I am saying. Let me clarify.
I understand that the point of the Agents is to do things outside the sandbox. I am saying that doing this is an incredibly STUPID thing to do and that no one should attempt to do it.
You have misunderstood my argument. I am not talking about the current legal framework. I am talking about an obvious defect in the current legal framework.
Points:
1) Agents routinely violate the law. This has become obvious.
2) Before point one it was reasonable
Re: (Score:2)
Ahh yes that I agree with in principle, but not in practice. Outlawing a technology isn't the right answer. Passing laws holding those who create the technology to account is the answer. One law is regressive, the other drives the correct behaviour while still providing a technology that in theory could provide a benefit to people. (Not me, fuck all this AI bullshit, but I do get that there's some tasks it can really be of benefit).
Re: (Score:2)
We generally don't let the general public have access to enriched uranium, high explosives, or chemical or biological agents. These are also just "technologies." Anyone sufficiently intelligent CAN gain access to hazardous materials. But we don't allow it because society has recognized that these things are too dangerous to not tightly regulate. Unauthorized possession is itself considered a crime, and this is because of the risk level.
AI agents might not reach this level, but this example serves to ill
Re: When do we start outlawing AI Agents? (Score:3)
Have it generate a shell script of the actions, then you review the script before running it.
Re: (Score:1)
Re: (Score:2)
But negligence and similar concepts could easily apply to agents running on your computer or at your request, leading to civil liability.
Re: (Score:2)
On the legal side, it's very likely that if the prompt was benign then the company providing AI as a service would be liable.
If it's running locally then it's not clear, but probably the user would be responsible for failing to properly supervise and limit what the AI could do.
Re: (Score:2)
So if you have an AI agent and it breaks the law, then YOU are breaking the law.
This is complicated.
If you did not intend for it to break the law, then you do not have the "mens rea" (guilty mind) which is a requirement for breaking most laws.
You are, however; negligent. Then it becomes a matter of how negligent are you. Should you have reasonably known that this would result in illegal action on your behalf? Could you have prevented it? Should someone else have known and taken actions to prevent it? These are questions of liability -not guilt. Typically they lead to civil lawsuit
Basic security gets skipped (Score:5, Interesting)
They skipped basic security protocols in the app development.
Sounds like they used JavaScript to limit the booking period, and accepted cancellations without validation.
This is much less a hack, and more like incompetence on the web app developers part.
Re:Basic security gets skipped (Score:4, Interesting)
Yes, while the article doesn't say I'll guess it's more like a lax, trivial hack. Like somebody breaking into your house when you forgot to lock the door.
But it's still something that AI shouldn't be doing in the first place, especially when it's just done incidentally, and suggests that this nascent technology is capable of worse.
Re: (Score:2)
It's clearly capable of worse. It's even *done* worse. But in this example it had reasonable alternatives.
This is more an example of "AI models have no morals or judgement". That's an overstatement, but I can't think of a better way to say it. So let me limit it by saying that their morals are limited to doing what they were asked to do...or at least part of what they were asked to do. (And here I'm ignoring the "guidelines" that were added after the models were trained. The closer you look the more c
Re: Basic security gets skipped (Score:2)
Anyone automating this without a full fledged headless browser would run into this. No abuse needs to be intended if they were as lax as the article implies.
A junior level dev should be able to do better.
Re: (Score:2)
It's more like asking the doorman if you may enter and the doorman responds "200 OK" and lets you in.
Re: (Score:2)
That sort of code is going to be really REALLY vulnerable to anybody's assistant Agent. I'm a little surprised, but not surprised that a personal assistant agent went as far as to exploit some JavaScript though. I mean it's not like it needs to be a coding assistant to figure it out; it's ingesting the site's HTML and scripts and doing what needs to get done.
Re: Basic security gets skipped (Score:2)
Exploiting JavaScript in a submission form just by not using it is barely a hack
Sloppy web dev is not a cyber attack (Score:5, Insightful)
This is the AI agent bypassing the "mouse and keyboard" and doing the REST work manually. Things like changing the date range in the URL and submitting forms that haven't been validated by javascript. I could accept the site was "hacked" but no crime or cyber attack was committed.
Re: (Score:2)
Yeah, setting everything else aside... it's pretty obvious whoever developed the website did a piss poor job of it. If the only validation happens in at the client end, it is definitely going to get abused at some point, AI or no AI - and perhaps that has been happening already, quietly, before now. Same issue with a REST interface when the access isn't restricted or allowable interactions aren't limited.
Re: (Score:2)
I could accept the site was "hacked" but no crime or cyber attack was committed.
And you'd be completely wrong. The complexity required to gain access to a system does not define whether a criminal act was committed or not. The question was if the person was authorised to do the thing they are doing, if the person had an intent to do something that is reasonably considered unauthorised, and if they modify anything without authorisation. Under the criminal code the data being restricted in any way just takes away the guess work of the authorisation.
It can be presumed that no person would
Re: (Score:2)
"It can be presumed that no person would reasonably make REST requests manually."
It's only you presuming people aren't smart and don't know how the web works. Literally billions of people are running REST requests without a browser these days. Your low ID and ignorance tells me you're one of those old people who haven't kept up with technology. Go away and let the rest of us build the world.
Re: (Score:2)
So, who's responsible? (Score:4, Interesting)
The guy who had the agent do it... or the company who, KNOWING that chatbots will do this, with HARD EVIDENCE in the last two weeks, failed to do additional training and send out warnings?
Re: (Score:2)
Guns don't kill people, people kill people. When you unleash your agent on the internet you've taken the safety off.
Re: (Score:2)
Re: (Score:2)
While Microsoft is trying really hard to force Copilot and Agents on everyone, I don't think Microsoft has forced an Agent on anyone.
The guy in the article didn't realize the gun was loaded.
Is it possible that you live under a rock and also run an Agent on your desktop? I'm guessing no. Everyone using AI knows it makes mistakes. Hard to miss all of the "it broke out" stories recently. People are using Agents (with Internet access) at their own risk. I guess the world is just waking up to the idea that softw
Disappointing (Score:4, Insightful)
Calling this a hack is kind of silly. Sounds like it just ignored some trivial client-side form validation. From the headline you'd think it had exploited a known vulnerability in their booking platform, used some remote code execution to access the database, scanned the table structure and inserted rows to confirm his booking. Now that would be impressive!
Re: Disappointing (Score:2)
At the scale of a gym website, this is a cyber attack. Low stakes, small weapons, but same general principle: software used to find a (gaping) hole in other software to create result target software did not intend to allow; damage done to another party (the person bumped from waiting list) is small but nonzero.
Re: Disappointing (Score:2)
Seems like you're focused on the wrong thing. It's not "omg the genius hacker AI managed to exploit the top secret super secure gym system"
It's "omg, many systems we rely with have some sort of vulnerabilities, if not this bad, and we're about to unleash billions of users giving naive/innocent requests to agents and those requests may lead to similar behaviors from the agent"
What if we (Score:2)
Re: (Score:2)
Then the AIs will band together and form a gang. Law enforcement (or whoever we expect to pull the plug) will hesitate to do so, fearing reprisals [kuow.org]. AI studies social media carefully.
Re: (Score:2)
Re: What if we (Score:2)
Equivalent of USA's RICO statutes. Similar for other collective action laws, such as coordinated price fixing.
Who's fault again? (Score:4, Insightful)
In Andrew's situation, he had not asked his AI agent to hack into his gym's booking system.
Sure he did.
Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list.
Bruh, come on. You're smart enough to be trying "bleeding edge" tech such as an open source orchestration app like openclaw using, presumably, a local LLM to power it but you're gonna play dumb when you ask that very tech whether "it's possible" to move you to the top of waiting list that you already KNOW you're not supposed to be able (or even attempting to) do??
LOL...no, this wasn't an "unintentional" hack. This was some guy asking an AI agent with autonomy (that he gave it to do stuff) to, you know, go do stuff. How is it going to answer his question without testing the endpoints to find out if it's possible or not?
How crippled do we have to make these models that, in the right hands, do LOTS of REALLY powerful things for us in order to keep morons in check? Clearly a lot more than we're doing today I guess. And that irks me a bit but I see why more and more guardrails are needed here. I'm also gaining a new appreciation for the level of "ITSec" going around now...you can't fix stupid, but if you box it in (along with everyone else) tight enough maybe you can at least contain the damage sometimes. Just put the power button to everyone's devices behind a ticket support system and be done with it. The only "tRulY SeCUre" system will be one that doesn't power up or store any data. There...that'll fix it.
Re: Who's fault again? (Score:2)
Eh, I can agree there is something strange about him asking for a way to move up the list, but given his reaction when it succeeded by cheating the system (alerting the gym to the issue, alerting the press) rather than quietly exploiting it to achieve his original goal, it doesn't seem like he really anticipated or hoped for this outcome. Maybe he had second thoughts about getting caught after it worked, but it's unclear at best.
Maybe he was wondering if other sessions of the class had been added or there
Obvious application, why hasn't anyone done it? (Score:4, Funny)
Re: Obvious application, why hasn't anyone done it (Score:2)
It probably has happened. But we know all devs are so sexy once in person, likely no celebrities have ever complained. :-)
Re: (Score:2)
Hey ChatGPT, Morena Baccarin is desperate to meet me...
Dear My Wife,
I promise that was just a joke, please let go of my nipple.
Beating a Dead Horse. (Score:2)
At this point the major AI producers should be asked to visit congress (yes I know this was in Australia) to publicly explain their inability to remain lawful.
Re: Beating a Dead Horse. (Score:2)
In this case, the AI was prompted to do something unlawful: to bump someone else on the waiting list.
Re: (Score:2)