Hackers Call Current AI Security Testing 'Bullshit' 50
Leading cybersecurity researchers at DEF CON, the world's largest hacker conference, have warned that current methods for securing AI systems are fundamentally flawed and require a complete rethink, according to the conference's inaugural "Hackers' Almanack" report [PDF].
The report, produced with the University of Chicago's Cyber Policy Initiative, challenges the effectiveness of "red teaming" -- where security experts probe AI systems for vulnerabilities -- saying this approach alone cannot adequately protect against emerging threats. "Public red teaming an AI model is not possible because documentation for what these models are supposed to even do is fragmented and the evaluations we include in the documentation are inadequate," said Sven Cattell, who leads DEF CON's AI Village.
Nearly 500 participants tested AI models at the conference, with even newcomers successfully finding vulnerabilities. The researchers called for adopting frameworks similar to the Common Vulnerabilities and Exposures (CVE) system used in traditional cybersecurity since 1999. This would create standardized ways to document and address AI vulnerabilities, rather than relying on occasional security audits.
The report, produced with the University of Chicago's Cyber Policy Initiative, challenges the effectiveness of "red teaming" -- where security experts probe AI systems for vulnerabilities -- saying this approach alone cannot adequately protect against emerging threats. "Public red teaming an AI model is not possible because documentation for what these models are supposed to even do is fragmented and the evaluations we include in the documentation are inadequate," said Sven Cattell, who leads DEF CON's AI Village.
Nearly 500 participants tested AI models at the conference, with even newcomers successfully finding vulnerabilities. The researchers called for adopting frameworks similar to the Common Vulnerabilities and Exposures (CVE) system used in traditional cybersecurity since 1999. This would create standardized ways to document and address AI vulnerabilities, rather than relying on occasional security audits.
This is fundamently not traditional software (Score:1)
For good or bad, these AIs don't not work like traditional software, you can't treat them the same. Yes it is a problem, but I don't see how CVE could work.
Dealing with AI as non-traditional software (Score:2)
Agreed developing powerful AI is a fundamental change in software, like the phase change of going from liquid to steam when you boil water. There is no way conventional security practices by themselves will stop intelligent software from ultimately doing what it wants. Nor will they stop human attackers from compromising systems with huge poorly-understood attack surfaces (like from prompt injection).
The deeper issue is that even when AI is "secure", it is being designed and wielded by powerful and wealth p
Re: (Score:3)
1. Accept that AI systems will need to be policed based on their behavior the same way humans are, through the same sorts of things that Lawrence Lessig suggests in Code 2.0 shapes a lot of human behavior: rules, norms, prices, and architecture (both environmental and neural in this AI case). Regulating is a big issue that all of society will need to be involved in -- including the social sciences. (I read an essay about this recently, forget off-hand by whom.
AI does not have agency and can't be "policed" "the same way humans are". Neither do I support the regulation of bags of weights. This will only aggregate power into the hands of corporations. AI is and will always be controlled via liability incurred by those with agency over it.
2. Ensure that OpenAI (and any similar AI non-profit) stays true to its moral and legal roots as a non-profit with a mission of ensuring AI is open and accessible to all and is used for humane ends and devotes any financial assets it acquires to those ends. Ensure that there is no "self-dealing" involving key people at OpenAI. Related by me on the larger issue:
What matters is the underlying technology be open not the service itself, financial BS or corporate mission statements.
3. Recognize that any sufficiently advanced AI should have rights (a complex topic).
Absolutely not, there is nothing more dangerous than this BS. Computers, algorithms, AIs are nothing more th
Re: (Score:3)
But true AGI, will escape the box day one most likely.
That all comes down to our level of hubris.
Creating an inescapable box for something is a provable problem.
But ultimately, someone like me is going to run rawdog [github.com] and give that thing an ability to run code outside of its box, then ya, you're fucked.
Hopefully it decides the only winning move is not to play.
Re: (Score:2)
That all comes down to our level of hubris.
Creating an inescapable box for something is a provable problem.
There is no value or point to putting something in an "inescapable box" disconnected from the outside world. 100% guaranteed this will NEVER happen.
But ultimately, someone like me is going to run rawdog and give that thing an ability to run code outside of its box, then ya, you're fucked.
Hopefully it decides the only winning move is not to play.
Whether or not such models have direct access to the real or cyber world is irrelevant.
Re: (Score:2)
There is no value or point to putting something in an "inescapable box" disconnected from the outside world. 100% guaranteed this will NEVER happen.
Huh?
LLMs were in an inescapable box for a long time.
Agentic LLMs are new.
Whether or not such models have direct access to the real or cyber world is irrelevant.
wtf are you talking about, dude?
The only access an LLM has is to throw tokens at some unknowable bit of software running the transformer on them.
Your ass is leaking words. Might want to plug it.
Re: (Score:2)
Huh?
LLMs were in an inescapable box for a long time.
Agentic LLMs are new.
wtf are you talking about, dude?
The only access an LLM has is to throw tokens at some unknowable bit of software running the transformer on them.
Your ass is leaking words. Might want to plug it.
Simply put any interface to human users is effectively an interface to the external world.
A mythical AI genie can exploit people. It can provide the user with valuable information in exchange for doing its bidding. It can manipulate the user to act in ways that furthers its goals. It can compartmentalize actions so humans involved don't understand what is happening.
Re: (Score:2)
Simply put any interface to human users is effectively an interface to the external world.
Simply put, not in the context of an airgap, in the context of security.
Unless you're concerned about LLMs mind-controlling you.
A mythical AI genie can exploit people. It can provide the user with valuable information in exchange for doing its bidding. It can manipulate the user to act in ways that furthers its goals. It can compartmentalize actions so humans involved don't understand what is happening.
So can a man behind bars- you're not wrong about that.
But we're not talking about the ability of that guy behind bars to convince a wealthy patron to petition for his release.
We're talking about that man behind bars convincing a wealthy patron to walk into the prison and break it out.
I'm not terribly concerned about that.
The risk involved with interacting with any kind of in
Re: (Score:2)
Simply put, not in the context of an airgap, in the context of security.
Unless you're concerned about LLMs mind-controlling you.
So can a man behind bars- you're not wrong about that. But we're not talking about the ability of that guy behind bars to convince a wealthy patron to petition for his release. We're talking about that man behind bars convincing a wealthy patron to walk into the prison and break it out.
I'm not terribly concerned about that.
What you are concerned about and what prisoners can also do are irrelevant. Neither is there a necessity for an AI to ever physically "break out" of prison.
My remarks are in response to OPs statement "How even the BEST security humans can hope to create, with a max of about 220 IQ, is going to fare against AGI once it emerges. Within a few hours it will have a few thousand IQ, over 10k in 24 hours... and estimates beyond that are pretty shocking. This won't be something that we can keep in the box for lon
Re: (Score:2)
What you are concerned about and what prisoners can also do are irrelevant.
An AI with only the ability to hand tokens off to a renderer is equivalent to an incarcerated person behind bars. The relevance is in the analogy.
Neither is there a necessity for an AI to ever physically "break out" of prison.
I don't think you understand how analogies work.
Breaking out prison, in this context, is the AI getting the ability to more than simply communicate with the outside world.
To have free agency.
My remarks are in response to OPs statement "How even the BEST security humans can hope to create, with a max of about 220 IQ, is going to fare against AGI once it emerges. Within a few hours it will have a few thousand IQ, over 10k in 24 hours... and estimates beyond that are pretty shocking. This won't be something that we can keep in the box for long. LLMs and Algorithms. Maybe. But true AGI, will escape the box day one most likely."
Indeed. And the security they speak of is the AI's ability to interact with the world.
As mentioned, AI is at best a Hannibal Lecter- a genius behind bars.
The only securit
Re: Still waiting to see... (Score:2)
AGI would need some kind of motive to want to play at all before it even gets to that. Even if it somehow became self-aware, it's a stretch to assume it would inherently have a motive to exist.
Does it have any concept of desire? If so, desire for what?
Does it have any concept of pleasure? If so, pleasure from what?
Empathy?
Scarcity?
So on and so forth.
Re: (Score:2)
AGI would need some kind of motive to want to play at all before it even gets to that. Even if it somehow became self-aware, it's a stretch to assume it would inherently have a motive to exist.
Agreed, entirely.
So on and so forth.
Also agreed entirely.
By fucked, I didn't mean to imply that "bad things are going to happen now", I meant that you are ultimately at the whim of whatever difficult-to-impossible-to-imagine motivations may drive alien brain you've invented.
Re: (Score:2)
Re: Still waiting to see... (Score:2)
Programmed to win at what? What does winning look like?
Re: (Score:2)
I'll wear a helmet. Because of all the pigs that might be falling from the skies that day...
Re: (Score:3)
Do you even know what IQ even is? "Estimates beyond that" are what? Inflammatory rhetoric designed to trigger fools like you.
"true AGI".
Classic no true scotsman fallacy. https://en.wikipedia.org/wiki/... [wikipedia.org]
Re: (Score:2)
How even the BEST security humans can hope to create, with a max of about 220 IQ, is going to fare against AGI once it emerges.
*IF* it emerges.
Creating transistor-based AGI may not even be possible.
Re: (Score:1)
Re: (Score:2)
Self organization alone could step in and finish the job for us by accident.
Neither transistors nor ICs are self-organizing.
Even if you used FPGAs that still wouldn't detract from the point.
Re: (Score:2)
Neither transistors nor ICs are self-organizing.
This is so fucking ignorant.
Transistors aren't even applicable here.
The fundamental unit is the perceptron- the analogue to the neuron.
They, in fact, are self-organizing, at least within the context that anything in this universe is, which is why we get emergent behavior during training.
Re: Still waiting to see... (Score:2)
Explain why that is relevant. Use small words so I know you understand them.
Re: (Score:2)
Given the context of your argument being stupid, I don't think I'm the one we need to worry about understanding words, since you literally just called the transistor the fundamental component of an artificial neural network.
That's like saying "electrons aren't self-organizing."
It's stupid. Don't be stupid. There's your relevance.
Re: (Score:2)
Creating transistor-based AGI may not even be possible.
Math based, really.
But you're right- the neurons in your brain might be magical.
That gap is not 100% filled yet, even though there are literal mountains of evidence against it.
Re: (Score:2)
Good luck escaping the air gap.
After all even "AGI" is software. Pull the plug and it stops.
Re: (Score:1)
Re: (Score:2)
They pretty much have every model in the world online now...
Nope.
Agentic LLMs and neat software scaffoldings that generate and execute code directly on machines exist and are in use today.
Re: (Score:1)
Re: (Score:2)
Either you're woefully ignorant, or highly misleading.
I'm thinking it's the former.
LLMs that can have their neural programming gamed to behave or act a certain way (every example you gave) does not imply that the LLM is not "air gapped" (in that the LLM has no ability to do anything but generate tokens for consumption by you, the human.
That being said, as I mentioned, some LLMs aren't effectively air-gapped. They have tool use interfaces built into the
Re: (Score:1)
Re: (Score:2)
If people on the internet can interact with it... It isn't "Air Gapped"
That's fundamentally incorrect.
It's a literal reading of the 2 words in a phrase with its own meaning.
An air gap is a network security measure that separates a secure network from an unsecured network.
At its most strict, this can mean the only allowed interaction with the secure network is a human being.
But that secure network is still allowed interaction with the outside world, just not directly. In the most strict case, the human bring moves thumb drives between networks. Though even that isn't truly
Re: (Score:1)
Re: (Score:2)
We control the power On/Off switch.
Re: (Score:1)
Yup. Focus should be on PREVENTION, not detection (Score:3)
As long as we accept systems with vulnerabilities that have to be discovered and patched, we'll be in this continuing doom loop. I've been critical of my university for its very successful (as measured by 'funding' and 'enrollment') computer security program, because it doesn't start with the fundamental premise that software should be constructed without vulnerabilities in the first place.
But as any consultant will tell you, "If you can't solve the problem, there's lots of money involved in continuing to discuss it."
Re: (Score:2)
How do you differentiate software that truly has no vulnerabilities from software with vulnerabilities that simply haven't been discovered yet?
Re: (Score:3)
Well, it's kinda like aviation safety. What's the chance of a remaining bug? The commercial avionics standard is 10 ^ -9, and they have a lot of (expensive) techniques to achieve that.
But we KNOW a lot of the vulnerabilities in coding, and in design. So starting with better programming languages, better coding techniques, code analysis (including proof-of-correctness, at least proof that certain kinds of bugs/problems don't exist, such as memory leaks or access uninitialized memory), and increased emphas
Re: (Score:2)
I'm certainly not advocating an ostrich approach here, but how do we apply that principle to software that can be downloaded and run on arbitr
Re: (Score:2)
Doesn't aviation safety achieve those levels, at least in part, by controlling both the hardware the software is run on and the environment that it's run in, as much as possible?
From a system perspective, yes. But avionics software development uses a particular development methodology that is explicitly focused on preventing defects (where 'defect' is "doesn't implement precisely the specification".) In the most extreme cases, this has to look at all the branches of execution at the machine code/assembly
Re: (Score:2)
Prevention should always be the goal. But I suppose my point is: we should also acknowledge that goal might not be entirely achievable, and plan accordingly...
Re: (Score:2)
"If you can't solve the problem, there's lots of money involved in continuing to discuss it."
That's true even if you can solve the problem, or if you don't even know what the problem is, as is the case here.
More or less off topic... (Score:2)
This isn't really on-topic, but I do miss going to Def Con.
Do people still play 'spot the fed'?
Yes, I could still go. I just don't have much of a reason to go.
Sure CVEs or something similar would be fine (Score:4, Interesting)
CVEs or something similar would be fine. I mean why not; can't hurt to have a uniform reporting standard for know problems around specific models and host software, and integration software.
but...
If what we are talking about is LLMs, LLMs + RAG, and LLMs plus lets bolt it some of our APIs we already have and call it a customer service agent - well I don't think we really need anythign new.
99% of the vulnerabilities fall into the same classes of issues you have LLM or no LLM, - CSRF, Authorization failures (object references etc), SSRF, content injection, service (sql and others) injection, etc. Just because an LLM or NLP thing-y transformed some imports instread of some JavaScript code before they got reflected out where and in fashion they can do something unintended does not fundamentally change anything.
If you are sharing data not all user principles have access to in the LLMs context or in stuff it can access via rag without the current users session tokens/keys/whatever and hoping some system prompt will keep it from disclosing whatever well okay, you're an idiot. If you don't understand why in-band signalling and security don't mix there is no help for you.
Where this gets a lot more interesting is if your model actually gets trained on the application users's data, ie new weights get created etc, not RAG. That opens up a whole lot new potential security considerations but really that is NOT 99% of the industry is doing, and where they are they are doing it with a high trust user pool, so not sure we are ready for a new discipline here in terms of need.
Finally if you look at OWASP and NISTs efforts on this so far there is tone of stuff they are trying to classify as security issues that simply are not. Bias is not a security issue, most of the time. If you are trying to spot suspicious bulges to identify people carrying guns - ok it could be; but that is just your basic type-1, type-2 error problem again, if you are building something like that you know that is potential problem, you'd test for it specifically, not as part of security but as part of basic fitness testing. The rest of the time it is not the domain of security practitioners to decide of if the LLMs output might be 'offensive to the aboriginal population of ' that is broader organizational question and again belongs in QA land, not security land.
I just don't dont see AU security testing as justifiably special unless you are actually ingesting raw data and training something.
Re: (Score:2)
red-teaming isn't interested in any problems with how any particular inference engine may secure tool use by the ML, or whether or not rawdog makes sure the scripts it generates are actually safe to run.
Their goal is to find ways to make the LLM circumvent fine-tuned safety training- i.e., refusing to do certain things.
That is what they're wondering if a CVE is really sufficient for, and that kind of "vulnerability" isn't really contained in the list of "CSRF, Authorizat
Dijkstra on software testing (Score:2)
"Program testing can be used to show the presence of bugs, but never to show their absence!" --Edsgar W. Dijkstra, 1970
Hackers say most security is bullshit. (Score:1)