AI Hallucinations Lead To a New Cyber Threat: Slopsquatting 51
Researchers have uncovered a new supply chain attack called Slopsquatting, where threat actors exploit hallucinated, non-existent package names generated by AI coding tools like GPT-4 and CodeLlama. These believable yet fake packages, representing almost 20% of the samples tested, can be registered by attackers to distribute malicious code. CSO Online reports: Slopsquatting, as researchers are calling it, is a term first coined by Seth Larson, a security developer-in-residence at Python Software Foundation (PSF), for its resemblance to the typosquatting technique. Instead of relying on a user's mistake, as in typosquats, threat actors rely on an AI model's mistake. A significant number of packages, amounting to 19.7% (205,000 packages), recommended in test samples were found to be fakes. Open-source models -- like DeepSeek and WizardCoder -- hallucinated more frequently, at 21.7% on average, compared to the commercial ones (5.2%) like GPT 4. Researchers found CodeLlama ( hallucinating over a third of the outputs) to be the worst offender, and GPT-4 Turbo ( just 3.59% hallucinations) to be the best performer.
These package hallucinations are particularly dangerous as they were found to be persistent, repetitive, and believable. When researchers reran 500 prompts that had previously produced hallucinated packages, 43% of hallucinations reappeared every time in 10 successive re-runs, with 58% of them appearing in more than one run. The study concluded that this persistence indicates "that the majority of hallucinations are not just random noise, but repeatable artifacts of how the models respond to certain prompts." This increases their value to attackers, it added. Additionally, these hallucinated package names were observed to be "semantically convincing." Thirty-eight percent of them had moderate string similarity to real packages, suggesting a similar naming structure. "Only 13% of hallucinations were simple off-by-one typos," Socket added. The research can found be in a paper on arXiv.org (PDF).
These package hallucinations are particularly dangerous as they were found to be persistent, repetitive, and believable. When researchers reran 500 prompts that had previously produced hallucinated packages, 43% of hallucinations reappeared every time in 10 successive re-runs, with 58% of them appearing in more than one run. The study concluded that this persistence indicates "that the majority of hallucinations are not just random noise, but repeatable artifacts of how the models respond to certain prompts." This increases their value to attackers, it added. Additionally, these hallucinated package names were observed to be "semantically convincing." Thirty-eight percent of them had moderate string similarity to real packages, suggesting a similar naming structure. "Only 13% of hallucinations were simple off-by-one typos," Socket added. The research can found be in a paper on arXiv.org (PDF).
Vet your dependencies. (Score:5, Insightful)
You have to do your research and make sure the packages you are importing are legit. This is true whether or not the package was recommended by an AI.
I guess sloth IS a risk. Vibe coders may get into the habit of just trusting whatever the LLM churns out. Could be a problem. But either way, it's still on you.
Re:Vet your dependencies. (Score:5, Insightful)
It's a great business model! Risk free! How can I invest in it?
Re: (Score:2)
I just a variation of Lucy's Psychiatry Booth in Peanuts. Get people to pay for advice while disclaiming any responsibility for it, anything that does not work out is a personal moral failing in the buyer. Same thing with 'self-driving' cars, etc.
Re: (Score:2)
How can I invest in it?
Use Vibe coding to create your own AI-based service.
Re: (Score:2)
Re: (Score:3)
This isn't about spoofing your module as someone else's- this is about creating one that is entirely novel- commonly hallucinated by LLMs.
The bad actors that create such a thing would love it if you verified cryptographic signatures. Wouldn't want you downloading someone else's malware, after all.
Re: (Score:2)
I wonder if you can even pre-can the hallucination? Maybe by publishing Medium articles with misspelled library names or something, which the AI's eat up and regurgitate later on, perhaps?
import requests ...becomes...
import re-quests
(or something)
Re: (Score:2)
I wouldn't be surprised if that type of LLM poisoning wasn't already either in discussion, or ongoing.
Re: (Score:2)
This technique would probably work.
But bear in mind you wouldn't be impacting LLMs that have already been trained; You'd be possibly impacting any future models that train by fresh scraping the web. It's likely to take months or years before a new website released today makes its way into something like ChatGPT, but still there are new models being made I'm sure that could be affected more quickly.
Re: Vet your dependencies. (Score:2)
Re: (Score:2)
Businesses sure have an incentive to encourage use of dependencies: it saves them a lot of money and gets their product to market sooner. They don't want to pay programmers a fortune to build something that has already been built, especially when they can just use it for free.
There ARE long term consequences of course. Inherited bugs, inherited security vulnerabilities, and you have to wait for someone else to fix it on their schedule. You have to keep updating the packages either way and sometimes that
Re: (Score:2)
I am not a coder. If I want a simple program to read data from a humidity sensor it involves python having to download random packages and dependencies.
Re: (Score:2)
There is an easy solution to that: become a coder :-).
And yes, read the sensor's specification page. it's actually quite fun to make one's own low-level library to communicate with it - and the joy when it works! it's priceless.
Re: (Score:2)
as a non coder myself, i pay for valid code from the sensor manufacturer/supplier and hope the vendor didn't cut corners.
or i can choose to become a coder and roll my own.
security is so inconvenient, like all the time
Re: (Score:2)
"Do your research". On what? The Golang, Rust, NodeJS, JPackage, CPAN, Pypi, or other source repo hosted tools with dependencies scattered everywhere? One may as well rely on the label "Made in America", it's impossible _by design_ to see past the "layer of abstraction" and verify the whole code stack. This is one of the primary goals of "object oriented" software, to conceal anything but the snippet you are tasked to work on.
god damn it (Score:2, Funny)
Re: (Score:2)
I am intrigued by your conspiracy theory and I wish to subscribe to your newsletter - provided it's not protected by a paywall. But maybe you are one of *them*... *puts tinfoil hat on*
Re: (Score:2)
your morons were so preoccupied with whether or not they could code, they didn't stop to think if they should.
Prior Art (Score:5, Informative)
Pfft. Hallucinations. I swear by librwnj (Score:2)
And you can too!
and yet, we're told that AI will... (Score:4, Insightful)
be doing all our coding in the future. "Who needs programmers anymore" seems to be the new mantra in the corporate corner offices.
It was bead enough when incompetent human programmers used unallocated memory or freed memory they were still using, but now we'll get to see the effects of "AI hallucinations"... oh, joy...
What could POSSIBLY go wrong? go wrong? go wrong? go wrong?...
Re: (Score:2)
I'd imagine that the folks who looked after horses in New York City in the late 1800s looked with similar disdain on those limited, buggy, undependable new streetcars. The difference is that these tools are improving far faster than the automobile did. They had several decades to come to terms with it. We don't.
Re: (Score:3)
I'd imagine that the folks who looked after horses in New York City in the late 1800s looked with similar disdain on those limited, buggy, undependable new streetcars. The difference is that these tools are improving far faster than the automobile did. They had several decades to come to terms with it. We don't.
They were pretty much a minority. NYC (and others) were suffocating in horse shit. They were ready for motorized transportation big time. Must have smelled something awful. So they traded horseshit for diesel fumes.
Re: (Score:2)
It wasn't just the excrement either, often the animals would die or have to be put down in the streets.
How do you move 1000 pound animal corpse? Right you don't without help or equipment. So those decaying corpses would lay there on hot cobble stones or brick road in the summer until they could be cleaned up.
Re: (Score:2)
buggy
In that case, the buggy is the older more reliable tech.
Re: (Score:2)
be doing all our coding in the future. "Who needs programmers anymore" seems to be the new mantra in the corporate corner offices.
It was bead enough when incompetent human programmers used unallocated memory or freed memory they were still using, but now we'll get to see the effects of "AI hallucinations"... oh, joy...
What could POSSIBLY go wrong? go wrong? go wrong? go wrong?...
As well, we need to reactivate 1970's nuclear reactors to serve up the AI referencing itself and delivering our malware. It takes serious power to do that!
Sounds like a William Gibson subplot (Score:3)
Re: (Score:2)
Re: (Score:2)
Worse, too.
Rehashing attack vectors (Score:2)
This is a known supply chain attack .. but now they added the label "AI". Someone must get paid per advisory.
Re: (Score:2)
I wonder if you can get paid per stupid comment.
Re: (Score:2)
There is AI in supply chAIn!
Re: (Score:2)
This is a "future supply chain" attack, because they're attacking a Supply chain that doesn't even exist yet.
Your programming AIs thanks to Vibe coding are now using dependencies that don't exist yet to derive your software.
This is bottom feeding (Score:2)
This isn't going to get major corporations who have internal AI. This is going to get the startup who has no real coders. Serves them right, I guess.
Re: (Score:2)
Nope, I'm wrong. Definitely going into internal systems, because everybody is fucking lazy.
Funny Nerd Names (Score:1)
Re: (Score:2)
Well, there are already 629296 packages in Pip [pypi.org], so one needs some imagination to find a name to stand out...
Now as for what do those 629296 packages do, no idea. I usually stick with the Python standard library when I code stuff. But I'm old-fashioned...
Gaps in functionality? (Score:2)
If these libraries are repeatedly hallucinated, do they represent gaps in functionality that should exist that human coders have missed? Do they represent open opportunities?
Re: (Score:2)
Like everything open source you it's probably done a dozen times over already because someone didn't like that the original author started the project on a Tuesday and had to fork it.
Who is installing random packages? (Score:2)
Re: (Score:2)
Hey, this package isn't abandoned. It has new commits this week! Just none last week or any time before that.
Re: (Score:1)
we solved this decades ago (Score:1)
Re: (Score:2)
good news everyone, tomorrow you won't need that highly underpaid undergrad intern to write the code.
No Slopsquatting (Score:2)
Only csoonline calls it slopsquatting.
The scientific paper does not contain the word, but lists it as "typosquatting, combosquatting, brandjacking, and similarity attacks"