AI Hallucinated a Dependency. So a Cybersecurity Researcher Built It as Proof-of-Concept Malware (theregister.com) 44
"Several big businesses have published source code that incorporates a software package previously hallucinated by generative AI," the Register reported Thursday
"Not only that but someone, having spotted this reoccurring hallucination, had turned that made-up dependency into a real one, which was subsequently downloaded and installed thousands of times by developers as a result of the AI's bad advice, we've learned." If the package was laced with actual malware, rather than being a benign test, the results could have been disastrous.
According to Bar Lanyado, security researcher at Lasso Security, one of the businesses fooled by AI into incorporating the package is Alibaba, which at the time of writing still includes a pip command to download the Python package huggingface-cli in its GraphTranslator installation instructions. There is a legit huggingface-cli, installed using pip install -U "huggingface_hub[cli]". But the huggingface-cli distributed via the Python Package Index (PyPI) and required by Alibaba's GraphTranslator — installed using pip install huggingface-cli — is fake, imagined by AI and turned real by Lanyado as an experiment.
He created huggingface-cli in December after seeing it repeatedly hallucinated by generative AI; by February this year, Alibaba was referring to it in GraphTranslator's README instructions rather than the real Hugging Face CLI tool... huggingface-cli received more than 15,000 authentic downloads in the three months it has been available... "In addition, we conducted a search on GitHub to determine whether this package was utilized within other companies' repositories," Lanyado said in the write-up for his experiment. "Our findings revealed that several large companies either use or recommend this package in their repositories...."
Lanyado also said that there was a Hugging Face-owned project that incorporated the fake huggingface-cli, but that was removed after he alerted the biz.
"With GPT-4, 24.2 percent of question responses produced hallucinated packages, of which 19.6 percent were repetitive, according to Lanyado..."
"Not only that but someone, having spotted this reoccurring hallucination, had turned that made-up dependency into a real one, which was subsequently downloaded and installed thousands of times by developers as a result of the AI's bad advice, we've learned." If the package was laced with actual malware, rather than being a benign test, the results could have been disastrous.
According to Bar Lanyado, security researcher at Lasso Security, one of the businesses fooled by AI into incorporating the package is Alibaba, which at the time of writing still includes a pip command to download the Python package huggingface-cli in its GraphTranslator installation instructions. There is a legit huggingface-cli, installed using pip install -U "huggingface_hub[cli]". But the huggingface-cli distributed via the Python Package Index (PyPI) and required by Alibaba's GraphTranslator — installed using pip install huggingface-cli — is fake, imagined by AI and turned real by Lanyado as an experiment.
He created huggingface-cli in December after seeing it repeatedly hallucinated by generative AI; by February this year, Alibaba was referring to it in GraphTranslator's README instructions rather than the real Hugging Face CLI tool... huggingface-cli received more than 15,000 authentic downloads in the three months it has been available... "In addition, we conducted a search on GitHub to determine whether this package was utilized within other companies' repositories," Lanyado said in the write-up for his experiment. "Our findings revealed that several large companies either use or recommend this package in their repositories...."
Lanyado also said that there was a Hugging Face-owned project that incorporated the fake huggingface-cli, but that was removed after he alerted the biz.
"With GPT-4, 24.2 percent of question responses produced hallucinated packages, of which 19.6 percent were repetitive, according to Lanyado..."
Cost/Benefit (Score:5, Funny)
Not only that but someone, having spotted this reoccurring hallucination, had turned that made-up dependency into a real one, which was subsequently downloaded and installed thousands of times by developers as a result of the AI's bad advice, we've learned.
If you're married, then you know that sometimes this kind of thing can be worth it just to avoid starting another argument.
Re: (Score:3, Insightful)
If you're married to the wrong person, then you know that sometimes this kind of thing can be worth it just to avoid starting another argument.
Fixed that for you.
Re: (Score:3)
Re: (Score:1)
You can't tell shit, my friend :)
Our marriage isn't work because we match each other very well, we both know when to compromise, and we deeply respect each other to name a few of the many aspects that make our relationship work perfectly.
We both know we're the lucky few in a world full of wrong matches.
Re: Cost/Benefit (Score:2)
Re: (Score:2)
That would be... very difficult, since we're together 100% of the time.
Re: (Score:2)
That would be... very difficult, since we're together 100% of the time.
Dude, marrying your conjoined twin is just weird.
Re: (Score:2)
Re: (Score:2)
No kids, but plenty of pets.
Maybe I should point out this is my second marriage (I learned a LOT from the first one).
We are together 100% of the time. Working from home is a blessing.
And yes, plenty of small compromises to go around, but they are all openly discussed and agreed; I'd say the count of compromises is 50/50.
I'm aware this type of relationship is very rare, but I've seen it at my maternal grandparents before. 53 years of blissful marriage, through pretty rough times in part, until one of them sa
Predicting the future (Score:2)
Re:The Register is a JOKE news site (Score:5, Informative)
Funny thing, this exact same scenario was reported last year [scmagazine.com].
In a June 6 blog post, Vulcan Cyber researchers explained a new malicious package spreading technique they call “AI package hallucination.” The technique stems from ChatGPT and other generative AI platforms sometimes answering user queries with hallucinated sources, links, blogs and statistics.
Large-language models (LLMs) such as ChatGPT can generate these “hallucinations,” which are URLs, references, and even entire code libraries and functions that do not actually exist. The researchers said ChatGPT will even generate questionable fixes to CVEs, and — in this specific case — offer links to coding libraries that don’t exist, either.
If ChatGPT creates fake code libraries (packages), the Vulcan Cyber researchers said attackers can use these hallucinations to spread malicious packages without using familiar techniques such as typosquatting or masquerading.
Sounds pretty factual.
Hans Kristian Graebener = StoneToss
Re: (Score:2)
Plausible though. If they don't bother code reviewing scripts from humans I doubt they're going to start code reviewing scripts from AI. The framework of taking someone else's code as is from a third party site that isn't under control and shoving it out to unsuspecting customers is not a good basis for reliable software.
Re: (Score:2)
"There is a legit huggingface-cli, installed using pip install -U "huggingface_hub[cli]"."
I don't consider that hallucination, if a thing actually exists just under a different name. Of course I don't trust El reg to report accurately, so it's not clear whether hub is actually part of the name or just the name of the parent project. And did the hub project do what the non-hub AI result thought it would.
I have these questions, but I don't care enough to look for more details. It would be nice if I didn't hav
AI is great... (Score:3)
...for research and fun
It should NOT be used for serious work
Re: (Score:3)
Indeed. The only time you can use it for serious work is if the task the AI is asked to do is significantly below you own level of expertise and you fully verify the results. of course, that probably will cost you more time than doing it yourself from the start.
Re: (Score:2)
Not if you have AI do it for you.
Re: (Score:3)
Indeed. The only time you can use it for serious work is if the task the AI is asked to do is significantly below you own level of expertise and you fully verify the results. of course, that probably will cost you more time than doing it yourself from the start.
Not so, depending on the task.
The task probably should be at or below your level of expertise, sure. But I'm using it frequently as a time saver. And it works.
"Given the following php code, add pagination functionality."
Boom, done. Could I have done it? Sure. In 4 seconds? Nope.
Re: (Score:2)
Can "AI" have added a nice security vulnerability? Sure.
Re: (Score:3)
Re: (Score:2)
A chainsaw is a great tool. Some fool chopping his leg off with one doesn't change that.
Re: (Score:2)
But it's practically never the right tool to chop vegetables...
Re: (Score:2)
Re: (Score:2)
Now find me a video of an elephant using a chainsaw...:-)
Re: (Score:2)
But companies are expecting to replace all their programmers with AI.... What could possibly go wrong?
Naive trust in LLMs + supply chains... (Score:3)
Doom Loop (Score:4, Interesting)
AI is entering a doom loop where it hallucinates then incorporates the hallucination into subsequent versions.
Eventually AI will be all hallucinations.
Re: (Score:1)
Re: (Score:2)
When you feed it a giant cesspool of invalidated data (The internet) you should not expect a single response to be accurate. none of these AI's are fed a carefully curated data set.
Needs a catchy name (Score:5, Interesting)
Re:Needs a catchy name (Score:5, Funny)
If the hallucination involves object oriented code, it'd be class hysteria.
Re: (Score:1)
I had a similar thought (Score:5, Interesting)
I asked an LLM (doesn't really matter which one, they are fail in this same way) for advice on how to load a specific file type, it gave me three possible packages to use to be able to land the file...
But not one of them actually existed. When pressed further on one of the frameworks that didn't exist, it doubled-down and gave me a website for the package - which did not exist.
And that led me to think, maybe I should build out that package. Not in order to create malware as described here, but because you already know some people in a similar situation will be directed right to your package of that specific name without having to do anything!
So it's pre-made marketing just waiting for a product.
Re:I had a similar thought (Score:5, Funny)
So now AI thinks up the ideas and writes the specs, and real people do all the work to make this crap work.
That's it... AI has now graduated to being the new pointed-haired boss.
Not exactly a boss.... (Score:1)
So now AI thinks up the ideas and writes the specs, and real people do all the work to make this crap work.
It's more like AI opens a door that you can take advantage of people going through.
Sort of more like a force of nature than a boss.
Or if you like, it is recognizing a shortcoming by pretending something does not exist when plainly it would be useful, and someone opts to fill that hole for the benefit of mankind. Although in truth that scenario feels a bit like AI is a boss. :-)
what about a fact-checker AI? (Score:4, Interesting)
predictable (Score:2)
Ironically, that this kind of stuff would happen with a big probabilistic prodiction engine was to be expected.
Let the AI write code, they said.
It's really good at it, they said.
Look how fast it generates code when I put in a natural language prompt!
What could possibly go wrong?
Programmers behind enemy lines (Score:3)