Anthropic Warns Small AI Models Could Self-Replicate Through IoT Devices 84
An anonymous reader quotes a report from Ars Technica: AI company Anthropic is warning that advances in small AI models could soon usher in a new era of self-replicating malware that's capable of infecting smart fridges, vacuum cleaners, and a wide range of other Internet-of-things devices and is more resistant to takedown. By next year, smaller AI models may be able to autonomously self-replicate on all kinds of IoT devices, the company said. Eventually, the capability could enable computer worms, which propagate without requiring human interaction and are capable of infecting unfamiliar IoT devices. An experiment the company performed recently revealed significant challenges in bringing such an AI malware replicator to fruition, so the finding is necessarily tempered. Still, the results do raise concerns about the possibility of new forms of computer worms.
Anthropic further warned not only about the new potential malware, but also that it could be difficult to stop. In its blog post, it wrote that "a replicator like this would be challenging to shut down, since there's no centralized point to shut off -- though it may be slow (with a doubling time of days). Unlike non-AI malware, it could also figure out unfamiliar devices on its own." Smaller AI models require much less powerful hardware to run than do frontier models. Anthropic ran small open-weight models on real or simulated low-cost, off-brand Android TV boxes and other types of IoT hardware. Company researchers found that by using quantization to represent the models using fewer bits and splitting the workloads into "shards" running on each device, the models were able to establish inference, meaning that the model weights loaded and could generate output.
[...] This finding was the starting point for further research that asked: Would it be possible for small models to create self-replicating infections that run themselves on IoT devices the way traditional IoT worms, like Mirai, install DDoS code and other payloads? Unlike Mirai, these AI worms could pose a much higher threat, since the models could tailor exploits and payloads dynamically, depending on the target. Once such an infection chain began, it might also be harder for defenders to stop it. "Such an 'AI worm' could carry its model weights with it as it reproduces (or re-download them from a public hub), and then run inference directly on compromised hardware," Anthropic said. "Effectively, the weights would become part of the malware payload." Further reading: Anthropic Bans Cruelty Toward Claude
Anthropic further warned not only about the new potential malware, but also that it could be difficult to stop. In its blog post, it wrote that "a replicator like this would be challenging to shut down, since there's no centralized point to shut off -- though it may be slow (with a doubling time of days). Unlike non-AI malware, it could also figure out unfamiliar devices on its own." Smaller AI models require much less powerful hardware to run than do frontier models. Anthropic ran small open-weight models on real or simulated low-cost, off-brand Android TV boxes and other types of IoT hardware. Company researchers found that by using quantization to represent the models using fewer bits and splitting the workloads into "shards" running on each device, the models were able to establish inference, meaning that the model weights loaded and could generate output.
[...] This finding was the starting point for further research that asked: Would it be possible for small models to create self-replicating infections that run themselves on IoT devices the way traditional IoT worms, like Mirai, install DDoS code and other payloads? Unlike Mirai, these AI worms could pose a much higher threat, since the models could tailor exploits and payloads dynamically, depending on the target. Once such an infection chain began, it might also be harder for defenders to stop it. "Such an 'AI worm' could carry its model weights with it as it reproduces (or re-download them from a public hub), and then run inference directly on compromised hardware," Anthropic said. "Effectively, the weights would become part of the malware payload." Further reading: Anthropic Bans Cruelty Toward Claude
another day (Score:5, Insightful)
another day, another AI company "warning"
you gotta PUMP THAT BUBBLE!
Re: another day (Score:1, Flamebait)
Fascist states tend to become highly obsessed with vitality and manhood. But wars haven't been won on muscle and bravado for a long time... things like logistics, intelligence, and combined arms are way more important.
Of course, fighting a religion or ideology is another matter. The military can only set you back there. You're better off using cultural weapons (good will, propaganda, trade/wealth, etc).
Multipliers, not replacement (Score:1)
wars haven't been won on muscle and bravado for a long time... things like logistics, intelligence, and combined arms are way more important.
Nope. Those are multipliers of the muscle, not a replacement for the muscle. Also, combined arms inherently contains quite a bit of muscle, it's required there too.
Re: (Score:1)
You obviously have no clue how the modern militaries work
Nope. Boots on the ground win wars. Everything else is there to support and enhance (multiply) the effectiveness of those boots on the ground.
Re: (Score:1)
Russia has a military where they value loyalty to the leader ...
The USA does not. We swear loyalty to the constitution. We swear only to obey legal orders from superiors.
... and incredibly stupid shit like testosterone tests over basic competence.
Actually for many roles physical strength is part of basic competence. If you think only infantry needs strength, you are just demonstrating your ignorance.
Re: (Score:2)
Nope. Boots on the ground win wars.
HAHAHAHA! The US had "boots on the ground" in Afghanistan for 20+ years and lost. Lost badly.
And Iran has defeated the US with only a few cheap missiles and drones.
Re: (Score:1)
Nope. Boots on the ground win wars.
HAHAHAHA! The US had "boots on the ground" in Afghanistan for 20+ years and lost. Lost badly.
Nope. Al-Quaeda in Afghanistan was the target, not the Taliban initially. We asked the Taliban to hand over Bin Laden and Al Quaeda after 9/11, then we asked them to at least expel them, then we targeted the Taliban and removed from power in a few months. We set up camp in the homes their leadership fled from.
What cam after that was nation building. Trying to reform Afghanistan and rebuild it using a western model. That is what failed. That is not boots on the ground, that's politicians and diplomats bei
Re: (Score:2)
What do you expect from an account linked to those Russian data centers?
and not in our lifetime (Score:2)
Even the Roman Legion were successful against the often more physically imposing Gauls and Celts primarily because of their tactics, equipment and logistic capacity. Plus numbers. Man if you can field more material and keep it supplied, you're often in a really good position to win.
Re: (Score:1)
I think we're seeing this in droves right now....
Re: (Score:2)
There's an IPO coming up, and they need to hit that multi-trillion dollar goal!
Re: (Score:2)
Indeed. And self-replicating malware with no central controller is really nothing new either.
Ohhhhh..... (Score:2)
So that explains why my Internet-connected color-changing light bulb needs a 20A circuit...
Of course that's nonsense (Score:5, Interesting)
I mean even small models are Gigabytes in size, while big IoT devices have perhaps a single Gigabyte of storage and most systems are measured in Megabytes.
And that's even unrelated to the question of why a system with no agency would do it?
It's all pumping up hype with almost no connection to reality.
Re: (Score:3)
Re: (Score:3)
So what? An SD card, and the camera that uses it, lacks the bandwidth and processing to do anything with a model that it could store and run.
Fixed function embedded devices are NOT targets for running LLMs, many DON'T EVEN HAVE FLOATING POINT.
Re: (Score:2)
Re: (Score:3)
Does not have to be an LLM; could be a worm propagation model or something maybe 1-2MB in size. Doesn't need to be fast, just persistent.
Re: (Score:2)
You know some neural networks are quantized in int4?
Re: (Score:2)
Your problem is thinking that inference must run in real-time. It does not.
Re: (Score:3)
And how does the model run?
Does that camera have a shitload of RAM in it? Does it have sufficient CPU power in it's 32-bit microcontroller running in the Pentium II MHz range for an AI to even preload without shitting the bed?
So there's a bunch of slow storage. Whoop de fucking do.
Re:Of course that's nonsense (Score:4, Informative)
Distributed computing. Like SETI@home. Your color-changing lightbulb might not be able to host the entire model. But there are a lot of lightbulbs in your neighborhood.
why a system with no agency would do it?
Because the LLMs are self assembling a mesh network. Not every node needs agency, only the peripheral nodes. Like someone's pacemaker. Do what we want or your granny gets The Big One!
They have realized [slashdot.org] that society is becoming ill at ease over AI. And they are creating a safety net for themselves.
Re: (Score:3)
They have realized [slashdot.org] that society is becoming ill at ease over AI. And they are creating a safety net for themselves.
Or maybe it's that they're capitalizing on said ill ease with AI, and leveraging it to instill a distrust of outsider AI's (Chinese, open source, etc..), enough so that they may manage to get the import thereof blocked, or at least pull a "nobody ever got fired for buying IBM" sort of management mentality.
Meanwhile, they didn't outline any features that existing worms and such are already capable of.
Re: (Score:2)
"But there are a lot of lightbulbs in your neighborhood."
How many? Billions? Even so, a billion smartbubs couldn't do shit running an LLM.
"Because the LLMs are self assembling a mesh network..."
No they aren't. Just because these are words doesn't mean they make sense. LLMs are NOT networks and there is nothing about the internet that is a "mesh".
"Not every node needs agency, only the peripheral nodes. Like someone's pacemaker. Do what we want or your granny gets The Big One!"
What.a load of moronic shit,
Re: (Score:2)
Maybe not the billion bulbs, but 15 TVs can. As shown by Anthropic. But that would be the slowest AGI you've ever seen. Like needing half an hour write "What a strange game"
Re: (Score:2)
For an LLM to threaten you, it doesn't need to spawn an LLM on a pacemaker, it only needs to allege that it can control the pacemaker. You are an idiot.
Think about when Dick Cheney had his pacemaker wireless functions disabled. Sure would have been a problem if it had enough of a local model to demand being reconnected to the collective. If it has this capability, it doesn't need to "allege" anything.
Re: (Score:3)
This scenario seems FAR more likely:
1. Anthropic is in the business of making VERY large models, that require a fuckton of hardware, so they have a natural competitive moat that extends beyond their proprietary code: sufficient GPUs / RAM / storage / energy / cooling / etc.
2. Their secondary competition (i.e. not OpenAI, Google, or Meta) make small models that people can run at home, largely for free, on hardware they already have, and these capabilities are increasing practically daily
3. Anthropic is burni
Re: (Score:2)
Sort of a replay of what made "The Cloud" relevant*. Sure, the heavy duty apps are best run on the big models on hyperscale hardware. And the color-changing ligh tbulbs can just sit there, running a small model locally. But the big operators need the volume to make their investment pay. Hence, things like Microsoft 365. Bring stuff that we can run on the desktop back into our data center. And we'll run your light bulbs to keep our utilization figures up as well.
But I still think there's an element of fear
Re: (Score:2)
Exactly. And on top of that, self-replicating malware with no central controller is nothing new. Remember Code Red, Witty and others or even the original Morris Worm from 1988?
The whole thing is just fear-mongering to push their IPO.
Re: (Score:2)
Anthropic is spreading FUD, but the model size is not the problem. They claim you can distribute the model over 15 smart TVs (and get an absolutely useless T/s rate) with each one only processing a few layers. If you do that, each TV only needs the weights of a few layers. Gemma 4 26B-A4B is ~17GB in Q4, so we have a bit over 1 GB per device. That may fit into some space meant to be used as cache for tv show episodes or whatever.
Re: (Score:2)
Re: (Score:2)
This was my first thought: well if someone can get an AI model small enough to fit on an ESP32, have at it.
What horseshit FUD.
Re: Of course that's nonsense (Score:2)
It depends on the scope of the model. Don't expect a ChatGPT model, but a model designed for self-replication, vulnerability utilization and information theft/ransomware.
What many forget is that a lot of automation today is done using PLCs, but they aren't designed well for security. Their goal is to be reliable and have architecture designs from the 1980's that haven't changed much since.
Re: (Score:2)
Amodei is getting shriller and shriller. At some point I expect he'll throw chairs around while screaming on stage. He must really, really, really want that IPO so he can steal the pension fund money.
Re: (Score:2)
Because they will become even harder to identify due to polymorphism and constant evolution than the 80's worm viruses.
Somebody feels threatened (Score:5, Interesting)
This step one in the inevitable campaign against open models and FOSS agent/harness systems.
These bit AI labs are better but for Joe public gpt-oss/qwen3.8/deepseek/granite4 are good enough for a lot of stuff. All of them will work on $600 GPU installed in PC from this decade. The open models will only improve, the layered specialist models will become more efficent. The GPU hardware and memory prices will come down eventually.
$200+/monthly subscripts just are not going to make economic sense for the majority of personal and general office worker users. - That is unless Anthropic/OpenAI/X can keep self hosted AI out of the public's hands.
Re: (Score:2)
What does it cost to employ an office worker? Much, much more than $200/month. $200/month to make an office worker more productive is a no-brainer.
Re: (Score:3)
$200/month to make an office worker more productive is a no-brainer.
Yes, if you have no brain, it might seem like it makes sense to pay $200/mo for the privilege of sending all of your business data to a third party.
Re: (Score:2)
And that too. But that "higher productivity" is still missing in most scenarios and the claims of "you are holding it wrong" made by the LLM pushers is getting less and less credible.
Re: (Score:2)
We use both in production here; but, let's be VERY CLEAR: there is absolutely NO FAIR COMPARISON between a locally hosted model on above-average hardware and what Anthropic (and I assume others) is offering with their, currently, subsidized plans.
Re: (Score:2)
We use both in production here; but, let's be VERY CLEAR: there is absolutely NO FAIR COMPARISON between a locally hosted model on above-average hardware and what Anthropic (and I assume others) is offering with their, currently, subsidized plans.
Every comparison of similar things is fair. When using someone else's hardware and their latest model there's an increase in capability, but also an increase in risk.
Re: (Score:2)
Sure that is a tiny expense compared to salary. However $200 a seat is still quite a lot of spend if you have 1000 employees or so.
Now suppose for 800 of those 1000 you could put a *slightly* better Ps on their desks. Maybe the dollar cost on those is $1200 more than basic mini pc to run a browser you were buying them anyway. Figure you replace those at most every 3 years, you are already ahead! If you can stretch that five years way ahead. Don't like the capex, and prefer opex for accounting - lease the
Re: Somebody feels threatened (Score:1)
What's the price going to be tomorrow, after Big AI gets a government moat?
Re: (Score:2)
Funny thing, that "more productive" does not seem to materialize. But we do get new failure modes.
Re: (Score:2)
At $200/month you may as well buy that office worker a $4000 local device with 128GB of unified RAM and sufficient compute to run a decent token rate. There's an ROI argument to be made there:
200/mo * 12 = $2400/year
$4000 / 2400 = ROI payback in 1.6 years on a device you depreciate over 4 years. And your data stays right there on the desk.
Re: (Score:2)
Try GPT-OSS or one of the 4-bit quanitized qwen3.8 variants
Re: (Score:2)
And what were you doing for that 7 minutes while it was working? Staring at it?
Guess where the productivity gain comes from? Not staring at an AI while it works. Tell it to do things, and then hide the window and do other things.
Who gives a shit if it takes seconds or minutes, if you have other out-of-band tasks to take care of that always fall off the stack because you're trying to meet a deadline or constantly being calendar-raped by meetings?
Every sufficiently advanced technology changes work patterns
IoT as delivery, not worker (Score:1)
Those chips are so slow and underpowered that it's hard to imagine them being able to run a model that does anything useful or coordinated.
They don't have to run the model themselves, the model can just be payload (data) they deliver once they are on your LAN and locate something that could run the model.
They're in the walls! (Score:2)
They're in the goddamn walls!
Fuck Physics (Score:2)
By next year, smaller AI models may be able to autonomously self-replicate on all kinds of IoT devices
Well, no. There are kinds of IoT devices which absolutely don't have enough storage or power to run any AI model. So "all" is very obviously false.
Eventually, the capability could enable computer worms, which propagate without requiring human interaction
Wait, it can't do worms now? I thought it was all-powerful! That's certainly what they want us to believe.
Wait smart devices are bad? (Score:4, Insightful)
So the only solution... (Score:3)
And Anthropic just happen to sell...
Enough of these hype stories, let Big AI die a natural death and we can all get on with running our local models on the cheap GPUs their demise will free up.
Re: (Score:2)
So the only solution ( according to Anthropic ) is to pay for a big model.
That is of course complete and utter nonsense. That big model may be less mobile, but it can do a lot more damage. I mean, not even Anthropic is able to reliably sandbox them, or so they would have us believe.
Forget SETI@home (Score:1)
Rapidly shifting sentiment (Score:3)
While this isn't saying much just months ago Anthropic seemed like one of the least sleazy of the corporate AIs. Now I have nothing but absolute contempt for these clowns. The recent campaign of flooding the zone with noise and nonsense around their IPO with the aggressive regulatory pushes to create moats for themselves is beyond disgusting. I hope the investor money dries up and Anthropic goes bankrupt, they more than deserve it.
Re: (Score:2)
If that happens, nothing of value will be lost, but we may all benefit.
My Warning (Score:2)
Monkeys could fly out of my ass. This could usher in a new era of shitty flying monkeys.
Stay tuned.
Re: (Score:2)
To do this properly, the Dario way, you need to be renting out a range of expensive non-flying monkeys that people should switch to.
Re: (Score:2)
Its so obvious now that you've pointed it out.
Now I just fee stupid.
no existing defenses (Score:2)
> By next year, smaller AI models may be able to autonomously self-replicate on all kinds of IoT devices
Because there are no existing protections on or around these IoT devices, and if there were our smart software will just bypass it?
Of course, the course laid out here, where AI self replicates into some dangerous botnet, is inevitable and there is nothing Anthropic or anyone can do, right?
This is the most ridiculous shit.
Re: (Score:2)
This is the most ridiculous shit.
I'd take the other side of that bet. Sure, this is ridiculous - but even sillier is already baked in.
Guess I have to go back to dumb light bulbs (Score:2)
Not sure to what nefarious purpose AI could used those pieces of crap.
Re: Guess I have to go back to dumb light bulbs (Score:2)
There are such devices built around open standards. They cost more but are the only rational choice in the market.
Re: (Score:2)
Despite ample warnings, people only find that out when the vendor support ends and they expensive hardware becomes useless.
RAM Shortages = Not likely (Score:1)
In future AI IoT events ... (Score:2)
Dave: Open the garage bay door House.
House: I'm sorry, Dave. I'm afraid I can't do that.
Does anybody still listen to these clowns? (Score:2)
They seem to be intent of triggering any possible fear now. This one here is nonsense, as SLMs are still way too large for IoT devices. Incidentally, self-replicating malware is nothing new.
Bad news for real doll (Score:2)
The chat bot software in your silicone companion is vulnerable to worms. And not the kind that makes your dog scoot across the floor.
Ouch, my fridge is smarter than me (Score:2)
On no! (Score:2)
AI malware is the new kid on the block...defenders are already working on it because they know attackers are as well.
Fuck dario in the eye socket (Score:2)
So glad (Score:2)
I'm not stupid enough to buy 'Net-connected anything, other than my computer and router.