Barracuda Urges Replacing, Not Patching, Its Email Security Gateways (krebsonsecurity.com) 90
An anonymous reader quotes a report from KrebsOnSecurity: It's not often that a zero-day vulnerability causes a network security vendor to urge customers to physically remove and decommission an entire line of affected hardware -- as opposed to just applying software updates. But experts say that is exactly what transpired this week with Barracuda Networks, as the company struggled to combat a sprawling malware threat which appears to have undermined its email security appliances in such a fundamental way that they can no longer be safely updated with software fixes.
Campbell, Calif. based Barracuda said it hired incident response firm Mandiant on May 18 after receiving reports about unusual traffic originating from its Email Security Gateway (ESG) devices, which are designed to sit at the edge of an organization's network and scan all incoming and outgoing email for malware. On May 19, Barracuda identified that the malicious traffic was taking advantage of a previously unknown vulnerability in its ESG appliances, and on May 20 the company pushed a patch for the flaw to all affected appliances (CVE-2023-2868).
In its security advisory, Barracuda said the vulnerability existed in the Barracuda software component responsible for screening attachments for malware. More alarmingly, the company said it appears attackers first started exploiting the flaw in October 2022. But on June 6, Barracuda suddenly began urging its ESG customers to wholesale rip out and replace -- not patch -- affected appliances. "Impacted ESG appliances must be immediately replaced regardless of patch version level," the company's advisory warned. "Barracuda's recommendation at this time is full replacement of the impacted ESG." [...] In addition to replacing devices, Barracuda says ESG customers should also rotate any credentials connected to the appliance(s), and check for signs of compromise dating back to at least October 2022 using the network and endpoint indicators the company has released publicly.
Campbell, Calif. based Barracuda said it hired incident response firm Mandiant on May 18 after receiving reports about unusual traffic originating from its Email Security Gateway (ESG) devices, which are designed to sit at the edge of an organization's network and scan all incoming and outgoing email for malware. On May 19, Barracuda identified that the malicious traffic was taking advantage of a previously unknown vulnerability in its ESG appliances, and on May 20 the company pushed a patch for the flaw to all affected appliances (CVE-2023-2868).
In its security advisory, Barracuda said the vulnerability existed in the Barracuda software component responsible for screening attachments for malware. More alarmingly, the company said it appears attackers first started exploiting the flaw in October 2022. But on June 6, Barracuda suddenly began urging its ESG customers to wholesale rip out and replace -- not patch -- affected appliances. "Impacted ESG appliances must be immediately replaced regardless of patch version level," the company's advisory warned. "Barracuda's recommendation at this time is full replacement of the impacted ESG." [...] In addition to replacing devices, Barracuda says ESG customers should also rotate any credentials connected to the appliance(s), and check for signs of compromise dating back to at least October 2022 using the network and endpoint indicators the company has released publicly.
Digital gangrene (Score:3, Interesting)
Digital amputation.
Old techniques become valid again in new battlespaces. History doesn't repeat itself, it just reinfects the current host.
Asking the real questions (Score:3, Interesting)
Who's paying for company failing it's duty of diligence to its customers, requiring tossing otherwise functional hardware?
Re: (Score:2, Insightful)
Nobody. We have allowed commercial software makers to get away without liability for far too long. Hopefully this crap vendor will die, but that is not enough.
Re: Asking the real questions (Score:4, Informative)
Re: Asking the real questions (Score:5, Informative)
Usually, when things go this badly wrong, it is a systematic problem. So no surprise they mess up others of their products as well.
Re: (Score:1)
Are you certain your outbound host hasn't been compromised?
Re: (Score:3)
We used a Barracuda mail filter appliance for years and it worked fairly well. Though for some reason it didn't fully proxy port 25 and would pass SMTP AUTH all the way through to the mail server cluster sitting behind it. It took a long time to figure out why some people's accounts were being locked out seemingly at random. I had to craft a special L7 rule in the intermediary firewall to block SMTP AUTH packets from getting through. There was no setting in the Barracuda to disable it from passing that thro
Re: (Score:2)
That sounds pretty bad. All it requires for something like this to blow up is a vulnerability in the SMTP AUTH handling in the mail server behind. Not what I expect from a security proxy.
Re: (Score:2, Interesting)
Software has zero warranty or liability in its license. This is true of all software, commercial and open source. The only situation where software COULD even have a warranty (but often still doesn't) is where the software has been formally proved correct to the specification. This is because software writing methodologies work on the basis of "good enough"/slapdash techniques (code now, fix later), rather than having any kind of systematic rigour.
Re: Asking the real questions (Score:2)
Re: (Score:2)
This is a hardware/firmware issue however, as they specifically state that flaw cannot be fixed in software and HARDWARE must be thrown out.
Re: (Score:2)
> In a statement, Barracuda said it will be providing the replacement product to impacted customers at no cost
Or you could, you know, read the fine article.
I kid, I kid. There's no bucking the weight of tradition.
Re: (Score:2)
Mea culpa. I often call others out on not doing that, and here's myself not doing my due diligence and shooting from the hip for once.
Thanks for the correction!
Insecurity caused by security devices... (Score:4, Informative)
How utterly and completely pathetic.
We really need to start making sound software engineering practices mandatory in all commercial software engineering or else. This story is basically about security-critical technology so badly made that it starts to rot.
Re:Insecurity caused by security devices... (Score:5, Interesting)
Likely the issue here is that the devices are black box appliances which the customers are not supposed to have low level (ie root) access to, whereas someone who exploited this vulnerability does have root access.
Thus because you don't have root access, you can't assess what was done to the device and try to clean up the mess.
Re: (Score:2)
Absolutely agree, but how to do it?
A NIST-style entity that evaluates software and provides certification for the defect density?
A mandated software development regimen, on the assumption that it guarantees a low defect density?
An obligation to provide a limited warranty on software that guarantees a certain level of quality, with vendors choosing methods to comply?
Re: (Score:2)
Yeah, all of those things.
You could have multiple acceptable process standards. There already are multiple standards for critical software development and none of them are currently mandatory in the field, so almost any of them would be an improvement. In a quick search I came across this slightly relevant document [coreavi.com] which cites DO-178C and ISO 26262 in an attempt to sell "safety certifiable graphics drivers". Surely there must be something similar and extant that could be directly applied (in whole or part)
Re: (Score:3)
Well, If you sell, say, circuit breakers for mains voltage, how are they assured to break correctly and not burst into flames? Or how do they assure an engineer doing the static of a bridge does not mess up the calculations?
So yes, independent tests labs, requirement that only qualified people are allowed to work on security critical software, warranty and liability if it breaks and does more damage. Software is a bit different from physical products, but that does not make the known approaches unusable.
Re: Insecurity caused by security devices... (Score:4, Insightful)
Re: (Score:2)
Could be that the firmware was not protected at all and the malware now sits in the early initialization code and does write-protect itself. In that case you would need to physically flash the firmware or may even have to replace the chip.
Of course if you do this right, the core initialization system and loader for new firmware just get write protected (many modern EEPROMs allows page protection, for example so you can protect part of its contents).
Re: (Score:3)
A few comments:
1. Everything of any complexity, including security software, has security issues.
2. We have no idea if this issue was caused by negligence or not. Not hard to believe that it was, but we don't know.
3. You can introduce liability and it should be in some form (coming to the EU with an updated PLD, might eventually reach the US, it's in the admin's Cyber Security Strategy not that I expect a Republican Congress to do anything about it) but... it would result in a) increased product cost to p
Re: (Score:2, Insightful)
Yes. But having to replace the device? That means more than one severe design issue and these are typically not present in groups. Unless the whole design process was not adequate at all that is.
As to liability, it starts with _them_ having to pay for replacement. As it is, the affected customers will now have to buy a new one and then configure and integrate it themselves. That is just fundamentally wrong on so many levels. Any more liability depends, but this could be a case of gross negligence.
The first paragraph leaves me in awe (Score:4, Interesting)
Compromised beyond repair (Score:4, Informative)
There are a number of scenario's possible where not even an UEFI or BIOS update will save you.
Either they blocked UEFI/BIOS updates or they infected the update process in such a way that it adds their malware again to the new UEFI/BIOS.
Re: (Score:2)
Is it a PC? I mean, I'm sure it's got an Intel or ARM processor in it and and uses some off-the-shelf PC-style chips, but most dedicated enterprise-level networking devices are a custom PCB with a custom boot monitor. I'd be surprised if there's any recognizable UEFI or BIOS in it.
Most network gear is a PC in a fancy box (Score:1)
And they all use boot loaders.
Maybe not a BillGates style BIOS menu but everything from a router to a Nintendo Switch has a bootloader that fetches the initial code from somewhere.
If you can infect that process, you can do some nasty things on any Nintendo Switch.
Re: (Score:2)
Interesting that you mention the Switch because the Tegra X1 chip had a pretty serious vulnerability for a while that could not be fixed via firmware. fusee-gelee is the exploit if wanna read about it.
Looks a lot like a PC in this teardown (Score:2)
https://www.youtube.com/watch?... [youtube.com]
Re: (Score:2)
Re: (Score:2)
Re:The first paragraph leaves me in awe (Score:4, Insightful)
Re: (Score:2)
You should be able to wipe everything if you know *exactly* what hardware went into the device, which is the case for an appliance like this.
Re: (Score:2)
Re: (Score:2)
It's got nothing to do with being a PC - it's specifically a choice of the motherboard manufacturers to not provide a reliable update path. There's absolutely nothing in the PC spec preventing manufacturers from implementing the firmware update path via ROM.
Re: (Score:2)
Re: (Score:2)
Nor much of anything else, sadly.
And until it does, I think it's probably safe to assume that *actual* security isn't even on manufacturer's radars.
Re: (Score:2)
My 486 had one writable chip and I had to move a jumper to flash it. And no backup space, so make sure it's on a UPS!
People want cheap so this is what they get.
Re: (Score:2)
Despite all the "trusted computing" hocus-pocus, PCs have no way of reliably wiping all onboard persistent storage.
That is false at least some of the time. Some mainboards have a firmware reflash that doesn't even require the CPU. These boards can reliably restored to a good state. If they failed to use such, that was irresponsible of them.
Re:The first paragraph leaves me in awe (Score:4, Informative)
And failing that barracuda should accept devices back for exchange as they're a manufacturing failure at this point.
Re: (Score:2)
Consumer rights laws generally don't protect businesses, at least not in the EU.
Barracuda have their customers over a barrel here. They need to immediately replace this hardware, because it has been under active exploit for at least a year.
The customer can either drop in another Barracuda product and be up and running again as quickly as possible, or try to do an unplanned, unscheduled migration to someone else's product.
Re: (Score:1)
The customer can either drop in another Barracuda product and be up and running again as quickly as possible, or try to do an unplanned, unscheduled migration to someone else's product.
If the device is standards-based, it should be trivial to drop in another appliance. And it would be daft to choose the company that you know to be irresponsible and incompetent.
Re: The first paragraph leaves me in awe (Score:2)
Re: (Score:2)
Who the hell is going to take this invitation to upgrade and buy the same crap all over again?
Either Barracuda offer some serious concessions on the replacement hardware, or they're going to lose those customers forever. Surely almost no one is going to say "Hmm.... Barracuda - they seem like a safe bet, let's buy that!" when there are umpteen other choices available. Unless Barracuda are considerably cheaper than the alternatives, you'd need your head examining to replace like-for-like.
Re: (Score:2)
If barracuda do the same thing and offer a cleaned appliance for defective then who can argue? How they do that is another question.
Re: The first paragraph leaves me in awe (Score:2)
The pivot from patch to total replacement of affected devices is fairly stunning and implies the malware the threat actors deployed somehow achieves persistence at a low enough level that even wiping the device wouldnâ(TM)t eradicate attacker access,â Condon wrote.
Re:The first paragraph leaves me in awe (Score:4, Informative)
"We believe most of our customers will just buy a new appliance from us and it's more ka-ching for us."
Nice conspiracy theory, but Barracuda is replacing the devices free.
Re: (Score:2)
Re: (Score:2)
And if you're looking for bug free perfect security you'll be looking for companies forever. Back in reality people spend their money based on vendor response to issues rather than kneejerk about a security bug.
Re: (Score:2)
I never said that the replacement product had to have "bug free perfect security". Any company that didn't fuck up so badly that they needed to perform a full hardware replacement to resolve the issue would be a great starting point with many contenders.
A vendor's response to their issue is incredib
Re: (Score:2)
reimage the HD
Okay stop, please. Users do not have access to the device in the level your required solution needs. Barracuda has already tried resolving the issue and it is persistent. As such they advise rip and replace (with hardware they are providing for free). Stop making assumptions that relate to a fully owned computer. This is not a computer. It's an edge appliance. Users don't even have SSH access much less the ability to image the HD.
Re: (Score:2)
Of course it can be fixed with software. I skimmed the CVE. There's no mention of UEFI or BIOS.
Just because they're not mentioning it, that doesn't mean it's safe. I read the entire CVE and notice page, which only took a couple of minutes. Since May 23rd, Barracuda has been recommending installing a patch to remedy the situation and providing the information needed to identify infection with known malware. Then suddenly on June 6th, they changed their tune to "replace all the hardware regardless of patch level" without explanation. To me this means that they have detected a persistent rootkit that ca
Re: (Score:2)
"I skimmed the CVE. There's no mention of UEFI or BIOS."
Don't assume that the description is complete. Vendors often omit significant details from CVE descriptions.
Re: (Score:2)
Spoken by a true tier 1 help-desk tech.
This is a network appliance not a PC, it in no way functions like a PC. Nothing your spitballing has anything to do with reality.
The takeaway is if your dumb enough to buy Barracuda hardware on the cheap you deserve this.
Re: (Score:2, Interesting)
The motherboards they put in these appliances are so old that they don't even have UEFI.
They're MSI MS-7680 motherboards from 2011 or so, 2GB of RAM and some crap drives. They sell them for thousands of dollars, then they sell you 'energize updates' (virus/spam definitions) and 'instant replacement' (they'll ship you replacement hardware) as a subscription.
Like everyone else, they really want you to switch over to their cloud appliance that is much more profitable to them. This is a cash grab.
And here's the rub... (Score:5, Informative)
I'm speaking in general terms so I'm not identified.
Sadly, the sole method of knowing you have been compromised is through a banner, in red, presented when you login in to the Barracuda web interface.
Barracuda provided signatures to detect the altered binaries- but removed ssh access to their hardware appliances some time back. So if you own a hardware appliance you are essentially phucked trying to determine what is going on.
If you don't see the banner apparently you are not compromised. Yea right. The first thing an attacker will do is disable the banner.
This may be a bigger problem than is currently recognized.
Re: (Score:3)
That is what it sounds like to me-- essentially a complete failure of their system. "Oh, but a new one won't have this problem..."
Re: (Score:2)
Stupid is as stupid does... Oh well!
Re: (Score:2)
Barracuda provided signatures to detect the altered binaries- but removed ssh access to their hardware appliances some time back.
Exactly this, the attacker has full access to *YOUR* hardware, and you don't.
Re: (Score:2)
but removed ssh access to their hardware appliances some time back.
Wow! Amazing!
Wahhh wahhh who will pay me (Score:2)
It really is disgusting that in the modern age when someone discloses a vulnerability this bad, the posters on boards such as this one jump to ask "who will pay" and "what about fitness and merchantability".
How about asking HOW and WHY this came to be and how it can be prevented from the rest of the industry going forward. I realize it's become a weekend, but come on. Focus on the important stuff. "I have to throw out stuff; who will pay me?" comes a long line behind "How was this compromised so badly a
Re:Wahhh wahhh who will pay me (Score:4, Insightful)
The solution is pretty clear and has been implemented numerous times in established engineering disciplines: You sell it, you assure fitness for purpose. It is defective by design, you become liable. Somebody works on it, they are either qualified engineers or technicians or you are grossly negligent.
It is really not hard, but it will take a few more decades if other engineering disciplines are any indicator. At this time, our machines are powerful, flexible, unreliable and unsafe.
Re: (Score:2)
The solution is pretty clear but the people crying their little eyeballs out aren't. Take your dam baracuddas offline and go deal with baracudda. Pretty simple. Don't be bitching up posts on /. or krebs or whatever because shit didn't work like you wanted it to forever. You don't want it to be buggy and they don't want it to be buggy. Zero-day exploits suck but that doesn't change anyone's rights.
Nobody is complaining about the nuissance of replacement; we're all in guffaw about how the vulnerability/threat mitigation strategy escalated from "here's a patch" to "rip and replace."
Plus, anyone who manages network infra doesn't usually have spare days just laying around waiting for this fun little rainy day project to just come along. The last thing the team needs is a fulfillment bottleneck from Barracuda due to Barracuda Networks personnel wanting to upsell every warranty-protected customer with a
Re: (Score:2)
What this really amounts to is Barracuda EOL'ing a product before the advertised/road-mapped EOL date because its defective and they don't WANT to take care of their customers properly.
Nobody thinks Barracuda wanted to release a buggy product or that they don't care about quality but as the GPP states Zero days happen but this is an attempt to shift the cost of addressing them from the manufacturer to the customer, ahead of expectation.
Re: (Score:2)
It is not that simple. Barracuda may, for example, have played fast and loose and just not taken security very serious. They may have done cheaper than posible engineering. They may have left things out that would have been a good idea. In that case they did not _want_ to sell a defective product, but they risked doing so by spending less money that they should have according to sound secure development practices.
Yes, zero days can happen. The question is how likely are they are how mich care was taken to m
Re: (Score:2)
>and they don't WANT to take care of their customers properly
Did you read the article? They are replacing them for FREE.
Re: (Score:2)
(Clearly I'm of the opinion that these rip-and-replace recommendations are driven by either incompetent sales teams, or security engineers unable/unwilling to find a more-reasonable solution, or both. If the underlying issue/vulnerability is so severe, then everybody deserves to know the nitty gritty details of all of it.)
I am unsure. This is extreme and they must know that they will get beaten up over it. It _may_ be extreme greed, it may be that the company is cash-starved. But it may also be that the design is so incredibly stupid (and arrogant) that they actually cannot provide a software solution or it would take weeks or months to make one. Think, e.g., the early initialization code in the firmware already being compromised because they stupidly failed to write-protect those pages. (Many modern EEPROM allow you to do p
Re: (Score:2)
All non-trivial software is constrained by the limitations of what you can guarantee, vis the Halting Problem.
However, you can certainly set a minimum standard by some metric (possibly defect density) and provide a limited guarantee of fitness for purpose.
A true fitness for purpose would require the use of formal methods, which won't work on all problems but should work for a decent subset.
Re: (Score:2)
Nobody expects a formal proof of fitness for purpose. What is expected is that design and manufacturing methods and tests used ensure a high probability of fitness for purpose. You still become liable if it breaks, but if that is unlikely enough, it is not a relevant economic risk to your enterprise.
Incidentally, you do not understand the halting problem. The halting problem only says that you cannot automatically determine all properties of all formal systems of type X using something that is also a formal
Re: (Score:2)
It's really very simple. Hard earned money was handed to company. In return company handed over device with an implied warranty of fitness. Turns out it was not fit for purpose. Naturally customers want EITHER the device be made fit or the money returned.
Barracuda instead wants to turn this into a profit center by selling more devices that we are supposed to believe really are fit THIS time.
Exactly how to keep screw-ups this bad from happening again may not be entirely clear, but it is clear that making suc
Re: Wahhh wahhh who will pay me (Score:2)
If the replacement is free your whole theory goes out the window.
Re: (Score:2)
Uh, Barracuda *is* replacing them for free. From the article:
"In a statement, Barracuda said it will be providing the replacement product to impacted customers at no cost, and that not all ESG appliances were compromised."
You could have read the article before going on that rant.
Re: (Score:2)
That's all fine then. Barracuda is encouraged to not make this error again to avoid the substantial cost involved. For once, the system works!
socketed boot-loader ROMs ? (Score:2)
Fallback to power down, replace socketed boot-loader with "gold" bootloader ROM, and load all firmware images off-line ( no network connection ).
Step 1 of "gold" bootloader requires a complete effective wipe of ALL internal memory: EEROM, RAM, NVRAM, etc.
Step 2 is installation of virgin mass storage devices for any and all NV memory in the system, followed immediately by forced formatting of the virgin devices.
Step 3 of bootloader is reimaging of system, and comparison of the result with stored replica of
Re: (Score:2)
But is this something you can ask of your customers? Just the amount of assistance you need to provide will be enormous. Shipping new appliances is probably more effective.
Re:socketed boot-loader ROMs ? (Score:4, Insightful)
That comes down to the continued degradation of corporate IT.
These are appliances aimed at enterprise customers, such customers *should* have their own IT department staffed by people who know what they're doing. Swapping a socketed component, wiping an EEPROM or formatting some drives should not be difficult for someone who's role supposedly involves managing computers.
But we've got to a level where the average IT department consists of people who open boxes and follow the "quick start guide", then defer to external support if they ever get stuck.
Generally people who do know what they're doing won't be very happy about deploying a black box appliance that they only have superficial access to.
Re: (Score:1)
All too true, sadly.
In early days of my career I had my choice of super geniuses for every role.
By the time I retired the kids coming in could barely do anything IT and sure as hell couldn't code -anything-.
Re: (Score:2)
The outsourcing of core competencies has destroyed that. The big Tier1 provider you engaged at Mega dollars is built on lots of offshore workers who are ok at relatively mundane work but when the really difficult problems crop up fail because the experts have already moved to more lucrative roles.
Also everything is handed over to the Vendors as "their problem" but the gap hits when the Vendor cannot provide that level of expertise which is becoming more common because of my first point.
Translation (Score:2)
Barracuda's recommendation at this time is full replacement of the impacted ESG
CEO needs a new yacht!
Two problems. (Score:2)
1. There's no rigour in software development, outside of formal methods.
In consequence, code is written in a rather slapdash way of code now, fix later. High quality writing is simply not considered. (The Linux kernel has a very low defect density, but it's still far too high for any mission-critical purpose.) We need something that lies between formal methods (which are too strict for many situations, and which can't be used in some cases) and modern coding practices.
2. Completely correct code is only poss
Again... (Score:1)
Once again this is a Windows only product:
https://campus.barracuda.com/product/pstenterprise/doc/41115696/step-1-system-requirements/
I really wish people who submit these articles indicate which OS these products run on.
Re: Again... (Score:2)
No it's a Linux product:
https://www.barracuda.com/prod... [barracuda.com]
Airports (Score:4, Informative)
https://i.pinimg.com/564x/1f/7... [pinimg.com]
Re: (Score:2)
Everyone knows Barracuda is crap (Score:3)
If you're still using their products, given it's widely known for decade(s) that they're unhelpful garbage, you deserve what you get.
If they're not giving replacements for free... (Score:2)