Some Def Con Attendees Forgive Crowdstrike - and Some Blame Microsoft Windows (techcrunch.com) 93
Fortune reports that Crowdstrike "is enjoying a moment of strange cultural cachet at the annual Black Hat security conference, as throngs of visitors flock to its booth to snap selfies and load up on branded company shirts and other swag." (Some attendees "collectively shrugged at the idea that Crowdstrike could be blamed for a problem with a routine update that could happen to any of the security companies deeply intertwined with Microsoft Windows.")
Others pointed out that Microsoft should take their fair share of the blame for the outage, which many say was caused by the design of Windows in its core architecture that leads to malware, spyware and driver instability. "Microsoft should not be giving any third party that level of access," said Eric O'Neill, a cybersecurity expert, attorney and former FBI operative. "Microsoft will complain, well, it's just the way that the technology works, or licensing works, but that's bullshit, because this same problem didn't affect Linux or Mac. And Crowdstrike caught it super-early."
Their article notes that Crowdstrike is one of this year's top sponsors of the conference. Despite its recent missteps, Crowdstrike had one of the biggest booths, notes TechCrunch, and "As soon as the doors opened, dozens of attendees started lining up." They were not all there to ask tough questions, but to pick up T-shirts and action figures made by the company to represent some of the nation-state and cybercriminal grups it tracks, such as Scattered Spider, an extortion racket allegedly behind last year's MGM Resorts and Okta cyberattacks; and Aquatic Panda, a China-linked espionage group.
"We're here to give you free stuff," a CrowdStrike employee told people gathered around a big screen where employees would later give demos. A conference attendee looked visibly surprised. "I just thought it would be dead, honestly. I thought it would be slower over there. But obviously, people are still fans, right?"
For CrowdStrike at Black Hat, there was an element of business as usual, despite its global IT outage that caused widespread disruption and delays for days — and even weeks for some customers. The conference came at the same time as CrowdStrike released its root cause analysis that explained what happened the day of the outage. In short, CrowdStrike conceded that it messed up but said it's taken steps to prevent the same incident happening again. And some cybersecurity professionals attending Black Hat appeared ready to give the company a second chance....
TechCrunch spoke to more than a dozen conference attendees who visited the CrowdStrike booth. More than half of attendees we spoke with expressed a positive view of the company following the outage. "Does it lower my opinion of their ability to be a leading-edge security company? I don't think so," said a U.S. government employee, who said he uses CrowdStrike every day.
Although TechCrunch does note that one engineer told his parent company they might consider Crowdstrike competitor Sophos...
Their article notes that Crowdstrike is one of this year's top sponsors of the conference. Despite its recent missteps, Crowdstrike had one of the biggest booths, notes TechCrunch, and "As soon as the doors opened, dozens of attendees started lining up." They were not all there to ask tough questions, but to pick up T-shirts and action figures made by the company to represent some of the nation-state and cybercriminal grups it tracks, such as Scattered Spider, an extortion racket allegedly behind last year's MGM Resorts and Okta cyberattacks; and Aquatic Panda, a China-linked espionage group.
"We're here to give you free stuff," a CrowdStrike employee told people gathered around a big screen where employees would later give demos. A conference attendee looked visibly surprised. "I just thought it would be dead, honestly. I thought it would be slower over there. But obviously, people are still fans, right?"
For CrowdStrike at Black Hat, there was an element of business as usual, despite its global IT outage that caused widespread disruption and delays for days — and even weeks for some customers. The conference came at the same time as CrowdStrike released its root cause analysis that explained what happened the day of the outage. In short, CrowdStrike conceded that it messed up but said it's taken steps to prevent the same incident happening again. And some cybersecurity professionals attending Black Hat appeared ready to give the company a second chance....
TechCrunch spoke to more than a dozen conference attendees who visited the CrowdStrike booth. More than half of attendees we spoke with expressed a positive view of the company following the outage. "Does it lower my opinion of their ability to be a leading-edge security company? I don't think so," said a U.S. government employee, who said he uses CrowdStrike every day.
Although TechCrunch does note that one engineer told his parent company they might consider Crowdstrike competitor Sophos...
Blackhat, not DEFCON (Score:5, Insightful)
Re: (Score:2, Insightful)
99% of people at BlackHat are there because a corporation paid them to be there. In most cases, it's a "cyber" security corporation. Of course people running the same kind of operation as Crowdstrike feel sorry for Crowdstrike.
Paid sponsors? Most “cyber” security companies are in business because of Microsoft’s tradition with “security”.
The irony could be easily lost if not for being so bitch-slap worthy.
Circle the Venture Capital Wagons (Score:2)
This is nothing more than circling the venture capital wagons to protect VCs.
If one recent VC now public company high flying stock gets bankrupted by one event, the other VC firms and stocks will be questioned.
Re: (Score:2)
I blame both (Score:2, Insightful)
Microsoft has been incompetent and supplying insecure software for almost half a century.
Crowdstrike is incompetent too and carelessly fucked up critical bootup files.
But here's the clincher: both Microsoft and Crowdsource need to exist and both need to be incompetent for the misery the Crowdstrike software unleashed to happen at all. One can't exist without the other or without the other's incompetence: Crowdstrike's very raison d'etre and selling point is to compensate for Microsoft's incompetence. They o
Re: (Score:2)
Re: (Score:2)
Technically true, Linux could be replaced by a BSD.
Sure, eventually. There's no BSD that's not way behind Linux.
Re: I blame both (Score:2)
Re: (Score:2)
Tell me your completely ignorant
My completely ignorant what?
of Linux, *BSD and general UNIX history without saying it directly.
I had more experience with different kinds of Unix as a teenager than you probably have now.
The number of things Linux has or does that *BSD did first would blow your tiny little mind.
As per usual for your arguments, that is irrelevant. We're talking about now, not years ago. Try to keep up with the calendar, sport.
Re: I blame both (Score:1)
You put up the claim, now show the evidence. What feature do you think is in Linux that the BSDs donâ(TM)t have.
Re: (Score:2)
You put up the claim, now show the evidence. What feature do you think is in Linux that the BSDs donâ(TM)t have.
SystemD? (runs for the hills)
Re: (Score:2)
If I had mod points today, I'd mod parent up, but I can't decide if I'd mod it "funny" or "insightful"....
Re: (Score:1)
You mean a declarative init system? Like launchd
Re: I blame both (Score:1)
Did you forget how to Google?
Re: I blame both (Score:2)
but what about all my favorite open source apps like steam, mono/moonlight, nvidia drivers, wine, mathematica, matlab, google chrome, etc.?
there is not even a âoewindows subsystem for bsdâ! meanwhile, linux is up to two, AND itâ(TM)s the backend for android!
linux is far ahead of any bsd on these important pieces of the open ecosystem of the bazaar.
Re: I blame both (Score:2)
Re: (Score:2)
Certainly avoids a lot of work and subtle implementation difficulties.
Re: I blame both (Score:2)
Re: I blame both (Score:1)
Wine and Mono both have BSD ports.
Re: (Score:2)
itâ(TM)s the backend for android
Equivalently, Android is a skin for Linux.
Re: (Score:3, Informative)
The same could be said when crowdstrike bricked linux boxes
Crowdstrike did not brick anything. The definition of bricking is rendering a device unresponsive to the point that it takes heroic measures to resurrect it, such as reprogramming the firmware. The fix for Crowdstrike's massive failure was relatively simple and did not require any such heroics. They did, however, render the Windows systems unable to boot.
Crowdstrike did not do the same to Linux systems. The Linux systems suffering from a similar failure from Crowdstrike were experiencing kernel panics afte
Re: (Score:2, Interesting)
Indeed. The two are not comparable. No idea why some people keep pushing that lie.
It is quite clear that the Linux issue did not make Linux boxes unbootable at all and that a reboot plus regular (if fast) remote management was entirely enough to fix the issue on Linux.
Re: (Score:1)
Indeed. The two are not comparable. No idea why some people keep pushing that lie.
I assume (generously, I think) that it could be because they want to say that this isn't Microsoft's fault because the same could have happened on Linux, and they got confused. But at best they are letting us know that they have no clue what actually occurred. Right from the first discussion about this here on Slashdot where the failures on Linux were reported as if they were relevant, I asked questions about whether it was actually the same or not. The discussion made it clear that it wasn't, so I proceede
Re: (Score:2)
Windows advocates (as opposed to customers, who may not have another realistic choice) don't work on logic, they work on emotion.
Apparently, they do. Of course, failure does not get much worse than this when technological characteristics are under discussion.
As to the currant AMD problem: Do not worry about it. Unless somebody does a successful root-compromise on your system first, they cannot even do the attack. And when they have done that root compromise, it is probably not worthwhile at all to invest the extra effort. Yes, it is embarrassing. Yes, it shows that AMD needs to fix some things as well. But practical impact from a ris
Re: (Score:1)
My concern is not that the attack is super likely, it's that it's super permanent. My processor is old enough that AMD has announced that there will not be a fix. But this should not be a warranty issue as this is a design failure. And they are ABLE to fix it for these older processors, but they are choosing not to. It's seriously embarrassing. This is some Intel-level bullshit. Now there are zero vendors worthy of trust.
mod this comment too (Score:2)
I have bought exclusively AMD processors for decades.
I don't like it when my vendor is incompetent at security.
It doesn't matter which vendor it is.
Re: I blame both (Score:2)
Re: I blame both (Score:1)
Re:I blame both (Score:4, Interesting)
I completely agree. Well said.
Incidentally, when (not if) we get real product liability (as has happened in _all_ other engineering disciplines eventually), neither Microsoft nor Cloudstrike will exist much longer
Re: (Score:2)
I completely agree. Well said.
Incidentally, when (not if) we get real product liability (as has happened in _all_ other engineering disciplines eventually), neither Microsoft nor Cloudstrike will exist much longer
I agree, liability is foundational. But, in 1986 the government gave vaccine manufacturers quite a bit of immunity, because they were getting sued. (That is not medical advice, that's just what happened.) I would guess that the big players in the IT industry only need to argue that writing bug free code is impossible -- and that includes that it's impossible to devise enough tests to find every bug -- therefore, there will always be damages -- but if you allow millions of people and thousands of companies t
Re: I blame both (Score:2)
Re: (Score:2)
Crowdstrike failed to actually test their patch, which tells me they have no QA. As a former QA worker, I've seen serious f*ups, but none that actually reached customers. Microsoft OTOH, has released buggy software for years, but nothing OS crippling that I know of.
Re: I blame both (Score:1)
Re: I blame both (Score:4, Insightful)
I have two cousins that used to do QA for them in Seattle, and while they slashed QA in the US, they hired in other countries like India, China and Romania where labor was 1/3 to 1/4 the cost (no idea what it is now, that was like 2005). I only kept a job in the US for a long time because we were US government contractors, but even that job went to developers because out of 10 people, who do you cut first? I've seen that product deteriorate and wrote about 50 bugs because I now support it - number fixed? 3. They recommend we upgrade to the latest and see if the bugs still exist (I have zero control over that).
Re: I blame both (Score:1)
This CrowdStrike bug specifically could not have happened on Linux because of guard pages and similar protections. The problem enabling it in Windows is there are many things that expect to be able to go out of bounds and not get killed off.
Conflict between security and usability (Score:5, Insightful)
This so fits the stereotype of what most people think of cybersecurity: "security == inconvenience"
The fact that supposed security experts would think so lightly of a piece of security software bringing down the systems it was supposed to protect shows how abyssal the state of the industry is. Availability was supposed to be one of the aim of security, and that seemed to be lost somewhere in all the buzzwords.
Re:Conflict between security and usability (Score:5, Interesting)
Re: (Score:3)
Here is another dark industry fact that shows how messed up it is: Most "security experts" cannot code, have no system administration experience, but also do not have real risk management expertise.
Re: Conflict between security and usability (Score:2)
Re: (Score:2)
Sad, but true.
Re:Conflict between security and usability (Score:4, Informative)
The fact that supposed security experts would think so lightly of a piece of security software bringing down the systems it was supposed to protect shows how abyssal the state of the industry is. Availability was supposed to be one of the aim of security
bluescreens force system to shut down to prevent running with corrupted state, to protect from further damage.
Re: (Score:2)
Availability was supposed to be one of the aim of security
bluescreens force system to shut down to prevent running with corrupted state, to protect from further damage.
You're both right. The BSOD is better than running insecure. But you know what's better than a BSOD? Security software that checks its own input to make sure that it makes sense before accessing memory locations based on that input. This isn't security 101 because they didn't even get that far, this is programming 101. You have to check your input just to protect from data corruption, let alone malice or hey, what about simple incompetence?
Crowdstrike isn't even competent at programming, what makes anyone i
Re: (Score:1)
The problem as I understand it is that a good security system allegedly needs Turing-Complete scripts that can be added during run-time. But because they run at the root level, they need to run fast. Validation of memory and values allegedly slows them down too much.
I'm not sure if they can use compiled DLL-like things that are checked vi
Re: (Score:2)
The fact that supposed security experts would think so lightly of a piece of security software bringing down the systems it was supposed to protect shows how abyssal the state of the industry is. Availability was supposed to be one of the aim of security
bluescreens force system to shut down to prevent running with corrupted state, to protect from further damage.
Forgivable if the system was under attack and CrowdStrike shutdown the machine as a defensive measure.
Taking down the system with a routine maintenance is more like a body checkup that knocked the patient into coma.
Re: (Score:2)
We need Yellow Screen of Second Chance ;-)
Re: (Score:3)
This so fits the stereotype of what most people think of cybersecurity: "security == inconvenience"
Agreed. Too many people fail to recognise that security is a spectrum with "Secure As Possible" at one end, and "Convenient As Possible" at the other end. The challenge is to strike a sane balance between the two, based on your particular threat model. At, or near, either extreme is usually not a good place to be.
Re: (Score:2)
Not quite, the threat model is only part of it. What you actually need to do is competent IT risk management and that is much more than just a threat model.
Anyways, having fundamentally incompetent coders (as Crowdstrike was clearly using) implement new features in system components that can make a system unbootable and then fail to adequate test and review these changes, is clearly a balance that will work only for "toy" systems.
Re: (Score:2)
Not quite, the threat model is only part of it. What you actually need to do is competent IT risk management and that is much more than just a threat model.
True. I was using "threat model" as a shorthand for "well thought out security policy and management". The model is just the foundation; you have to implement practical solutions on top of it.
Re: Conflict between security and usability (Score:2)
Re: (Score:3)
The "Cybersecurity" team at my company has always been in the "as secure as possible, and we decide how you do things. and nothing you can say will change our minds. we have one policy for everyone, and we won't change it for anyone".
this also affects developer machines. We were recently notified that PowerShell access will be removed entirely, and activated in a case-by-case basis only and only if cybersecurity considers the justification for it good enough (they won't. they don't want to deal with anythin
Re: (Score:2)
The "Cybersecurity" team at my company has always been in the "as secure as possible, and we decide how you do things. and nothing you can say will change our minds. we have one policy for everyone, and we won't change it for anyone".
this also affects developer machines. We were recently notified that PowerShell access will be removed entirely...
I get the impression that there's something about cyber security which makes space for this kind of mindless rule making. We have a list of controls and our job is to make everyone follow the controls, like. It's like you have to wear a space suit and a deep diving suit and steel toe capped boots and a safety helmet and strap a defibrillator to yourself and have hazmat suit and a biological warfare suit, all at the same time.
I suspect the reason is that children can learn to understand rules in a simple way
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
The fact that supposed security experts would think so lightly of a piece of security software bringing down the systems it was supposed to protect shows how abyssal the state of the industry is. Availability was supposed to be one of the aim of security, and that seemed to be lost somewhere in all the buzzwords.
I completely agree. This was an abysmal failure, which Microsoft set up (due to low security and bad boot processes on Windows) and Crowdstrike triggered. Nothing of this is the slightest bit acceptable and should open up both enterprises for shared unlimited liability. I feel deeply insulted that people that apparently think not much has happened are calling themselves "security experts". They clearly are anything but.
Re: (Score:2)
While both Crowdstrike and Microsoft played a part, it was Crowdstrike that toppled the first domino. They need to own this. You can fuck up any OS if you have low level access and Anti-virus systems need low level access to stop boot viruses. The real blame should go to whoever released the patch without doing enough testing and just as importantly, whoever decided to roll it out on a Friday and all at once. And just because Linux didn't suffer a boot loop doesn't mean Linux is rock solid. I know, I u
Re: (Score:2)
In the case of Availability; part of that has just been folded into generic "IT"/"Operations", with just being real disciplined and careful about backups on the low end in terms of technical hotshot status; and SREs doing clever things with monitoring and orchestration on the medium/high end; but if you want to
Re: (Score:2)
This so fits the stereotype of what most people think of cybersecurity: "security == inconvenience"
The fact that supposed security experts would think so lightly of a piece of security software bringing down the systems it was supposed to protect shows how abyssal the state of the industry is. Availability was supposed to be one of the aim of security, and that seemed to be lost somewhere in all the buzzwords.
There's a third factor:
Complexity is the enemy of both security and convenience.
To borrow the slogan [wikipedia.org]
The network is the [thousands of other people's] computer [who all have their own incentives which don't necessarily align with your own].
If a system is so critical that it must have deeply invasive security software which is getting live updates, because someone might click a bad link, then why is that system being used for anything critical? Why aren't all these "critical" systems being isolated? Why is eve
RCA? No thanks. (Score:2)
Forgiving Crowdstrike? Really? (Score:3, Interesting)
CrowdStrike didn't have process in which before they released any updates to production, they would just release them to a actual canary environment to make sure it didn't catch all the machines on fire. Which you need to do when you have kernel code.
Not to mention that the driver seems to be poorly written with a ton of sanity checks missing.
Finally, Microsoft did propose an API to allow security vendors what they needed in user space. The EU killed it as anti-competitive because MS wouldn't also have to use it, but continue to use kernel drivers, completely ignoring that user-mode code is way easier to write, making it easier for new and smaller business to compete and nothing MS did would prevent you from using kernel drivers if you wanted to.
Re:Forgiving Crowdstrike? Really? (Score:5, Insightful)
Not to mention that the driver seems to be poorly written with a ton of sanity checks missing.
To me, this is the real root of the who-to-blame map. You can put a little bit on Microsoft for just being bad at everything, but ultimately Crowdstrike didn't check their input and that makes them fundamentally incompetent. They have no business being allowed to manage anything.
Microsoft did propose an API to allow security vendors what they needed in user space. The EU killed it as anti-competitive because MS wouldn't also have to use it
A more accurate way to write that is that Microsoft didn't want to use it. If Microsoft made that the only way to do those things and then went on to use that mechanism themselves so that their competing products didn't have an anticompetitive advantage, they could have done so. But that's not what they wanted to do. Now here we are, at peak stupid — incompetent, insecure OS gets taken down expensively by incompetent, inadequate security management software.
Re: (Score:2)
Now here we are, at peak stupid — incompetent, insecure OS gets taken down expensively by incompetent, inadequate security management software.
Not quite complete. The missing part is that incompetent operators first select the grossly insecure OS, then see massive damage done by the incompetently written security software ... and learn apparently absolutely nothing from that disaster.
The usual damage level where government engineering regulation stepped in hard in the past is a few hundred people dead. Since the customers are clearly too stupid to act proactively, I guess we will see that one happen here as well. What a shame.
Re: Forgiving Crowdstrike? Really? (Score:1)
Re: (Score:2)
Well, then MS could simply have given a respective _credible_ assurance, one with real penalties and regular independent verification they pay for (which would have been peanuts). They chose not to and their history spoke not in their favor.
This is 100% on MS. The EU is not to blame at all. Unless you think having actually working anti-trust law is a problem?
Re: (Score:3)
Finally, Microsoft did propose an API to allow security vendors what they needed in user space. The EU killed it as anti-competitive because MS wouldn't also have to use it, but continue to use kernel drivers, completely ignoring that user-mode code is way easier to write, making it easier for new and smaller business to compete and nothing MS did would prevent you from using kernel drivers if you wanted to.
That is a direct lie. The EU "killed" nothing. The only requirement was equal access for 3rd party tools and MS tools. MS chose to not limit their own tools to that API and hence decided to open everything. You do not seem to understand how anti-trust law works and what it purpose is.
Re: (Score:1)
Microsoft shouldn't be giving that level of access (Score:2)
Re: Microsoft shouldn't be giving that level of ac (Score:1)
Re: (Score:2)
The EU never required it. All MS would have had to do was make it credible that their own security software would not get special access. They can still do that.
The process here is that the EU finds illegal vendor behavior. Then the vendor is expected to make a proposal, which gets reviewed by EU experts and the competitors that have been wronged. The only acceptable proposal MS made was kernel-level access for the competitors and that one got accepted. They could have proposed other things. Simply promisin
Re: (Score:2)
Re: (Score:2)
That is really just complete bullshit. You need to stop hallucinating or you will never understand how things actually work.
out of jail card (Score:2)
Crowdstrike is paying left and right in the hope they will minimize consequences of their mess. The press should not help them with this.
On the other hand, as a declared Microsoft hater, I would not condemn Microsoft for the level of access they allowed, level of access is granted by the users, OS is expected to do what the user says. And users (sysadmins, management) were guilty for putting trust in Microsoft and Crowdstrike for anything mission critical.
Re: (Score:3)
Microsoft is still guilty of setting the whole mess up and not only by the interface design, but by the general bad state of Windows security and a flawed boot process.
Yes, MS users are stupid. But human engineering history amply shows that customers generally cannot judge product risks and quality competently. Hence things only ever got better with product liability and/or regulation. Well, I guess we will need a few 100 people dead next time. Maybe then things will change. In other engineering fields, tha
Re: (Score:2)
Microsoft does a lot to CREATE DEMAND for security products. They never thought much about security or gave it priority except when it is support or damage control and as always, PR.
We might not have the CPU mess we do right now with remote management and trusted boot etc. if MS wasn't pushing things around in that space. Now we've got Pottering messing with Linux who works for MS...
its paid dudes from big corp Security Teams (Score:1)
I see this another way (Score:2)
Fortune reports that Crowdstrike "is enjoying a moment of strange cultural cachet at the annual Black Hat security conference, as throngs of visitors flock to its booth to snap selfies and load up on branded company shirts and other swag"
If I were there, I would too. It's potentially a "do you remember when" moment, as in "do you remember when Crowdstrike destroyed Delta Airlines" or for that matter, "do you remember when there used to be a company called Crowdstrike".
(Some attendees "collectively shrugged at the idea that Crowdstrike could be blamed for a problem with a routine update that could happen to any of the security companies deeply intertwined with Microsoft Windows.")
Yes, some attendees know it could happen to them, because their software also sucks and also runs on Windows. They don't want to talk too much just in case.
Who gets most of the blame.... (Score:1)
If not crowdstrike, if not microsoft, the users then shoulder most of the blame? I guess some inventive tech person could have commited to testing the environment on their own time then publish the results...? I'm being sarcastic btw ./
defcon is just security larp (Score:1)
collecting stickers and hoping not to get outed as they are all feds.
What's the time? (Score:1)
Say, what's the time?
Just get me some
Big Mac fries to go.
Ground floor, coming up!
Yeah, but no (Score:1)
Well, no. Crowdstrike actually managed to fuck this up on Linux earlier in the year, too. This guy has no clue and hasn't been reading the news.
Blaming Microsoft for this is as much of a stretch as blaming Intel or AMD for the fact that their CPUs executed the buggy code CrowdStrike shipped.
Not damaging the opinion these people have of Cr
Re: (Score:2)
BS (Score:3)
Blame Crutch or Broken Leg (Score:2)
What's needed is a Manhattan Project (Score:2)
The WinTEL model obviously showing its age and despite masses of sticking-plaster such as CrowdStrike, can never measure up to the task. Since a lot of peoples jobs depend on not knowing this - nothing is going to change.
Bullshit (Score:2)
No, THAT claim is bullshit, because exactly this kind of problem has happened before both on Linux and Mac, both precisely because the exactly same level of access was given to the software that had the flaw.
"Most Def Con Attendees Are Complete Morons" (Score:2)
There. Fixed the title for you.
"And Crowdstrike caught it super-early." (Score:2)
>> "And Crowdstrike caught it super-early."
Crowdstrike caught what super-early? Definitely not the bug. :-D
I remember first reading Crowdstrike's explanation on The Register. That Reg article was needed because Crowdstrike's explanation was behind a paywall -- some sort of 'authenticated-corporate-account-only' page. So they were certainly not ready to mitigate the chaos they caused the world -- millions of mandays were lost, perhaps tens of millions.