CrowdStrike Overhauls Testing and Rollout Procedures To Avoid System Crashes (securityweek.com) 36
wiredmikey writes: CrowdStrike says it has revamped several testing, validation, and update rollout processes to prevent a repeat of the embarrassing July outage that caused widespread disruption on Windows systems around the world.
In testimony before the House Subcommittee on Cybersecurity, CrowdStrike vice president Adam Meyers outlined a new set of protocols that include carefully controlled rollouts of software updates, better validation of code inputs, and new testing procedures to cover a broader array of problematic scenarios.
In testimony before the House Subcommittee on Cybersecurity, CrowdStrike vice president Adam Meyers outlined a new set of protocols that include carefully controlled rollouts of software updates, better validation of code inputs, and new testing procedures to cover a broader array of problematic scenarios.
bets it'll happen again anyway? (Score:2)
the old, bittervet in me has seen failures like this too many times to trust....
Re: (Score:2)
You do not even have to be bitter or a vet to not give them another chance. Their screw-up as far too extreme. Fool me once ...
The only IT company that can survive mistakes this bad (and has made mistakes on the same level just recently) is Microsoft, because too many people have painted themselves into a corner there.
Suggested headline (Score:3)
Headline: CrowdStrike Overhauls Testing and Rollout Procedures to Avoid System Crashes
Suggested clarification headline: CrowdStrike Overhauls Testing and Rollout Procedures to Stay in Business
Re: (Score:2)
That nicely clarifies it. Hopefully they will still go bankrupt in a way clearly attributable to their screw-up.
Re: (Score:2)
It just won't happen the same way again. Maybe.
Re: bets it'll happen again anyway? (Score:2)
Question (Score:2)
How many senior managers have admitted responsibility and resigned?
I'll take a wild guess and say "none."
Re: (Score:2)
I hope they all get fired, I'm still waiting for my not-worth-the-trouble UberEats card!
Re: (Score:2)
And hence the problem will _not_ get fixed (it is not fixed now, there is far too much broken with their processes and leadership). The real problem they have is organizational dysfunction stemming from massively prioritizing of profits over engineering. I mean, even one smart intern would have done a better job than they did.
The only way to fix organizational dysfunction is to kill the organization. I hope the market will do that. Anybody with some actual IT skills will either be moving away from them or a
Re: (Score:2)
And hence the problem will _not_ get fixed (it is not fixed now, there is far too much broken with their processes and leadership). The real problem they have is organizational dysfunction stemming from massively prioritizing of profits over engineering. I mean, even one smart intern would have done a better job than they did.
The only way to fix organizational dysfunction is to kill the organization. I hope the market will do that. Anybody with some actual IT skills will either be moving away from them or already has done so.
Here's the problem with killing the organization that has proven itself to be dysfunctional. If there's a history of any kind of profit for the company? Those C-Suites and managers will be sucked up by competitors, and their ineptitude *WILL* be utilized to do the same thing to many, MANY other IT related companies. There needs to be more than just a dissolution of the company. There needs to be real responsibility. These folks, even if they did get shit-canned, would be snapped up by others because of the
Re: (Score:2)
Indeed. Established engineering has liability, including management liability. We need that or this crap will continue and get worse.
Re: (Score:2)
Hey, come on now... they fired Joe in the development group and Fred over in SQA. I mean, what more could you ask for? /s
Very wise (Score:3)
tweaks to provide customers with additional contro (Score:3)
just tweaks?
What about stuff like
rollout groups?
per system install time windows?
Re: (Score:3)
Their customers can't win.
If they install updates immediately, they risk being bricked. If they delay or do a slow roll out, they risk being hacked.
Remember that the goal is not to avoid being hacked, it is to avoid liability. They have business continuity insurance, they just need to make sure it doesn't get invalidated by not ticking the right boxes when it comes to "best practice" security.
Sure... (Score:3)
This would have been an excellent decision if they had made it about a decade ago. Instead this is merely damage control and PR.
Overhaul all you want (Score:2, Troll)
Overhaul all you want, the cat's out of the bag now. Just a matter of time before this threat vector becomes used more and more. Windows needs an enema!
Re: Overhaul all you want (Score:2)
IT needs an enema, to flush out the Microsoft.
Re: (Score:2)
More like a hardcore chemotherapy with a lot of hard radiation on top. The cancer sits deep.
Re: (Score:2)
Troll? Lighten up people, it's just Windows and Microsoft I'm bashing on. It's not like I'm saucing apples here.
In related news (Score:3)
The Department of Education is still dealing with the online FAFSA fuckup from last year, and anyone old enough to be reading this probably remembers the ObamaCare marketplace rollout from a decade ago.
Seems that software is still hard. Moreso when you think you can powerpoint your way to success.
Re: (Score:3)
But really, while software is still hard, the list of bloody beginner's mistakes and gross violations of the state-of-the-art they made is extreme. That has nothing to do with software being hard and everything with massively prioritizing profits over engineering.
Re: (Score:2)
Seems that software is still hard.
Things are even harder when you're actively incompetent - like Crowdstrike. This isn't the first time their updates have caused problems, it's just the first time they fucked that many people at once that they got dragged in front of the senate.
Easier solution (Score:3)
I found the easier solution to make sure that Crowdstrike can't fuck my entire network and company, is to just not run Crowdstrike.
Re: (Score:2)
I found the easier solution to make sure that Crowdstrike can't fuck my entire network and company, is to just not run Crowdstrike.
+1 Insightful, if I had Mod points
Re: (Score:2)
At this time, that is the only sane solution.
Re: (Score:2)
That's good, but what's the alternative. And before you rattle off a list of companies, make sure you have vetted their internal processes to ensure they won't cause a similar kind of incident. And before you say "no companies I don't need them" turn to your webcam and say "you're welcome" to the person who actually controls your network.
how can I trust them? (Score:3)
Re: (Score:2)
They're going to fuck it up, in more ways than one.
Re: (Score:2)
You cannot. Anybody that is not moving away from Crowdstrike is about as incompetent as they are and will get hit again. You cannot fix a release process that is this extremely broken. You have to start over with a different set of people and, in particular, different management.
Pay extra to get your update last (Score:2)
So the finally follow the state-of-the-art? (Score:2)
I will believe it when I see it. The mistakes they made are on another level of incompetent and stupid.
And there was much rejoicing (Score:2)