Former Equifax CEO Blames Breach On One Individual Who Failed To Deploy Patch (techcrunch.com) 255
Equifax's recently departed CEO is blaming the largest data breach in history on a single person who failed to deploy a patch. TechCrunch reports: Hackers exposed the Social Security numbers, drivers licenses and other sensitive info of 143 million Americans earlier this summer by exploiting a vulnerability in Apache's Struts software, according to testimony heard today from former CEO Richard Smith. However, a patch for that vulnerability had been available for months before the breach occurred. Now several top Equifax execs are being taken to task for failing to protect the information of millions of U.S. citizens. In a live stream before the Digital Commerce and Consumer Protection subcommittee of the House Energy and Commerce committee, Smith testified the Struts vulnerability had been discussed when it was first announced by CERT on March 8th.
Smith said when he started with Equifax 12 years ago there was no one in cybersecurity. The company has poured a quarter of a billion dollars into cybersecurity in the last three years and today boasts a 225 person team. However, Smith had an interesting explainer for how this easy fix slipped by 225 people's notice -- one person didn't do their job. "The human error was that the individual who's responsible for communicating in the organization to apply the patch, did not," Smith, who did not name this individual, told the committee.
Smith said when he started with Equifax 12 years ago there was no one in cybersecurity. The company has poured a quarter of a billion dollars into cybersecurity in the last three years and today boasts a 225 person team. However, Smith had an interesting explainer for how this easy fix slipped by 225 people's notice -- one person didn't do their job. "The human error was that the individual who's responsible for communicating in the organization to apply the patch, did not," Smith, who did not name this individual, told the committee.
Ob (Score:5, Funny)
He's Spartacus!
Human Error??? (Score:5, Insightful)
Re: (Score:2)
What carefully parsed weasel words.
So the patch had passed testing, but wasn't applied? The only alternative is that someone has to instruct them specifically to start testing every patch in their ecosystem.
Shouldn't someone be seeing a report of all unapplied patches and how old they are? Yell at the testing group if they age too much?
Re: (Score:2)
The sentence is a mess, I don't know how to read it.
You don't need to read a blog to know the current stable version# of Apache. Know that patching is a recurring issue with all software.
It sounds like they're trying to 'baffle with bullshit' IMHO. Throw someone under the bus then pull him back at the last second, act like heros.
Re:Human Error??? (Score:5, Insightful)
Absolutely. Human redundancy is just as important as network/system redundancy. If the organization isn't set up to continue working even if someone gets hit by a bus, that's a management failure. It's not a single individual. Who was responsible for checking that the work was done as required?
Re: (Score:3, Funny)
Even Mormons are deployed with full redundancy.
Re: (Score:2)
Re: (Score:3)
Exactly... You should have defense in depth, not only to counter someone who fails to apply a patch but also to try and mitigate against attacks against vulnerabilities for which there is no patch.
Re: (Score:2)
Re: (Score:2)
Are you implying it was an "inhuman problem"?
It's even worse (Score:5, Insightful)
Any number of reasonable things could have caused the patch to be missed, but you'd expect $250M spent over three years to provide a few more security processes beyond, "Fred forgot to apply the patch." The attackers were spreading through their systems over several months without detection.
Also, way to lead from behind. Every corporate officer I've met has shared one tenet with all others: they are responsible for everything that their team does, good and bad. If some employee several rungs down the corporate ladder fails, it's because the leadership above them failed to hire or train them correctly or put in the right processes.
Re: (Score:2)
Absolutely.
Not to mention the comment about them just now getting serious on cybersecurity in the last 3 years...your very company exists on critical/sensitive information handling, security should have been #1 priority from the git go. He's making it sound like they are just a small company trying to do what it can.
Sounds like a senior dev ops nija (that was probably the IT golden boy) just got thrown under a bus...
Re: Human Error??? (Score:5, Insightful)
Re: (Score:3)
At the retirement party for one of my best managers I asked him one question: "What's your secret to being a great manager?" his answer: "My only job is to enable my people to do theirs."
Sure, one person may not have applied the patch. But it wasn't that persons fault that a process wasn't followed, that the appropriate funding was available, that assorted checks and balances were in place, and that IV&V didn't happen. That falls on management.
This is the same scapegoating VW tried to get away with in t
Re: (Score:2)
the largest data breach in history
And then two stories down there's:
Yahoo Triples Estimate of Breached Accounts To 3 Billion
So now 140M > 3B? In fact there are a number of breaches larger than the Equifax one.
Re: (Score:2)
It appears that a recipe belongs to grandma. What belongs to the recipe?
Re: Human Error??? (Score:2)
recipe's, the
The beard under the period and the shift key for cap "T". God, keep up man!
But on a more serious note, I had to "correct" the iPhone 3 times to type that quote.
Re: (Score:2)
"Improve processes to prevent revealing our fuck ups to the media, government and customers."
They spent $250 mil and had 225 people in 'security theater,' but only ONE GUY actually doing security, as you always need a reliable scapegoat to justify spending more money on looking secure than being secure.
At least that seems to be what they are saying, as none of the rest of the 'security theater' team is being blamed. He must be the Argentinian that authorize
Nice to have Cyber Security Team (Score:5, Interesting)
Re:Nice to have Cyber Security Team (Score:5, Insightful)
What do the other 224 do?
Re:Nice to have Cyber Security Team (Score:5, Funny)
What do the other 224 do?
Apparently, not watch that one guy...
Re: (Score:3, Funny)
Meetings to plan the schedule of that one guy.
Entire Team/Policy FAILURE (Score:2)
This wasn't a single person failure. If it was, that means the policy that setup that single person to be able to fail a mission critical issue is at fault. Also at fault is the actual PROTECTION OF DATA! How in 2016 and 2017 does ANY COMPANY have UNENCRYPTED PERSONAL INFORMATION on ANY COMPUTER/DATABASE which is attached to the INTERNET?!??! And this is in a company that is touting that it has spent billions on cybersecurity. Sure you may have spent the money on cybersecurity, but you certainly didn't take
Re: (Score:2)
Indeed. This was probably one person, completely overworked with no help and no verification of his work by others. That is a process and leadership failure.
I smell bullshit. (Score:5, Insightful)
If .25Bn has been invested then there's sure as hell no process that could have allowed a single critical patch go unchecked as described. There's teams, or should be teams of people watching these things.
I smell a really shitty cop-out excuse.
Re:I smell bullshit. (Score:5, Insightful)
You missed the best part, 3 years ago, they didn't even have a security department. At least according to his throw the wage slave under the bus testimony. He's distracting you with this tale of rouge employee while dropping a bombshell you didn't even notice.
3 years ago the company responsible for approving credit for all americans had NO information security department. According to the CEO's testimony they had zero budget and not a single employee dedicated to security of their IT networks. That's grounds for jailing him IMO.
Re:I smell bullshit. (Score:5, Interesting)
I caught that part but was much more incensed by the lame attempt to parry liability.
Re: (Score:2)
The other 224 were responsible for making sure the coffee maker got cleaned once a month, along with, as mentioned by an earlier AC...writing his schedule.
*Equifax employs approximately 9900 employe
Re: (Score:3)
3 years ago the company responsible for approving credit for all Americans ...
Technically, Equifax and the other credit bureaus don't approve credit to anyone, they simply provide a centralized source for credit information. Individual lenders make approval decisions based on this information - which is available to, and can be challenged by, the borrower.
Re: (Score:2)
Good question. I suspect they have a defense that they don't tell anybody anything about you at all, they merely report what others tell them about you.
I'm not saying that this is a good defense.
Re: (Score:2)
I agree. Full liability with his personal fortune and significant prison time. How the hell can you run a company this size and with data this critical without a competent IT security division? Negligence does not get more gross than this.
Re: (Score:2)
Okay, as much as I hate to appear like I'm defending Equifax in any way... You simply can't really make such a statement of fact like that from the information given so far.
Smith said when he started with Equifax 12 years ago there was no one in cybersecurity. The company has poured a quarter of a billion dollars into cybersecurity in the last three years and today boasts a 225 person team.
All we can infer for certain is that sometime between 12 years ago and 3 years ago a dedicated cybersecurity team was formed, and what the last three years combined budget was.
That doesn't really mean there were no security-focused employees before that, of course, as the job was probably rolled into the general IT budget and operational
Re: (Score:2)
Yeah, but that's before someone invented this "cyber" bullshit. Back 12 years ago, it would have been "network security" or "computer security".
Re:I smell bullshit. (Score:5, Informative)
It's either utter incompetence or bullshit.
At the enterprise level and especially for PCI compliance there should be 3 independent levels where this could have been caught: 1) applying the patch, 2) monitoring patch compliance, 3) vulnerability scanning. Organizations that really care about security also have a Web Application Firewall (WAF) or other Intrusion Detection System (IDS) or Intrusion Prevention System (IPS) which would have been a fourth level that could have prevented this attack.
Blaming this attack on one person when there should have at least been 3 levels of prevention with at least 3 different teams involved is stupid.
1) Patch Management Solution: In the enterprise, this should be a software solution (like Quest KACE or IBM BigFix type solutions) that monitor the patches on each endpoint and apply patches on a schedule after they are tested. Most organizations have a 30 day patch cycle although critical remote vulnerabilities like this should have been escalated sooner.
What would have been reasonably possible is for the person responsible for escalating the patch to apply sooner than 30 days could have missed escalating it. However, the normal 30 day cycle then should have caught it.
a) Patch application
b) Patch monitoring
In some organizations there is one team that applies the patches (and is usually involved in testing the upcoming patches) another team that monitors the patch levels. In other organizations they are the same team although there should still be independent checks for application and monitoring.
2) Vulnerability Scanning: Especially anything that is visible to the Internet should get vulnerability scanned at least every 30 days. A decent remote vulnerability scanning software should have picked this up. Tenable's Nessus which is one of the industry standard vulnerability scanners tests for CVE-2017-5638 which is the vulnerability that effected Equifax. Nessus started testing for it on March 14th.
3) Web Application Firewall: Web Application Firewalls will block known attacks before they hit the application. A decent WAF should block known vulnerabilities such as the one that hit Equifax as long as it was up to date. That said a lot of companies I have worked with tend to run WAFs in intrusion detection mode instead of intrusion prevention mode due to false positives and not wanting to block legitimate traffic. Some companies I have worked with are much better than others at going through the alarms, how quickly they respond to alarms, and filtering out the false positives so that the alarms are easier to manage. Usually for Web applications you will have a WAF rather than a general purpose IDS/IPS as the WAF will have access to the unencrypted traffic although there are ways to have IDS/IPS products have access to the Web server private certificates to decrypt the traffic.
Re:I smell bullshit. (Score:5, Insightful)
After reading this it occurs to me that it's much more likely someone sold the info rather than had it hacked.
Re: (Score:3)
The
Re: (Score:2)
Re: (Score:2)
If .25Bn has been invested then there's sure as hell no process that could have allowed a single critical patch go unchecked as described.
Spending lots of money does not guarantee a good process. Heck it could be evidence that the process was poor from the beginning.
I smell a really shitty cop-out excuse.
I don't think so. I have no doubt in my mind that Equifax are truly incompetent on every level including the ability to come up with a process that is resistant to such human error.
This isn't a "cop-out excuse". This is evidence of severe missmanagement and their shareprice deserves to be slaughtered as a result.
the tree must be watered (Score:3)
To quote Thomas Jefferson, "The Tree of Bare Fucking Minimum Standards of Responsibility and Decorum must be refreshed from time to time with blood."
Re: (Score:2)
"paraphrase", not "quote".
But good one nevertheless.
Horseshoe nail (Score:2)
Buggy whips are gone, but the need for horsehoe nails remains.
Re: (Score:3)
Ah yes, the blame game (Score:5, Insightful)
"It was his fault. That's why I sold my company stock when I found out about the breach rather than inform anyone except the other folks in the executive suite."
Re: (Score:2)
"i'm getting a killer golden parachute because i'm worth that much. Really guys, they wouldn't give me this much money to retire if I wasn't. Ergo, totes not my fault, and now it's not my problem either"
Re: (Score:2)
well, you see, CEOs are paid so much money for the singular and unique value they offer to a company faced with challenges only few have ever surmounted. it is necessary to pay a large salary because the rewards he can bring are so large that there is a lot of competition. and apparently even more money if he fucks it up, because hey he deserves it.
that last part is sorta weird, but as long as you don't ever think it even remotely applies to you, you'll be fine, pleb.
Re: (Score:2)
I guess it was a winning "Gamble"?
Wow, that's scummy (Score:5, Insightful)
"The human error was that the individual who's responsible for communicating in the organization to apply the patch, did not,"
What a scummy thing to say, and he doesn't even realize that the statement makes Equifax look even worse.
With a couple of hundred people on the security team, the idea that it's a single person's responsibility to tell everyone to apply a patch is ludicrous. If it's true, then that's institutional incompetence.
I've been working in computer security for years, and do you know what I and all of my coworkers do? We keep up on computer security developments, particularly newly discovered vulnerabilities. And we discuss them. And send emails about them.
Even if the one team (not individual) who is responsible for ensuring that our own systems are patched for some reason fails to do that job, there is exactly zero chance that this would go unnoticed.
If that's not how it works at Equifax, that's the fault of Equifax, not some single individual.
Re: (Score:2)
Re: (Score:3)
I'm not so sure how effective that would be. In most of the places that I've worked, nobody takes vacation time unless management forces them to.
It only makes him look worse on /. (Score:2)
Regular people don't like us. They never have. When computers made it so they had to depend on us that didn't make them like us. It lead to resentment and deepened their hate.
Mark my words, this'll work like a charm.
huh? (Score:5, Informative)
bollocks. Yes, that.
Any security organization which relies on a single individual's action or inaction to remain in good standing is simply fairytale.
Every good process which involves a human in the loop, should always ensure that at least one more is present to enforce check-and-balance objectives.
There is a good reason why all commercial flights have two pilots as a default.
Let me state this: when you see management pointing one single downstream individual for such an event, there are at least TWO levels of management at fault.
Re: (Score:2)
Any security organization which relies on a single individual's action or inaction to remain in good standing is simply fairytale.
Fairytales are just that. However there are plenty of truly incompetent organisations.
A lot of people have called this as an "excuse" or "scapegoating" or "bollocks". I call it evidence of top down severe missmanagement of the company.
Such BS (Score:5, Informative)
Re: (Score:2)
My, aren't you the naive summer child.
The modern CEO is never to blame and, even if they are, they get their bonus and a golden parachute and cash out their stock options on the way out. If they were particularly bad, they might lose a some or all of their final yearly bonus, but they'll still walk away with more wealth than any of the 99% will earn in their entire lives.
Re: (Score:2)
Doesn't matter. The Captain of the ship is responsible for the safe operation of the ship--even if he's sleeping in his bunk (Exxon Valdez & USS McCain.)
C-level whiny bullshit (Score:2)
The CEO is responsible for hiring a CIO that do their fucking job properly. Moreover, if
$225 million isn't much (Score:5, Informative)
The company has poured a quarter of a billion dollars into cybersecurity in the last three years and today boasts a 225 person team.
Spending $225 million over 3 years isn't really that much when you consider the type and amount of personal data Equifax has on us.
JP Morgan Chase spent $500 million in 2016 alone [forbes.com], Bank of America spent $400 million on cyber security in 2016 although they have an unlimited cyber security budget [forbes.com], Citibank's cyber security budget topped $400 million and Wells Fargo spends roughly $250 million per year. [forbes.com]
Re:$225 million isn't much (Score:4, Interesting)
All of those you cite are banks with numerous branches, subject to robbery and internal theft. They have security cameras which send their video over the internet, all branches are connected to multiple financial networks including their own, and lots of mundane paperwork is computerized. Securing all of these things counts as 'cybersecurity' and goes beyond what Equifax has to deal with, for the most part. If someone breaches/hacks Equifax, and they can ignore it/cover it up, then it's business as usual, so why spend money on it? It's only once the mandatory disclosure laws went into effect they took cybersecurity seriously.
Re: (Score:2)
Not to mention comparing three companies that each have over 200,000 employees and $60bn to $90bn turnover to one that has 10,000 employees and less than $4bn turnover.
Proportionally Equifax appear to be spending substantially more on information security than those banks.
Re: (Score:2)
Failure of way more than one person (Score:5, Insightful)
Failing to apply the patch would be the failure of that one person to order the patch applied, plus the failure of his superior to notice that an action item hadn't been handled, plus a failure of the security team to notice that a ticket hadn't been completed, plus the failure of the head of the security team to notice his subordinates had uncompleted tickets sitting there. All this stuff should be tracked, and where I work it is and we have daily status meetings where stuff like this gets asked about, and development team managers and product managers have weekly status meetings where lack of progress on tickets and what needs done about it is a standard agenda item.
Accountability means managers and executives are just as accountable for work getting done or not getting done as low-level employees are expected to be.
Re:Failure of way more than one person (Score:4, Insightful)
Plus a failure of their regular security auditing process to detect that a machine was running a version of software below the minimum allowed version. All this stuff should be detected programmatically in a company that size. This was not a failure of one person. This was a complete failure of the entire security organization at every level, which usually points to either a complete lack of leadership, inadequate budget to hire sufficient qualified staff, or (more likely) all of the above.
Re: (Score:3)
By this time it's very obvious it's nothing to do with a security team.
This is just smoke from the CEO trying to protect his ass.
Re: (Score:3)
This is just smoke from the former CEO, trying to avoid prosecution.
FTFY.
Re: (Score:3)
Re: (Score:2)
Re: Failure of way more than one person (Score:2)
To put it in laymens terms, the servers are suffering from an alternative to "burn-out", where patches are not applied, infrastructure isn't replaced, and/or eventually internal support has to be provided for products which are no longer viab
Struts being an application framework... (Score:5, Informative)
Struts is an application framework, which means it is an application dependency. That means that every Struts-using application within Equifax would have needed to be upgraded, to be tested at least on the new version. That is the job of more than one person!
It is possible that Equifax's application servers (Tomcat, JBoss, etc) were configured with Struts being provided at the container level, but even that would be a full upgrade of multiple application servers within the company - a platforms team responsibility. However I suspect Struts would have been incorporated into the application itself at build time (as a dependency library).
I do not know how many applications Equifax's systems are made up of, but certainly the company I work for has dozens or hundreds to build up a trading platform (or two or three!). I imagine it is similar at Equifax.
I also cannot imagine a security team of 225 people having just one person be responsible to notifying and reminding of critical library vulnerabilities and updates for the entire business.
This smells of "VW Single Rogue Engineer" to me. Clearly bullshit.
Re: (Score:2)
I'm sure they're using ansible, puppet, chef, some kind of config. mgt. app
I love your optimism.
Gotta be someone (Score:2)
who was too busy posting on /.
"The buck stops somewhere else" (Score:3)
Sign on the desk of CxO's everywhere
(contrast this with the US Navy, where the captain of the Fitzgerald was relieved, even though he was not on deck when the collision occurred and in fact was almost killed by the accident. Subsequently, the Navy relieved several higher ranking officers, including Flag officers, for supervisory failures.)
So what you're saying is (Score:5, Insightful)
Re: (Score:2)
What about the CSO? (Score:2)
Somebody in Management decided to hire a totally incompetent and unqualified CSO. Nice omission there Mr. BS CEO.
Yep, one single bad it guy.... (Score:3)
"Integrity"? (Score:2)
The Ex-CEO, talking about the guys who cashed in their stock, said (from TFA):
I’ve know these individual for up to 12 years. They’re men of integrity.
First, his comments about the "one individual" demonstrates that he himself isn't a man of integrity, so his vouching for them means nothing.
Second, "men of integrity"? Hahahahahahaha!
A small man must die (Score:2)
When I was in grad school one of my professors talked about his. Many weak leaders, when faced by a crisis, will respond with a form of "A small man must die," instead of taking responsibility for the weakness in leadership and design that allowed the crisis to evolve in the first place.
Re: (Score:2)
Yes, this is true. One of the signs of someone who lacks integrity is that they finger-point when the shit hits the fan.
In this particular case, though, I think it's worse than just finger-pointing. I think he's straight-up lying.
Hope the scapegoat got paid off too! (Score:3)
Expecting the CEO to know _anything_ about what goes on in the IT department is expecting too much. Executives have no clue what's going on outside of the boardroom, and the only time they ever get any sort of information is from management consultants or the odd 'red alert' that bubbles up to the CFO/CIO/COO/CSO. There is absolutely zero chance that the CEO of Equifax has any idea what patch level of Apache Struts is running on their Internet-facing services.
I wonder if he just went to the CIO and said, "give me a name, anyone remotely responsible for patching, so I can say I fired someone over this." I've never had it happen to me, but I have worked with people who were scapegoats in a major incident. Sucks when you're the one holding the bag...
Re: (Score:2)
Not when your entire business is IT.
what a bs. (Score:3)
Re: (Score:2)
They DID have multiple layers of security. The highest one was 'sell the stock before disclosing the breach."
Oh wait you meant data security. Nevermind.
Re: (Score:2)
Who the fuck even deploys struts on a web server.
implementing multiple layers of security
Do you have even the slightest bit of evidence that they didn't?
Miserable little fuck (Score:5, Funny)
What a miserable, no good, lying, sniveling, double crossing, douchebag, fuckface, fucktard, dickwad lying little bitch.
From his resignation letter:
"I'm outta here suckers! Let me throw a few of you worms under the bus on my way out. Not my fault. Fuck you and goodnight."
Love, dickwad in charge, Ret.
P.S. Bitch better have my moneyyyy!
Makes me wanna shout "GWAAH!" (Score:2)
One person??? (Score:2)
Ahhh .... (Score:2)
Re: (Score:2)
exactly my thought too.
No checks and balances at Equifax? (Score:2)
How could a single person be responsible for this, with nobody assigned to verify? No redundancy or assistance whatsoever? For something so important?
They need to find out who is responsible for setting things up so stupidly.
Automatic patching? (Score:2)
In my security enclave, I automatically run patches on test systems as soon as they are released, I don't even have to do anything and monitors would let me know as soon as a critical event occurs.
And then all I have to do is move the patches from the testing channel to production and they get deployed, but even that is something that could be scripted or automated if the testing doesn't fail.
I literally spend less than 1% of my time on patching systems anymore and I manage almost 200 of them by myself.
Engineers Should Pay Attention to This (Score:2, Interesting)
If you work in engineering, you need to see the writing on the wall. No longer are you going to be indemnified for mistakes you make at work, even if you are forced to make them by bad management policy or lack of basic resources. No longer will the penalty for grievous error be a simple firing.
Face the music. If you make a mistake that causes what ends up being a tortious harm, you are going to jail.
Re: (Score:2)
This is why, if you work at a place that doesn't engage in even the most basic of engineering best practices, you should quit that job and find one that isn't actively terrible.
One individual at the top (Score:2)
Who tolerated an environment where there was no concern for the security of the data they collected on all of us.
That's the one person who is responsible. Not the scapegoat he is pointing at.
Like any CEO (Score:3)
Any good stuff that happens I did, give me a big bonus. Any bad stuff that happens, blame. The old saying *hit rolls downhill has never been truer.
This is called "violating the 4-eyes principle" (Score:2)
And that is a leadership failure. If you do this right, for each critical role, there is one person that does the change, one that verifies it has been done and and at least one that can take either role if one of the others is sick or on vacation.
Anyways, in the end it is _always_ the CEO that is responsible. This person is a coward and unfit for a leadership position, i.e. typical large-company CEO material in this sad world we live in.
And also... (Score:2)
Re: (Score:2)
The Usual (Score:2)
Re: (Score:2)
So you're upset that, contrary to the news reports and comments here on Slashdot, the CEO explicitly did take responsibility in front of a Congressional committee hearing?
Hi, where do I sign up to sitting in front of a Congressional committee and embarrassing myself for $90m?
That's the sort of responsibility I can handle.
Re: (Score:2)
Sigh. None of those 225 people had responsibility for applying the critical patch. I haven't looked into the Struts issue in detail but it's possible development work would be needed to implement the updated version.
Even without that, there's a full deploy/test/release cycle required, so you have a full dev/test team, a release team and sysadmin support. That's not the security team.
The security team will have at least a dozen people doing nothing other than user privilege management. Who can access which s