Got Malware? Get a Hammer! 254
FuzzNugget writes "After the Economic Development Administration (EDA) was alerted by the DHS to a possible malware infection, they took extraordinary measures. Fearing a targeted attack by a nation-state, they shut down their entire IT operations, isolating their network from the outside world, disabling their email services and leaving their regional offices high and dry, unable to access the centrally-stored databases. A security contractor ultimately declared the systems largely clean, finding only six computers infected with untargeted, garden-variety malware and easily repaired by reimaging. But that wasn't enough for the EDA: taking gross incompetence to a whole new level, they proceeded to physically destroy $170,500 worth of equipment (PDF), including uninfected systems, printers, cameras, keyboards and mice. After the destruction was halted — only because they ran out of money to continue smashing up perfectly good hardware — they had racked up a total of $2.3 million in service costs, temporary infrastructure acquisitions and equipment destruction."
Re:that's how u.s. government "develops" (Score:1, Informative)
A person once told me, if you country is in ruins, pick a fight with the US. They will destroy your country but build it up better than it was before, truer words never spoken.
Re:Economic Development Administration? (Score:2, Informative)
Most cost overruns are due to scope creep. Customer solicits bids, contractors bid, one wins, shortly after contract is awarded the customer changes requirements.
General cycle is:
Customer asks if they can change a requirement
Contractor says it'll cost $$$ (usually a pretty big number, because many requirements are difficult to change after you've architected your system to the original requirements)
Customer says "sure"
Costs skyrocket.
As an example, with the last presidential helicopter.
Government requested bids
Companies submitted bids
Lockheed won
Shortly after contract award, White House came up with a list of "we need this, this, and this, or THE TERRORISTS WILL WIN!" - effectively wanting to transform the new aircraft from an executive transport aircraft to a flying tank
Lockheed gave a pretty big number because these new requirements blew the original weight budget of the selected aircraft
Customer said "sure"
Aircraft now needs uprated engines, an improved transmission, strengthened rotors, etc. - original bid was a minimally modified airframe, just avionics systems integration.
Costs went through the roof.
Re:Economic Development Administration? (Score:5, Informative)
Devil's advocate:
I've worked at private companies, for education institutions, in the public sector, and in the Federal government. None are perfect, none are completely horri-bad.
All places have had those people who I had zero clue what their function was, but they always had a nice office.
It is easy to pick on government, but go to almost any work environment, and you will find the same thing.
Re:Economic Development Administration? (Score:4, Informative)
And why the hell would there be $2.3 million in service costs to destroy $170,500 worth of equipment?
RTFS.
service costs, temporary infrastructure acquisitions and equipment destruction
Or, RTFA for the details:
The total cost to the taxpayer of this incident was $2.7 million: $823,000 went to the security contractor for its investigation and advice, $1,061,000 for the acquisition of temporary infrastructure (requisitioned from the Census Bureau), $4,300 to destroy $170,500 in IT equipment, and $688,000 paid to contractors to assist in development a long-term response. Full recovery took close to a year.
Still outrageously stupid, but I think $4,300 to destroy $170,500 is a reasonable cost. I think the other costs--the ones with 6 or 7 figures--are the ones you should focus on.
But really, isn't giving US companies #2.3 million what the Economic Development Administration is supposed to be doing anyways? Better than spending it on the salaries for these government employees.
Re:Economic Development Administration? (Score:4, Informative)
$823,000 for the security contractor that adviced them to do that destruction?
Read the story, or at least read the summary.
The contractor did not tell them to do that. The contractor found exactly 6 machines, which they recommended by re-imaged.
This stupidity was not the contractors fault.
Re:Economic Development Administration? (Score:5, Informative)
best buddy system.
that's why.
That could be true; however, have you read the audit paper written by OIG in PDF (http://www.oig.doc.gov/OIGPublications/OIG-13-027-A.pdf)? It is very interesting and contains what the auditor (OIG) thinks where to blame (although those who are at fault simply brush the responsibility to others). Everything seems to be from miscommunication between DOC CIRT and EDA, and both did not know about this miscommunication until too late (the end of 2012, about a year after the incident).
What happened (from the audit paper) was that the incident handlers from DOC CIRT sent out 2 notifications to EDA regarding the US CERT notification. The first notification simply listed all 146 components, and EDA thought all of them were infected. Then the incident handlers from DOC CIRT sent the 2nd notification with accurate analysis of only 2 infected commponents, but the notification did not clarify or mention that the 1st notification was inaccurate (wrong). As a result EDA thought all 146 components were still infected.
Then the EDA selected and submitted 2 components to the DOC CIRT as a process to verify whether they were infected. Apparently, the EDA submitted the 2 components mentioned in the 2nd notification, and the result came back positive. As a result, the EDA thought that all 146 components were infected.
It got worse when EDA already knew that their IT system is outdated and needed a lot of updates/patches (since 2006 from NSA and OIG system reviewed) but they never fixed the issues. They believed this incident was an attack from nation-state actors (hackers), so their system could be extremely vulnerable to the attack. As a result, their system could open a hole to other systems' access. Therefore, the system was isolated.
Keep in mind, the Chief Information Officer (CIO) believed that this incident is from hackers. Then the EDA hired an external security company (contractor) to come in and assess the situation/system. The contractor found no actual malware infections. However, the CIO of EDA asked for a guarantee that there is non-existing of infection at all in the system [CIO is trying to safe his behind because of his belief]. The contractor could not give a guarantee due to the different between "could not exist" and "did not exist" of infections. That let to destroying the hardware part.
During the wait for recovery, the EDA entered into an agreement with Census to use their resources (e-mail, Internet, laptops, etc).
This is not done yet (and not included in the summary of this topic). The EDA did not listen to the recommendation from NSA or DHS about recovery plan -- quickly & fully recovery IT system. The EDA wanted a whole new system. This would cost $26 millions in total and won't be finished until the end of FY2014.
In summary, the miscommunication and other factors escalate the issue to be worse and worse. 1.DOC CIRT incorrectly handled the notification
1.DOC CIRT did not admit that their 1st notification was wrong to EDA
2.EDA did not verify the 2nd notification against the 1st with DOC CIRT
3.EDA did not submit random components (from 146) for verification
4.EDA IT system is outdated and has never been fixed/patched
5.CIO of EDA wanted to cover his behind by asking for a guarantee which is unrealistic
6.EDA wanted a whole new IT system which cost $26 millions
What do these people learn from the incident? No punishment but simply recommendations Deputy Assistant Secretary and the CIO of EDA (page 17 of the report/page 22 of the PDF file)! This situation is very similar to a big corporation making a mistake, and as a result, tax payers paid the price and nobody who were involved in the incident was punished.