Colonial Pipeline Sought Cyber Chief Months Before Criminal Hack (bloomberg.com) 71
The company targeted in the biggest pipeline hack in history began searching for a cyber-security chief two months ago. From a report: Colonial Pipeline sought someone with a master's degree in computer science to develop and maintain "an incident response plan and processes to address potential threats," according to the company's website. The ad also was posted on LinkedIn and job-seeking sites. A criminal hack paralyzed North America's biggest fuel pipeline late last week choking off almost half of the gasoline and diesel burned on the U.S. East Coast. Gas stations across several states have run dry amid panic buying and soaring retail prices. "The cybersecurity position was not created as a result of the recent ransomware attack," the company said in an email.
. . .so. . .what happened to the 30 applicants ?? (Score:5, Insightful)
. . . Or. as I suspect, people walked away after salary discussions. . .
Re: (Score:2)
Maybe. I would have given them the $1-million an hour.
Re: (Score:2)
Re: (Score:3)
. . . Or, as I suspect, people walked away after seeing how clueless about security the top management was there . . .
Re:Backups? (Score:5, Informative)
'Restore from backup' is easy if you have one thing to restore and/or you are 100% sure the rest of your network has not been affected in any way. It does no good to restore some systems only to have them re-infected by other systems. And verifying or restoring ALL of your systems is going to be a lot of manual work and could take a LONG time.
Re:Backups? (Score:5, Interesting)
This ++.
I work form a consumer security vendor. We got breached (technically bought a breached company, but found it too late), allegedly by a nation state actor.
The nature of the hack got us precisely into this situation: backups untrusted.
We started rebuilding all our internal infrastructure almost from scratch. We even got rid of some enterprise-grade routers.
Basically every binary was rm'ed from all systems, everything reinstalled from scratch, and a trusted source.
Took more than a year. first six month were hell to get anything done. Let alone the first week.
Re:Backups? (Score:4, Insightful)
Another kinda import thing is to actually recover from backup regularly, say weekly or daily, and get the recovery time down to a few minutes. So when you need it for real, you just run the usual procedure. For a big system like an oil pipeline, it's probably complicated, but that's why you need to have the procedure down and slick.
Re:Backups? (Score:5, Informative)
Restoring from backup can be easy, or difficult. If I restore a file server, and it gets nuked again, then it is pointless. I have to hunt down the endpoint or endpoints, flatten those, and proceed from there.
Then, there is data exfiltration. A lot of ransomware gangs are not just encrypting stuff, but slurping the data offsite, so they can threaten to sell it or publish it, which can do as much harm, if not more than outright nuking it, and a number of them threaten to dump the data if they see a restore happening.
Backups are one tool. However, security isn't just about availability and integrity, it is about confidentiality, and guarding against data exfiltration requires a number of layers.
Master's degree? (Score:3)
Talk about letting perfect be the enemy of good, a couple of Pimply Faced Youths could've put some basic security and backup plans in place quite easily.
Re: (Score:2)
Re: (Score:2)
Even if Carol from HR runs ransomware every day, it shouldn't spread to pipeline control systems, and even if it did, they should be able to get them back online from backups, because hardware failure is a real thing too.
Re: (Score:2)
Why on earth would HR be on the same network as the control systems? Or ... why would the control systems even have internet access?
Re: (Score:2)
"why would the control systems even have internet access?" because recreating the abilities of the internet for your pipeline structure is expensive, and can easily be defeated with a USB.
More generally, recreating the internet for each of water, electricity, sewage, and other critical systems is just plain dumb.
Re: (Score:2)
because recreating the abilities of the internet for your pipeline structure is expensive, .
This seems like the perfect use case for private IP services (e.g. MPLS).
and can easily be defeated with a USB
Which requires physically being onsite. That's not how these attacks work and couldn't operate at the volume and levels they do today if it required physical access. These Ukranians aren't flying over to the US to plug in thumb drives into SCADA systems.
More generally, recreating the internet for each of water, electricity, sewage, and other critical systems is just plain dumb.
Doesn't have to be the internet, just need an IP based WAN, which are sold by every major telco company (ATT, Lumen, Verizon, Windstream, etc etc etc).
Re: (Score:2)
It is possible to disable storage class USB as well.
Re: (Score:2)
Who said the control systems were affected? The company said they shut down the pipeline after they were attacked. It could be the billing system that was attacked, and they shut down so as not to be giving away free gas for a week or so.
Re: (Score:2)
But internet access isn't needed if machines on your LAN are the vector. A lot of modern control networks are split off on VLANs but there are always machines that need to see multiple of those, and if they can talk to both an infected one and the one with your controls, they pass the bug around.
Re: (Score:2)
Not to mention modern computers approaching paperweight functionality without 24/7 internet access.
I mean I know it "can be done" but it often involves hours of obscure fuckaroundry to make it work reliably if not an a bunch of extra work to maintain patches or other updates due to every software vendor ever expecting permanent internet connectivity anymore.
Like I want to "blame" Colonial as much as the next guy because I can almost hear their executives making the same bonus-protecting excuses as every oth
Re: (Score:2)
Not really. I routinely work on an isolated network. If it worked the first day, it continues to work every other day. In the case of the linux machines curated updates are available from locally hosted repos periodically. Windows machines typically can make do without. If someone needs to check their hotmail they can do it on a different machine, on a different network.
Re: (Score:1)
The "masters degree" requirement is a poison pill. Pencil whipped "masters" degrees are awarded like candy in India. Employers that want cheepo H1B labor know this.
And no, it doesn't take 6 years of university to establish backups, control attack surface or update systems.
If only there was a way. (Score:1)
If only there was a way for a pipeline to work without a computer. Now itâ(TM)s like we are back in n the 60s when all oil had to be shipped in trucks. /s
Re: (Score:3)
"The Colonial Pipeline is fully functional. The hack just affects Colonial’s ability to invoice customers. So until they can get paid, they're going to hold the east coast ransom." https://twitter.com/RobletoFir... [twitter.com]
The old IT canard that companies don't get (Score:4, Insightful)
"Why do I need all these IT people? Everything is working fine! They don't seem to be doing anything"
When the old story is the fact that everything is working fine means they are in fact doing what you pay them to do.
I would say these events should be a wakeup call for companies but that's wishful thinking...
Re: (Score:2)
The thinking in non-tech companies:
When all is quiet, it means those IT people aren't doing squat and don't deserve a bonus.
When there is excitement, the IT people failed to do their job so don't deserve a bonus.
Competence oozing out of that company (Score:4, Insightful)
First question: what the hell are a pipeline's valves - or whatever got turned off / messed up - doing on the internet?
Second question: how did this not get corrected in five minutes by calling Billy-Joe Bob down at the station to do a manual override?
Once you've considered both those questions, maybe it gives you an idea on how seriously they wanted someone competent to take care of their IT security.
Re: (Score:2)
I am sure they weren't directly connected to the Internet.
I would even be willing to bet that the control systems were unaffected by the infection directly.
My guess is that the systems that monitor the controllers were rendered inoperable and therefor the whole thing needed to be shut down for safety if for no other reason.
Re: (Score:2)
If something physical was actuated over the internet, it was connected to the internet - intentionally or not.
Re: (Score:2)
Second question: how did this not get corrected in five minutes by calling Billy-Joe Bob down at the station to do a manual override?
What station? It is likely that the valve controls are simply remote I/O in the middle of nowhere with no local personal on station. The pipeline in question is 5,500 miles long. You are not going to be stationing people at regular intervals as a manual backup along such a distance (and not even considering the people needed to cover 3 shifts plus vacationing sick leave)
As an analogy, take a look at the little shacks at the base of cell phone towers. How many of them do you think are actually manned 24
Re: (Score:2)
I didn't say anything about a station next to each valve along the entire pipeline, 19th century railway-style. I'm talking about THE station, or the 2 or 3 stations, where some dudes in overalls can turn the remote valves on and off with a dumb electrical switch, and monitor the pressure with analog pressure gauges. Surely anybody who runs a pipeline starts by getting those right before computerizing / automating the system.
Surely when the computer fucks up, someone can ask the dudes in overall to do overt
Re: (Score:2)
I didn't say anything about a station next to each valve along the entire pipeline, 19th century railway-style. I'm talking about THE station, or the 2 or 3 stations, where some dudes in overalls can turn the remote valves on and off with a dumb electrical switch, and monitor the pressure with analog pressure gauges. Surely anybody who runs a pipeline starts by getting those right before computerizing / automating the system.
Surely when the computer fucks up, someone can ask the dudes in overall to do overtime and drive the pipeline by hand.
Surely...
Dumb electrical switches and analogue pressure valves are the equivalent of 19th century railway-style. If they exist, then they only exists at local I/O stations in locations that are not manned 24x7.
Automated machinery is built with automation in mind, and tacking on purely manual controls for everything is an unwarranted expense.
Re: (Score:2)
That's fine and dandy if you run a water pipeline to the Dasani bottling plant or something. But the Colonial pipeline is a critical piece of infrastructure, and typically that sort of business is required by law to provide service availability to within a certain sigma.
Think 911 service from telcos for example: I guarantee you a telco that fails to service 911 calls for any length of time because they penny-pinched on manning the local exchange as a failsafe - or whatever passes for an exchange these days
Re: (Score:2)
What makes you think this pipeline is 'critical infrastructure' of the type that can't have any outages? Do you think this pipeline goes directly into your neighborhood gas station? It does not. The reason there are shortages is because idiots hear 'pipeline shut down' and immediately go buy gas, whether they need it or not. Then the local gas station runs out of gas. They can't get more gas because the trucks that deliver it are busy delivering to the other gas stations that ran out of gas. The shorta
Re: (Score:2)
You seem to be missing the point that in the 21st century (heck even the end of the 20th century), everything is data and the only physical connections between a device and an operator will be at a local station. Everything else is done remotely in some control room nowhere near the pipeline. Yes the operators will most likely have redundant data links and redundant controllers, but the control mechanism will be data and not long runs of control wire*. And that system will have been designed for the requ
Re: (Score:2)
Re: (Score:2)
Re:Competence oozing out of that company (Score:5, Informative)
First question: what the hell are a pipeline's valves - or whatever got turned off / messed up - doing on the internet?
Second question: how did this not get corrected in five minutes by calling Billy-Joe Bob down at the station to do a manual override?
Those systems weren't compromised. As a precaution they shut down the pipeline while the other systems were restored.
The company called a precautionary shutdown. U.S. officials said Monday that the “ransomware” malware used in the attack didn’t spread to the critical systems that control the pipeline’s operation.
https://www.nbclosangeles.com/... [nbclosangeles.com]
This should be a non-issue. The reason they chose to shut down the pipeline was because it wouldn't have any impact on consumers. The Colonial Pipeline has multiple terminals along its length. Each of these terminals, which is where the fuel is loaded onto tanker trucks for distribution, has gigantic tanks that hold the various products.
The same pipeline is used to transfer many products - the various grades of gasoline, diesel, kerosene, fuel oil, etc. So at any given time, only one specific type of fuel is being transferring to specific terminals where it fills the gigantic tanks. Shutting down the pipeline for several days would not result in any of the tanks running empty at any of the terminals.
This entire shortage is being caused by social media posts that are inciting panic. People have rushed to the pumps (and gas stations have relatively small tanks), and the stations have run out in a single day of binge buying. It's the *exact* same BS that caused the toilet paper shortage. There was never a supply issue with toilet paper, and no one was consuming any extra toilet paper. It's just that people went out and bought all they could get and the supply chains for pretty much ANY product is not robust enough to ramp up when that happens.
Re: (Score:3)
It's the *exact* same BS that caused the toilet paper shortage. There was never a supply issue with toilet paper, and no one was consuming any extra toilet paper. It's just that people went out and bought all they could get...
Not entirely true. There was a drastic shift in form factor, packaging, and product. Commercial toilet paper in office buildings is invariably giant rolls of single ply, made from lower grade pulp, sometimes recycled pulp, sold only wholesale. Home toilet paper is much smaller rolls of (mostly) double ply, made from higher grade pulp, usually virgin pulp, and is packaged and shipped to many many more locations for sale at retail. Combine all that and the shift was quite substantial and the shortage of r
Here, have a conspiracy theory! (Score:1)
Wait until they find that the attack was executed from IP addresses assigned to Blitz (gas can manufacturer). ;-)
Critical infrastructure should all be air-gapped (Score:2)
Considering even the most advanced software in the world including the Linux kernel, iOS, Android, Chrome all report weekly or monthly security issues, it should absolutely be the case that critical infrastructure should be air-gapped. I don't think it's reasonable to assume that all companies everywhere in the world become experts in cybersecurity if even Apple can't prevent their custom chips and OS from being hacked.
Critical infrastructure should be disconnected from the internet and require physical a
Re: (Score:2)
And if there was a leak or a fire or something you would be saying 'why can't they just shut it off remotely, there is no way someone should physically have to be there in a catastrophic situation like this'.
Re: (Score:2)
Considering even the most advanced software in the world including the Linux kernel, iOS, Android, Chrome all report weekly or monthly security issues, it should absolutely be the case that critical infrastructure should be air-gapped
Even if strict air-gapping isn't possible, it's certainly possible for the computers involved to be linked via a dedicated VPN and nothing else on that entire network to have direct internet access. It's done all the time in some industries.
Re: (Score:3)
Critical infrastructure should be disconnected from the internet and require physical access in order to make catastrophic changes like this. Anything less leaves you open to attacks of varying levels of sophistication, from simple denial-of-service to advanced penetration and ransomware.
You might be of a different opinion the first time you had to deal with a client on the other side of the world, in a totally different time zone, who doesn't natively speak your language, has a different cultural understanding of how work is done, isn't competent in using the tools that have been provided to them, when you are trying to debug a software issue that has caused their plant to grind to a halt, the only way to debug it is to view the operation in real time , during the middle of a worldwide pan
Re: (Score:2)
Re: (Score:2)
Critical infrastructure should be disconnected from the internet and require physical access in order to make catastrophic changes like this. Anything less leaves you open to attacks of varying levels of sophistication, from simple denial-of-service to advanced penetration and ransomware.
The pipeline works. What was impacted was the "making money from it" part. Hence they took the pipeline offline despite it being fine.
What are the odds... (Score:2)
That the attackers keep an eye out for head Cyber Security position postings and use those to select companies? See a posting. Check their website to see if they have one already. If not, time to start focusing your attention.
Re: (Score:2)
Pretty much 100%. The only reason why it still takes some time is because there are many more such ads than attackers. Also, the attackers can see by the ad staying up that the position remains unfilled. One reason why you really do not want to be in such a position as a company. If you let things rot for too long, the predators will smell it.
WHY!?! (Score:2)
Re: (Score:2)
"Why are these key infrastructures connected to the freakin' public Internet in the first place?" because you have no understanding of modern systems.
Re: (Score:2)
Re: (Score:3)
Re: (Score:2)
For a commercial enterprise, billing is more important than actually providing its service. Hence this is actually more serious than if the pipeline systems had been attacked. (I am joking, but only to a degree.)
High bar qualification? (Score:3)
Re: (Score:2)
Well that, and do people really want to short-change themselves when it comes to expertise? Security seems to be a position one wants to fill with the best.
Re: (Score:2)
Looks like hackers answered the ad (Score:1)
and aced the exam
Gen. MacArthur (Score:3)
Causal relationship? (Score:2)
So are the black hats scanning the net for security job listings at major companies, hoping to find a vulnerable corp and get in before they hire?
Re: (Score:2)
So are the black hats scanning the net for security job listings at major companies, hoping to find a vulnerable corp and get in before they hire?
Or right AFTER they hire, when security mechanisms are still in the research or flux stages. B-b
Re: (Score:2)
So are the black hats scanning the net for security job listings at major companies, hoping to find a vulnerable corp and get in before they hire?
I would say with an ad like this there is no "hoping" involved.
False flag incoming (Score:2)
Colonial Pipeline is fully functional. (Score:2)
Colonial took the pipeline offline over billing issues. This has nothing to do with the pipeline's ability to move fuel. https://twitter.com/RobletoFir... [twitter.com]
Re: (Score:2)
Well, since this is a private enterprise, not being able to bill means not being able to pump fuel. After all, making money is way more important than actually delivering a product or service in capitalism.
Gasoline shortage (Score:2)
Nothing to see here, Biden's team is on it!
On Monday, Energy Secretary Jennifer Granholm proclaimed “It’s not that we have a gasoline shortage, it’s that we have this supply crunch.”
Today, Jen Psaki informs us that "So, 48 hours ago, we said: At this moment, there is not a supply shortage. That was accurate at this moment. We also said that we're continuing to monitor very closely what the impact will be."
Re: (Score:2)
Win2008R2 unpatched servers (Score:2)
Can I make a wild guess about Colonial? (Score:3)
So my guess is that their IT department is 100% or close to it outsourced. The outsourcing company is Indian and they may possibly have a few people onsite on work visas. What few IT staff they have are so busy that nobody has time to deal with security. They are a 100% WIndows shop. So their overworked, outsourced IT department just made their network really really easy to get around in. And whoops! Permissions aren't segregated very well, so once you get in, you can get to everything. Backups? Well, maybe they do those, but they probably don't test them. And disaster recovery? Sorry, but that would cost money. And spending money on good IT staff isn't what they want to do.
Did I miss anything?
I wonder whether that job ad triggered the attack (Score:2)
I mean, the ad seems to pretty clearly say "we are in bad, bad shape".