Hackers at Center of Sprawling Spy Campaign Turned SolarWinds' Dominance Against It (reuters.com) 49
An anonymous reader shares a report: On an earnings call two months ago, SolarWinds Chief Executive Kevin Thompson touted how far the company had gone during his 11 years at the helm. There was not a database or an IT deployment model out there to which his Austin, Texas-based company did not provide some level of monitoring or management, he told analysts on the Oct. 27 call. "We don't think anyone else in the market is really even close in terms of the breadth of coverage we have," he said. "We manage everyone's network gear." Now that dominance has become a liability -- an example of how the workhorse software that helps glue organizations together can turn toxic when it is subverted by sophisticated hackers. On Monday, SolarWinds confirmed that Orion -- its flagship network management software -- had served as the unwitting conduit for a sprawling international cyberespionage operation. The hackers inserted malicious code into Orion software updates pushed out to nearly 18,000 customers.
[...] Cybersecurity experts across government and private industry are still struggling to understand the scope of the damage, which some are already calling one of the most consequential breaches in recent memory. [...] Experts are reviewing their notes to find old examples of substandard security at the company. Security researcher Vinoth Kumar told Reuters that, last year, he alerted the company that anyone could access SolarWinds' update server by using the password "solarwinds123" "This could have been done by any attacker, easily," Kumar said. Others -- including Kyle Hanslovan, the cofounder of Maryland-based cybersecurity company Huntress -- noticed that, even days after SolarWinds realized their software had been compromised, the malicious updates were still available for download.
[...] Cybersecurity experts across government and private industry are still struggling to understand the scope of the damage, which some are already calling one of the most consequential breaches in recent memory. [...] Experts are reviewing their notes to find old examples of substandard security at the company. Security researcher Vinoth Kumar told Reuters that, last year, he alerted the company that anyone could access SolarWinds' update server by using the password "solarwinds123" "This could have been done by any attacker, easily," Kumar said. Others -- including Kyle Hanslovan, the cofounder of Maryland-based cybersecurity company Huntress -- noticed that, even days after SolarWinds realized their software had been compromised, the malicious updates were still available for download.
even when it's easy, security is hard (Score:2)
Hey guys, another site just got hacked... (Score:1)
So, my friend tells me he was doing some "security research" and it turns out a whole bunch of videos that used to be available online are suddenly just completely gone. These damned hackers are really making a mess of the Internet.
Re: (Score:2)
You made my day ! Thanks a lot !!!
Re: (Score:2)
It's because no one at the company actually cared.
Re: (Score:1)
Either that or they were paid to not care, paid a whole lot to care a whole lot less about security. The modern age, tech companies are crippled by this stuff, financially, especially the share price of public ones, those that get gutted by being hacked in a big way and those that get the business from the now gutted corporation. There is a whole lot of financial incentive to do this, we are talking billions of dollars and that pays for a whole lot of corruption, it will only get worse. The US government se
Somewhere, someone... (Score:3)
...said to a supervisor, "Hey, we still need to change the default password." The dev supervisor told them to put it on the punch list and before the week was out, the dev that brought it up got replaced by an H1B visa holder.
Re: (Score:2)
Re: (Score:2)
You'd think if you told someone you were going to securely monitor their network (is that what these guys do?) and your security consisted of the password "{companyName}123" you'd get sued. A lot.
The entire model is upside down (Score:5, Insightful)
While this attack came from the upgrade channel, all these top-down management products are a danger as they put administrative read-write creds for critical devices on a steaming pile of attackable code surface, a general purpose box with a bunch of messaging middleware, usually multiple database products, a web server, backend protocol drivers, file image server, etc, etc, etc. Even when not used for upgrading images, often the critical managed devices do not have sufficient authorization granularity to restrict these credentials. Where used to generate configurations an incomprehensible mess of a tree full of templates and overrides is the inevitable result.
There's a place for centralized configuration auditing, but the device should be the one that has creds for the management system, not the other way around, since the management system has much more leeway to restrict the authorizations given to a device. Also, the actual effective configuration running on the device should be the record of authority. If there's a need for distributing configurations at scale, or planning configurations ahead of an install, that should be done on a separate system from monitoring and auditing, and that system should be dirt simple and support prompting for credentials at deployment time rather than storing them.
On the device side better control over secrets and crypto material during backup and tech support operations is all that's really needed under a bottom-up model.
Re: (Score:2)
I suspect part of the problem is a skill gap. Personally, I take bottom up to the full extent. Changing a network means logging in to individual routers in an appropriate sequence. If you do that, there is no single point where the bad guys can subvert your entire network, but there are also no tools to help people who don't actually know how it all works and how the configurations fit together to make a functional network.
Network monitoring should be a read-only operation. Backups of network configuration
Re: (Score:2)
I suspect part of the problem is a skill gap. Personally, I take bottom up to the full extent. Changing a network means logging in to individual routers in an appropriate sequence. If you do that, there is no single point where the bad guys can subvert your entire network, but there are also no tools to help people who don't actually know how it all works and how the configurations fit together to make a functional network.
Network monitoring should be a read-only operation. Backups of network configuration should be a push operation to a server with version control.
Good plan but it doesn't scale.
Re: (Score:2)
Actually, it does if you know what you're doing and engineer the network right.
Re: (Score:2)
I recently had to make a small config change on ~800 network devices. There is no practical way to do that without automation. And my company is only mid-sized, I know people who manage thousands of switches and routers centrally.
Re: (Score:2)
Now ask yourself why you had to make the small change on 800 devices. Without more information, I can't really comment further.
Re: (Score:2)
I occasionally have to make changes on hundreds of devices. I do this with a 300 line script that does not know the device credentials until I provide them and forgets them when it's done. That's all the automation you should really need except in complicated WAN setups where you have to sequence operations to protect connectivity during the changes. If you have to get more complex then you need to know how to code. Or go home. None of it should require a box running hadoop/mongodb/tomcat/blah/blah/bla
Re: (Score:2)
My moto is that anything that ca
Re: (Score:2)
I agree on the skill gap. Too many people that want a "magic box", because they cannot actually do the manual version. People that can do the manual version realize that this should be read-only and that attacking the frigging monitoring should never result a compromise of the network and systems. But these people are not who buys the magic boxes.
That said, anybody that falls to a supply-chain attack like this one and then takes forever to notice has incompetent management, incompetent engineers and broken
Re: (Score:2)
Re: (Score:2)
Others -- including Kyle Hanslovan, the cofounder of Maryland-based cybersecurity company Huntress -- noticed that, even days after SolarWinds realized their software had been compromised, the malicious updates were still available for download.
This tells us everything we need to know about this company. They suck.
Pretty much. For things to be this broken, there has to be an high level of rot in there.
Isn't it Lovely? (Score:2)
Re: (Score:2)
Microsoft isn't even the problem much anymore, think about that.
Re: (Score:2)
Re: (Score:2)
A lot of single points of failure exist on the internet, and we will read about them as they get abused over the coming years. But sure lets put everything into one of the handful of clouds and use a single source for all worldly updates.
Microsoft isn't even the problem much anymore, think about that.
Au contraire. Microsoft is now a problem as a cloud provider. Although the last one to have messed up spectacularly is the (these days) ever semi-competent Google.
Re: (Score:2)
Well, when our SOC contacted me in the night with this, my first response was "A supply-chain attack. Nice!". Fortunately the guys took my advice and shut this crap down fast.
I called it (Score:2)
anyone could access SolarWinds' update server by using the password "solarwinds123"
Default password problems [slashdot.org]. and anonymous security sources often lie [slashdot.org].
But hey, saying "it was an advanced attack by the Russian Military" makes it so much easier to shift blame/focus, even if there's no evidence.
Re: (Score:2)
Re: (Score:2)
They didn't go for quick ransomware attacks or sabotage, but played the long game, stayed under the radar, and slowly infiltrated customer networks.
Oh yeah, no hacker ever does that other than the Russian government.
Oh wait no, that's been a common hacker technique since the 90s. Hack in, ask, "What is here?"
Re: (Score:2)
Hack in, ask, "What is here?"
...And the specific tools and methods they use once they're in. No, there is no conspiracy to pin any breach on "the Russians", and it's not that hard to tell a script kiddie from a state actor.
Re: (Score:2)
...And the specific tools and methods they use once they're in.
Which tools? Show the evidence or GTFO.
Re: (Score:2)
Re: (Score:2)
Not to mention the only actual observable is the claim by the company.
Press release 1:
"We got hacked by Russian intelligence."
Press release 2:
"We got hacked by someone, probably a bored kid in a basement somewhere. Possibly not wearing pants."
Re: (Score:2)
To leap from this unrelated issue to the firmware compromise, you must really not care about this issue beyond finding an excuse to claim it wasn't Russia, because if you did two cents worth of research you'd find out that was just a read-only account probably used by sales guys to give potential customers eval access to software. Commonplace in the industry. In fact a good number of companies post their images with no password needed, where they have good control over their own used equipment market or a
Re: (Score:2)
you must really not care about this issue beyond finding an excuse to claim it wasn't Russia,
I'm here to mock dumb companies who use crappy security practices and come up with lame excuses when they get hacked.
In fact a good number of companies post their images with no password needed, where they have good control over their own used equipment market or a reliable licensing system.
So what you're saying is SolarWinds has solid security practices? Give me some of whatever you're smoking because it's the good stuff.
Re: (Score:2)
I'm saying easy read-access to their firmware images was neither the source of their current problems, nor unusual, nor a particular security risk. Even when requiring a login. the protection afforded is only a legal formality... hordes of customers have the same images and can distribute them widely if they wanted to, without any real risk to themselves as long as they do so anonymously. They are not secrets in any real sense of the word,
Re: (Score:2)
They probably thought it was OK because every component needs to be signed. Oops.
https://www.fireeye.com/conten... [fireeye.com]
Re: (Score:2)
The attacker needed read/write access, the ability to embed a back door into the official software (without breaking anything), and then needed to be able to sign the result with the Solar Winds cert.
This wasn't a bored script kiddie.
Re: (Score:2)
True but you can't say, "It wasn't a script kiddie" and jump to the conclusion that it was a nation state. There are many people who are able to use a cert to sign software, and I dare say I am able to do it myself.
Re: (Score:2)
There is a tendency to assume only a state actor could pull it off, and perhaps that was true 20 years ago, but given the size of some cybercrime operations, it IS possible that this was a private effort.
How about using nagios instead? (Score:2)
Maybe not as easy to just hand off to some external company, but roll your own or something, and keep it inside your firewalls.
Beware of security appliances (Score:5, Informative)
We used to use a WAF until I learned that it was managed by people outside our country where there was no extradition treaty. Moreover, it was a total managed service black box with no access of any kind, so I had to simply trust that it was secure. That trust was probably OK, but my philosophy is that you cannot trust what you cannot inspect. This usually means that you treat everything as not trusted. You can work with this by assuming end to end encryption has you covered. Usage of black box security appliances erases trust in end to end encryption when you intentionally install certificates on an untrusted device.
Unfortunately, these appliances that perform automated vulnerability scanning and many aspects of penetration testing are part of the huge computer security certification scam. Shaking companies down by withholding certification until they spend enough money is a standard practice in my experience.
Re: (Score:2)
Ah, I have a product your employer may be interested in. We should discuss business, comrade. Uh, I mean, honoured sir.
Re: (Score:2)
We used to use a WAF until I learned that it was managed by people outside our country where there was no extradition treaty.
Nice! Talk about having one bad apple in a company like that...
This shit will continue (Score:2)
Until companies can be held legally liable for their bad practices and vulnerable software.
There's no incentive to fix the problem.
Re: (Score:2)
Indeed. Even in the most extreme circumstances, the worst that can happen is that the CEO gets a golden parachute. Perverted incentives to the max.
Re: (Score:1)
Password Needed To Be More Secure (Score:1)
As other have pointed out and are probably right that someone, sometime said "That password is a bit sh!t.", and the response was, "We can't, do you have any idea of how much that will cost?", then that someone pushed the point and was fired/replaced for not being a "team player".
Hey Solarwinds, how is that money saved working out for ya?
It is almost beyond belief that this was allowed to even occur, and I agree it is criminal negligence. It has d
Re: (Score:1)