Windows Server Hosting Provider Still Down a Week After Ransomware Attack (zdnet.com) 129
An anonymous reader shares a report: A ransomware infection has crippled the operations of a US-based web hosting provider for almost eight days now, several of the company's disgruntled customers have told ZDNet today. Impacted are all Windows-based servers owned by A2 Hosting, a provider of virtual private servers (VPS) and WordPress hosting services. The infection, which took place last week on April 23, has led to a week-long downtime that A2 staff has struggled to fix, leading to an unending stream of complaints and desperate pleas for help from customers bleeding money with each passing day of downtime.
What, no good backups? (Score:5, Insightful)
A <strike>ransomware infection</strike> lack of a good backups or the ability to restore them quickly has crippled the operations of a US-based web hosting provider for almost eight days now,
There, fixed that for you.
Re:What, no good backups? (Score:5, Insightful)
--Amen. This is why you also need to do Disaster Recovery TESTING and restore your backups every so often, even if they're just into virtual machines. Even for home use. But if you call yourself a "business" and don't do this, a real disaster can put you OUT of business.
Re: What, no good backups? (Score:1)
That's not the real world, that's self delusional fucktards making excuses for the fact they don't know what they are doing
Re: (Score:1)
That's not the real world, that's self delusional fucktards making excuses for the fact they don't know what they are doing
Actually those fucktards are the ones running the world.
Re: (Score:2)
If you don't, then your business can die due to your neglect.
Re:What, no good backups? (Score:5, Insightful)
In the real world you will find yourself (and every other employee) with all the time in the world to wish you had tested backups. This is not a "nice to have" business want, only important to businesses with the budget to cover it. This is an absolute, do-or-die, kiss the entire business goodbye without it requirement as has been show time and time again.
.
.
Or maybe it will happen to *that other guy*
Bites you back in the arse (Score:2)
you don't have the budget for the equipment to restore on.
Until that critical moment when the backup strategy fails (because it was untested) and business goes down.
Then you'll be losing way much more money that what would have been spent on testing correctness of backups.
Re: (Score:3)
Tagline: I'm willing to admit that *I just might* be wrong... Are you??
Yes, I'm willing to admit that you're wrong.
;-)
Question (Score:3)
Genuine question....
Re: (Score:1)
It's your *data* that matters, not so much your system. Unless the patch is a major upgrade that might somehow make the user-space setup incompatible (not likely), you just need to back up the user-space. Then you should be able to install a patched OS and backup the customer's data. OTOH, if you've just got images of compromised VMs, then perhaps restore those images and use a firewall to limit network access to just the IPs needed to download and install a patch. Then restore, patch, re-enable full ac
Re: (Score:1)
Re: Question (Score:4, Interesting)
Re: (Score:1)
According to the summary, they are disabling RDP in the restored images. Apparently the attack comes through RDP.
Yup. If it is in fact GlobeImposter.
GlobeImposter installs through a brute force attack against RDP logons.
This is not a Windows vulnerability if you consider that RDP is not turned on by default in Windows, and also that brute force attacks only succeed when the sys admins allow lazy passwords.
This one is entirely a failure on the part of either incompetent admins or it is an inside job.
Re: (Score:3)
Hmm, seeing as how RDP is the primary way that Windows servers are administered it raises the question, just how are customers accessing the machines now? VNC?
Re: (Score:3)
Hmm, seeing as how RDP is the primary way that Windows servers are administered it raises the question, just how are customers accessing the machines now? VNC?
Yep. VNC into the network, and then RDP. Fact is that any routable IP will get attacks and any routable IP with RDP will get triple that in specialized attacks. Our remote server got thousands of fake login attempts a day when it was accessible. Just had to put everything behind the firewall, have them use VNC into the network an then remote in to the server. And sometimes, they remote into a server that they then use to remote to the real production server with a different login and password.
Re: (Score:2)
But VNC is just the same, it will get hammered with thousands of malicious login attempts every day.
The proper way to fix it is to set up a VPN into the network, and then RDP.
Re: (Score:2)
But VNC is just the same, it will get hammered with thousands of malicious login attempts every day.
The proper way to fix it is to set up a VPN into the network, and then RDP.
You are most likely correct as that is what I meant. We have a VPN into our network, not a VNC. Too many TLAs.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Also assuming your backup is full or almost full, you can always sandbox your restoration system and recover as much as possible, beginning with the most valuable portions with an OOB scenario. That is like repairing a broken Windows image by booting a Liv
Re: (Score:2)
Find out which under paid or under threat employee, provided the route in. I can see one server going down but spreading, something suss going on in there. In the tech world, it just takes one, one in the right position at the right time, to sell access for thousands of dollars, that costs millions of dollars and even more. Being cheap ass with tech employees and or out sourcing become way more dangerous.
How much damage, can be done how quickly, how much can tech corporations lose how fast and how much can
Re: (Score:3)
Re: (Score:3)
If you quickly restore the backups, the ransomware quickly trashes everything again.
If you are the sort of operation that has the ability to quickly restore and isolate the problem, you are spending more on hardware and people, so you have to charge more. Then a fair percentage of your customers say "OMG If I switch my business critical site from these people to that other provider nobody's ever heard of, I can save $50 a year!!! HOLD MY BEER!!!". The tens of people complaining ARE those "hold my beer" peop
Re: What, no good backups? (Score:1)
Re: (Score:2)
They don't outdo anybody if they go with the dirt cheapest possible host and don't keep their own backups of their valuable data. At least not for long. For example, the people who are still down hoping the hosting provider can manage to restore their data sometime soon.
Re: What, no good backups? (Score:1)
Re: (Score:1)
I am wondering if they foolishly kept their backups on-line. Most of the ransomware I have seen will quietly encrypt the computer's hard drive it lands on first and then it will start looking down the shared network drives as well. I have often wondered if these larger outfits used NAS boxes with backup software that is continuously updating so that they can recover (when there's no ransomware involved) almost to the minute if they have a disk failure or the customer deletes by mistake etc.
I only look after
Well, they *are* using windows... (Score:4)
Funny, you don't hear of these types of major, long term ransomeware on Linux servers.
Windows has its place, but IMHO....the commercial or government high end server room just isn't that place.
Re: (Score:2)
It is true that any OS is only as secure as you make it. But it;s a lot easier to make it secure when it's all laid out for you to see and there is no army of lawyers ready to swat at your hands when you try to lift the hood.
Re: (Score:3)
While we love to Rant against Windows. Windows Server is actually jammed packed with security features, that isn't available with Linux.
The real problem is the defaults. By default most Linux Distributions are more secure then Window. But a good Windows Administrator with proper support from the company, can have a Really stable and secure windows environment, where the server is really locked down to do the job it was intended to do, and avoid everything else.
Re: (Score:1, Troll)
what nonsense, I've seen windows server do nothing but fail, allow intruders and spread infection over the decades, it's garbage.
the "security features" you speak of are a farce, making major news as they fail time and again.
your emperor has no clothes, you are a shill for garbage
Re: (Score:1)
I've been securing linux and windows for decades. Each has their place and neither has a fundamental and significant security advantage.
A good technologist doesn't let their OS fail.
Your OS failed...
Re: (Score:2)
Well you will get that with a wizard install, select all features, and hit Next until it is done, and plug the server right into your main internet connection.
In my experience I had my share of Unix/Linux security problems. Issues with users setup with root access, services installed that are a security problem, with web configuration tools open to the general internet. Unpatched ancient ports open with security problems, Telnet enabled...
Re: (Score:3)
You will also get that with highly trained and certified windows admins following long lists of things to secure such as the places I've worked over the last two decades. Windows is just unfit for business use.
You linux problems were caused by deviation from defaults the big server distros use. Users don't get root access by default, telnetd is not installed, web configuration wizards to make non-admins into "click-and-point admins" etc. Someone deliberately built a house of cards.
Re: (Score:2)
haha, random AC mocking actual professionals from their mom's basement. you'll never get laid with that mindset, boy. get out.
Re: (Score:2)
who uses red-shat's garbage? not my employer and not me.
Re: (Score:3)
While we love to Rant against Windows. Windows Server is actually jammed packed with security features, that isn't available with Linux.
The problem with Windows is also that the security features are quite tricky to understand and manage.
Windows looks simple from the outside, but when you want to do serious stuff then it's hard to get it right.
Re:Well, they *are* using windows... (Score:5, Informative)
In my day job, I manage both Linux and Windows servers.
You and GP are correct. Windows has a lot of security features available, but they aren't enabled or configured by default.
Linux distros usually assume that people want security. Defaults will be configured to be reasonably secure, and while that may break some software, it is expected that sysadmins will be competent enough to fix whatever broke.
Microsoft assumes that people want working computers. Updates change as little as possible, and new security features that might break compatibility are usually left waiting for opt-in, or left with empty configurations so they don't affect anything. Sysadmins are expected to be reading Microsoft's security bulletins, and reconfiguring their systems appropriately for the new security features.
Our stereotypes of systems are exactly what you'd expect from these policies. Linux systems are secure, but require a lot of break-fix work and substantial expertise to keep the enterprise running. Windows systems can be made secure with a lot of work and awareness, but often that becomes a sysadmin's lowest priority.
Re: (Score:2)
Windows has a lot of security features available, but they aren't enabled or configured by default.
Note quite. Almost all internal security features that the system itself is in control of is enabled by default in the modern world. Here we're talking about things like UAC, aggressive anti-virus, firewalling on public networks etc.
What isn't enabled by default are things that require user interaction: bitlocker setup if it wasn't done during setup, and they are then typically the result of a nag campaign.
Likewise (and relevant to this case) things that require secondary services are also not enabled. In t
Re: (Score:2)
Windows Server is not that different, but the security policies usually are, just by nature of being in an enterprise.
Offhand, I'm thinking about things like granting administrator rights... Instead of just making users be "Domain Admins", a proper enterprise configuration has users be given the least necessary privilege, like "DNS Admins", "Backup Operators" , or "Remote Desktop Services Users", or creating new groups appropriate to the organization and using Group Policy to assign the rights that group ne
Re: (Score:3)
Windows marketing (even for the server variants) was always based on the idea that it's "easy to use" and "got a gui", the selling point was always that you don't need to hire highly trained, rare and experienced admins to run it like you do for unix.
As you point out, you do need to hire good admins, and they are just as expensive (if not more so) than unix admins, and just as difficult to find.
Re: (Score:2)
> Windows Server is actually jammed packed with security features,
And layers of vulnerabilities not exposed for a normal UNIX or Linux server. The remote management tools, such as the classic PowerShell, rely on CIFS file-sharing access to the "C:" drive. The escalation of privileges needed for basic system access or normal software configuration is much less well defined, especially with the broad variety of installation tools which _must_ be supported and the lack of visibility into those tools.
Re: (Score:2)
It can happen on Linux, but in general, Windows is more common workstations and desktops (where the maximum ransomware payoff is), and it is harder to inject malware into Linux (though not impossible.)
All malware needs is a user context in Windows, and they can do a lot of damage. In fact, a number of ransomware variants don't even bother trying for admin rights, since the big money is made by nailing the user's documents.
Re: (Score:2)
Funny, you don't hear of these types of major, long term ransomeware on Linux servers
Does MongoDB [networkworld.com] count? That's the closest I could find.
Re: (Score:2)
Funny, you don't hear of these types of major, long term ransomeware on Linux servers.
Yeah it's funny how we don't hear {insert specific thing affecting one platform to make a biased point} affecting {insert other platform I am praising} very often. {Note: don't forget to not mention all the other things that affect that other platform otherwise I would look rather stupid}
No sorry you look stupid anyway. Poor backup strategies combined with leaving open attack vectors does not care which OS you are currently running, and in the past year we have run enough stories on Slashdot of major provid
That sucks. (Score:4, Funny)
Then again, Wordpress on Windows, lol? File this under "what did you think was going to happen"
Never her of A2 (Score:5, Insightful)
So a mid sized hosting company, got hit, and they didn't have a proper disaster recovery plan. Now customers are pissed.
I wonder how many employees will get fired for this. Probably the ones who put in a formal request for a disaster recovery process and extra storage to allow for backups in case of such a problem.
Re: (Score:3)
I wonder how many employees will get fired for this. Probably the ones who put in a formal request for a disaster recovery process and extra storage to allow for backups in case of such a problem.
I see you are wise in the ways of corporations.
Re: (Score:2)
I have been around the block. Disaster Recovery is often an expensive part of hosting, which is considered a component that may never be used. Companies that are trying to run at low costs, will often do the bare minimum for this. And companies trying to explain why they charge more have a hard time justifying to customers why they are doing so much extra overhead.
Granted this has been getting better, but still it is an issue.
Re: (Score:2)
Fundamentally the issue is that it's hard for the customer to see. You might be paying out the nose for a fancy provider, but do you actually know whether they're really doing all this work behind the scenes? Or will they just up and disappear when something hits the fan?
Re: (Score:2)
And this is also why you should be wary of outsourcing all your stuff.
Re: (Score:2)
Except for the fact, for most companies they don't have the budget to host it themselves, and will end up with a Server under the bosses desk, being used as a foot stool. In which he will probably log into and browse the internet with.
Re:Never her of A2 (Score:5, Interesting)
Got laid off, they pissed me off doing it. About 2 months later got a call from them. Seems nobody had been doing backups, the existing backups got lost in the layoff shuffle, and the Windows admin that took over my job messed up and lost all their data. They asked if I had any backups, I looked at my bookshelf with maybe 2 years of monthly backups, smiled, and said "nope".
It's been 30 years, statute of limitations is long past, and the company went tits up almost 30 years ago.
Re: (Score:1)
Heh. Got laid off at a major "content delivery network" provider. I had cleaned up the kernel source code, cleaned up the crap branches written by a stack of former NetApp employees working for us who were busy changing whitespace and not doing actual kernel work. Found the *one change that mattered*, which was a public domain Squid tuning modification and merged it back to the clean Linux kernel code base for use for future kernel work, and it got held up behind QA delays for a year. Got laid off, gave my
Re: (Score:3)
should said my rate starts at $150/HR with an min service commitment
Re: (Score:3)
Here here. This is what I did. I was back up monkey for a company. Of site backups was the trunk of my car. They didn't want to do a proper tape rotation to offsite storage. I finally manged to talk them in to a safety deposit box in a bank out of town. Every week on the way home I would drop off new tapes and pickup the old ones.
One down sizing later and I was working for a new company. I got a call a few months later from the old company. They had an issue and couldn't get into the deposit box
Re: (Score:2)
I quoted them $75 to come down and sign the papers. I thought i was reasonable considering i would have to take time off and actually drive 200 miles.
$75 for a 4-hour drive? You robbed yourself with that one.
Re: (Score:1)
Wish I had a chance to bite that apple. I would have said - Why yes I do. It's going to cost you. I'll be right over to discuss this.
Who knows, maybe it was for the best. Maybe they'd sick their attorneys on you and force you to turn it all over and cost you a bunch of money.
Never know.
I hope everything worked out for you.
WORDPREES ON WINDOWS?? IIS? (Score:2)
WORDPREES ON WINDOWS?? IIS?
Re: (Score:2)
A full stack of turds.
I'll never forget this whenever I see job ads for "full stack developers".
Use the cloud (Score:5, Funny)
Eventually, you will get rained on.
Happened To Me Once (Score:5, Interesting)
Years and years ago, I was using a hosting provider that went down "due to an infection in their servers." Unlike this case, it wasn't ransomware, but a worm which was an easy fix. (Remove server from the network. Reboot. Apply patch. Reconnect to network.) They were down for over two weeks without contacting their customers who obviously were panicking. I managed to connect to their SQL server at one point and quickly made an updated backup of all of my data. (I had a backup of all of my files already but had slacked off on data backups.) Other people weren't so lucky. Then, without any further notice, they vanished. I later found out that the owners set up shop as a brand new web hosting provider. Basically, instead of fixing their issues, they just ditched their customers and started over with a new name.
That experience taught me the value of having regular backups. Always assume that your provider might go out of business tomorrow and have contingency plans.
backups (Score:1)
Re: backups (Score:1)
I was just reading an article yesterday about ZFS where the author says "In a redundant setup, such as four mirrored hard drives, it absolutely guarantees you will never lose one bit of data.. ZFS can self heal and recover data automatically." So who needs backups? :rolleyes: LOL
Some people just never learn, even the hard way. They'd have to admit they were wrong first.
AE911Truth Org
Re: (Score:1)
Re: (Score:2)
A2Hosting: Haven't seen problems. (Score:3)
Helpful point. Thanks. No problems with Linux. (Score:3)
I just modified a page on our A2Hosting web site. No problems.
Was the vulnerability associated with WordPress? Or with Windows?
In this article, of the Top 3 Windows Hosts, A2Hosting is number 1: Linux Hosting vs. Windows Hosting" (2019): 6 Differences & 6 Best Hosts [hostingadvice.com].
Wordpress vulnerability? Or Windows Server? (Score:2)
Windows Server Vulnerability Statistics [cvedetails.com]. 37 vulnerabilities in 2018.
I should have been more careful in commenting. (Score:2)
We use A2Hosting Linux servers.
Could they have done better? (Score:5, Interesting)
I am a customer of A2Hosting, running on one of their Windows-based host offerings, and I was affected by the problem.
My site is close to meaningless for anyone but me, so let's dispense with berating the obvious idiocy of hosting on Windows in the first place, and also my reasons for doing so.
Prior to this incident, my site had close to zero issues, and was always _very_ fast. I was quite happy with that. I can't really compare their support to others, but I wasn't super impressed in general, though issues did get resolved.
I am also unfamiliar with the hosting business, and disaster plans for same - but let me throw out some guesses:
First, I've read that they have in the neighborhood of 250,000 customers. I will guess wildly that perhaps 10 percent of those are on Windows hosting for whatever reason.
I believe they shut down _all_ of their Windows servers once they realized what was going on. I'm imagining that all of this means that they have to re-build any number of server boxes from scratch, and then find a way to restore twenty five thousand sites from backup (files, and databases separately, of course). As much as I hate what happened, I have to admit that it sounds like an extremely daunting problem to solve. It doesn't really surprise me that it took a number of days - in my case, 4 days, for my site to come back up completely (albeit with dramatically poor performance as compared to before).
My original reasons for choosing Windows hosting in the first place have diminished quite a bit, and I will certainly be migrating off of Windows. I am also leaning toward choosing a new hosting provider, but I haven't decided yet. I really don't know if they could have done better, since details of the event are still sketchy. I'd really like to know if their response was actually poor in the face of the magnitude of the problem, or if it's near the best that could have been done in those circumstances.
Re: (Score:1)
Irony? (Score:4, Informative)
The day before they got hit they posted this on their blog: https://www.a2hosting.com/blog/security-breach-plan/
Windows on the server? Madness! (Score:2)
MS barely have their desktop offering under control and it is unclear whether it really has a future. Using Windows on the server is pure insanity.
This is not news (Score:2)
Customers responsible for backups (Score:2)
Apparently the hosting contracts say that customers are responsible for backups. Why would business customers accept this requirement and then not have their own backups? On the other hand, it doesn’t look like A2 was offering to put up clean instances for customers who wanted to do their own restores.