Hackers Wipe US Servers of Email Provider VFEmail (zdnet.com) 157
Hackers have breached the severs of email provider VFEmail.net and wiped the data from all its US servers, destroying all US customers' data in the process. From a report: The attack took place yesterday, February 11, and was detected after the company's site and webmail client went down without notice. "At this time, the attacker has formatted all the disks on every server," the company said yesterday. "Every VM is lost. Every file server is lost, every backup server is lost. This was more than a multi-password via SSH exploit, and there was no ransom. Just attack and destroy," VFEmail said. The company's staff is now working to recover user emails, but as things stand right now, all data for US customers appears to have been deleted for good and gone into /dev/null.
There were NO offsite backups????? (Score:5, Insightful)
No offsite backups? No tapes????
Who designed the disaster plan for these guys?
Re: (Score:3)
No offsite backups? No tapes????
Who designed the disaster plan for these guys?
Same geniuses as Wells Fargo?
Re: (Score:3)
It's all in their private cloud, of course!
Re: (Score:3)
You mean offline.
Nothing happened to any particular location.
Re:There were NO offsite backups????? (Score:4, Insightful)
It *is* a PITA to put a tape in your bag, open up the fireproof safe at home, throw it in, get the *correct* one out, put it in your bag, and remember the next day to put that where it needs to be. And repeat. /s
I did that for years. And I slept a little better.
Re: There were NO offsite backups????? (Score:2)
Actually I had three clients I did off-site tape rotations for, for about 6 years. One made regular random requests for tapes to do a directory test on. The other was a bank, they did a full scan and compare. My own, the company tapes, since I was the sysadmin, I did compares quarterly. DAT and higher capacity tapes were in use, LTO and such. Never had to restore my own, don't know if the bank did, but the other client was fastidious.
I was much lazier with my own server backups, having just software tapes u
Re:There were NO offsite backups????? (Score:4, Funny)
From a cannon. Into the sun.
Re: (Score:3)
No offsite backups? No tapes????
Who designed the disaster plan for these guys?
The plan was a disaster - mission complete!
An online copy is not a backup, guys. It can be a great cache of a backup, but it's not a backup. Who still doesn't know this?
Re: (Score:3)
Re: (Score:2)
Any backup is a backup
Semantics.
A copy is a copy in the copy is a backup and it doesn't really matter where you store it as long as your whole strategy makes sense.
Anything online, especially at the same site or accessible from the same site, is a convenience, not a backup strategy. Sure, it's handy when someone says "oops", but it doesn't protect your business.
I certainly don't blame people for trying to get off tape backups, and they are kind of a huge and slow pain in the ass.
LTO-8 is 360 MB/s assuming no compression, for a single drive. An internet pipe that give you better than 3.6 Gb/s upload is impressive for a small business.
Cloud backups make sense when all your data is already in the cloud. You don't really have much choice at that point. Or if your data changes
No backup can be a feature (Score:5, Interesting)
That can be both a bug and a feature. No backups mean that there's no cache of deleted emails. Some users may want the ability to truly delete data, not have it able to "appear" due to legal proceedings 5 years from now.
I'd say it's on the users to back up their email using a client that locally caches IMAP folders or downloads via POP3.
Re:No backup can be a feature (Score:4, Insightful)
It would seem more practical to just limit the stored backups to the last n copies, like you do with rotated log files.
If it can only come back for two weeks or something, that is sufficient for most use cases.
Re: (Score:1)
That can be both a bug and a feature. No backups mean that there's no cache of deleted emails. Some users may want the ability to truly delete data, not have it able to "appear" due to legal proceedings 5 years from now. I'd say it's on the users to back up their email using a client that locally caches IMAP folders or downloads via POP3.
I used to do that, starting in 1993. But I've used so many different computers since then, so many different email clients. My archive got too big to fit conveniently on my computer's storage. So then I was stuck with a load of separate volumes of backups that were hard to search. I wrote software to merge volumes of archives when I upgraded to bigger disks, also to export them into other formats.
In the end, it was too much work for an inadequate solution. Now I just pay $8/month for an Exchange365 account.
Re: (Score:2)
I don't really back up my mail. I copy one mailbox to two others via IMAP and POP3. Another mailbox I copy via IMAP. And then I also have a copy in my beloved (/s) Microsoft Mail. It's not really a backup, I think, just copies. And I have a glorious spam library dating back to the 90s in some of it, just too lazy to clean it up.
If you remember spam from the 90s, you know why some of it I've had to delete.
Re: (Score:2)
Re: (Score:3, Interesting)
The business plan probably.
If you do make backups, you are too expensive, certainly cannot compete, and will go out of business. No income for you.
If you do not make backups, you may make a nice buck for a while before the thing explodes in your face. Hell, maybe you are lucky and it never explodes at all.
Regardless, at least you will make money for a while. So this scenario is clearly the winner. Screw the damage to your future ex customers, that is not your problem.
Re: (Score:2)
A more important question is "Why were the backup servers accessible from the email servers?"
A good network design has the backup servers isolated from the production servers. Only the ports need for backup should be allowed. Even if using a copy over SSH, it is possible to set it so the backup servers can access production, but block all access from production to the backup servers. I should never be able to gain console/terminal access on the backup servers from production.
Re: (Score:2)
Re:There were NO offsite backups????? (Score:4, Insightful)
Once you're in the front door, you're going through the system. Only offline backups can be trusted to 'be there'.
And no offline copies of the VM environment? I think of those as especially precious. DO I want to rebuild those from scratch? Nope.
Re: (Score:2)
And no offline copies of the VM environment? I think of those as especially precious. DO I want to rebuild those from scratch? Nope.
They probably didn't build them to begin with, odds are they did it all with someone else's containers and they had no clue what was actually running on those systems.
Re: (Score:2)
They probably didn't build them to begin with, odds are they did it all with someone else's containers and they had no clue what was actually running on those systems.
^^^^^THIS.
Yep, they most likely took some base container, maybe modded it a bit, and threw it into service. They likely have no idea at all how it was configured or what was in it.
People are always concerned with the data and forget about the infrastructure that it lives in.
Re: (Score:2, Insightful)
What has a higher chance of getting owned? A network accessible box wide open to the web, or a backup server that can only be accessed by SSH via a specific management VLAN?
Re: (Score:2)
If you've compromised the server, you're well on your way to all the connections. Firewalling failed, for sure.
Re: (Score:2, Funny)
This is why DevOps is a bad idea.
Re: (Score:2)
You want to explain that one?
What the fuck does this have to do with DevOps? And besides, if their DevOps guy can find his ass without a flashlight and a map, he'd have some kind of disaster recovery plan, even if it was just daily scripted snapshots of the server VMs.
Unless of course they were running on bare metal, in which case 2002 called and wants to introduce you to a product called VMware ESX.
Even MST3K knows... (Score:2)
"Keep Circulating the Tapes!"
Re: (Score:3)
No offsite backups? No tapes????
Who designed the disaster plan for these guys?
No, no.. The Admin E-mailed the backups to himself every night.... They are all in his inbox... Don't worry, he encrypted them.
Re: (Score:2)
Re: (Score:2)
It also implies that a rogue employee could have done this at any time.
I ran a large credit union IS department for years and made sure that no one person, even me, could have pulled this off. Various on-line (but in-house at local and remote site) backups done minute to minute in most cases and off-line backups done daily. Various permissions required to access electronic data stores, and different people with physical access. Tapes taken offsite every day AND MOUNTED AND READ AND VERIFIED at the remote si
Re: (Score:2)
My guess, bean-counter type management after ignoring advice from the technical people.
Re: (Score:2)
I don't know if, in this case, that's a fair criticism. VFEmail is providing a realtime service and going offline for any length of time has very serious repercussions.
Tape backups aren't going to have the users' most recent emails and it could take days to fully restore prior emails. From the users' perspective, this is extremely inconvenient and they're probably going to take their recovered emails and go elsewhere.
To run an online service in today's world, particularly an email host, means continual upti
Backups? (Score:3)
Re: (Score:1)
Plot twist: the last remaining copy of the encryption key is backed up on the encrypted backup tapes.
Re: (Score:2)
OH the bitcoins
Re: (Score:3)
Plot twist: the last remaining copy of the encryption key is backed up on the encrypted backup tapes.
Yea, but it's "12345".... What idiot uses THAT as a combination?
Remind me to change the combination on my luggage..
Re: (Score:2)
My other backup missed all my dot files....
pour some out for the lost data...
Re: (Score:2)
:drinks:
On the other hand, don't hire somebody who hasn't realized this mistake exists.
Re: (Score:2)
nice!! (Score:3)
offsite tape backup is sounding good right about now
Re: (Score:2)
offsite tape backup is sounding good right about now
Don't worry, the admin was E-mailing the backups to himself every night for safe keeping...
You mean just the online backup servers... (Score:5, Interesting)
Every file server is lost, every backup server is lost.
So, that's the online backup servers, but what about the offline backups... there were offline backups, right? RIGHT???
I am starting to wonder if I don't need to ask every single electronic service I interact with to put in writing what tighter backup policies are. I imagine my stuff on gmail servers is safe... but that is truly only my imagination, who can say for sure even they have offline backups (that can be restored from)??
Re: (Score:2)
Re: (Score:3)
Re: (Score:3)
Re: (Score:2)
Re:You mean just the online backup servers... (Score:5, Insightful)
Also, depending on how nasty they were being, they might have lurked long enough to poison the offline backups too. People tend to not actually check them till something goes wrong.
AND, when they check, some 70% turn out to be insufficient or not restorable. Most turn out to be nearly useless for anything but giving you a warm fuzzy feeling as you trot them off to offsite storage.
Having a backup plan is one thing, TESTING your backup plan is the next level.... However, revising your backup plan and TESTING your backups are restorable on a regular basis is the only way to know it will work when the chips are down. IF you don't do all this work, it's NOT really backed up, regardless of how many tapes you put into storage.
Re: (Score:2)
That's the grunt work you farm out to the intern.
You DO have interns, right?
Re: (Score:2)
This was how I discovered backups weren't quite working right for a specific piece of closed source software (something like Sharepoint before there was a Sharepoint). The company had gone out of business so there was no support. I was tasked with learning this piece of unsupported software so my first step was to restore the product's proprietary backup file using the instructions provided by the vendor in order to create a dev environment. After working with this environment for a few weeks I realized
Re: (Score:2)
Re: (Score:2)
Unless they are a publicly traded corporation under Sarbanes-Oxley review.
That's one of the things any competent audit will ask for evidence of - working backup restore.
Re: (Score:2)
Also, depending on how nasty they were being, they might have lurked long enough to poison the offline backups too. People tend to not actually check them till something goes wrong.
Perhaps that's when the ransom request shall materialise.
Re: (Score:2)
They probably don't. Not in the sense we think of, e.g. tapes.
Re: (Score:2)
The only thing I would realistically worry about with Google is, what happens with a really big natural disaster that destroys one (or more) entire data centers? I know they replicate a lot, but would the realistically have everything fully replicated spatially...
At least as someone else mentioned about POP email, I do have a local copy of all my email - I would possibly just lose some attachments, but probably nothing I cared much about anyway.
This incident is a good time to consider that issue though, es
Re: (Score:2)
Several years ago Google lost all their online storage for some chunk of users. They talked about restoring from tape, so a least once upon a time they had tape backups.
Re: (Score:2)
>This incident is a good time to consider that issue though, especially as I was thinking about moving to Protonmail...
I use both. I haven't checked my VFEmail account yet however I'd have no hesitation going back to them. I found their service to well put together, I paid for an account mainly because it wasn't scanning every email to serve me ads.
I've found Protonmail to be excellent, straightforward UI which is clean and intuitive. Nothing to complain about either of these services.
I think though if you are not backing up stuff from your email accounts you are missing the point of using these services. If you are r
Re: (Score:3)
The only thing I would realistically worry about with Google is, what happens with a really big natural disaster that destroys one (or more) entire data centers?
AWS.
AWS makes lots of copies of every damn file and scatters them all over the world in geographically different Availability Zones.
So even if the entire us-east-2 (Ohio) AZ is blown off the map by a nuke, AND eu-north-1 (Stockholm) is also blown off the map, along with Tokyo, Sydney, and Frankfurt, your file is still floating around in us-east-1 or eu-west-3 or ap-northeast-3, etc etc, about a dozen other AZs.
You'd need a genuine global disaster to lose files from AWS, and at that point I probably wouldn't
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Off-site backups can be accessed without physical access if it was designed poorly, and there's no reason to assume they had off-line backups...
Re: (Score:2)
Backups are quite often useless and offline backups are usually weeks if not months old and take many hours to restore. Some 70% of "backups" turn out to be broken in some way or another, including not actually backing up the right data, not backing up data that's in a restorable format, and when compressed (as is often done) has unrecoverable bit errors or dropouts that render the whole backup set as good as empty.
Why? Few folks take the time to do backups right, verify they can read the data off the m
IMAP/POP3 provider... (Score:5, Interesting)
Re:IMAP/POP3 provider... (Score:4, Informative)
Re: (Score:2)
Sounds like a cleanup operation (Score:5, Interesting)
Maybe someone needed an email to disappear to avoid public embarrassment or legal trouble.
Re: (Score:2, Insightful)
If you're in a tight enough spot that you need to contact some hackers to annihilate an email company then you also probably don't have the time to wait around while they figure out if they can even get into that email company to do the job.
So:
1. They were already in and held the sword of Damocles over this company's head for a long time without them even knowing it just waiting for someone to fork over enough money to make it worth their while to let the sword fall
or
2. They had help from an insider employe
Re: (Score:2)
Coincident with the Muller cover-up news? This is an easy job for the spooks who hoard zero-days.
Re: (Score:2)
Hillary? Is that you?
Pull not push for backups. (Score:2, Insightful)
Second offsite backup that pulls, not pushes.
- A push backup leaves a trace that there is a backup and to where it is being pushed.
- - Just track the push and wipeout the backup as well.
- A pull backup is only visible from the pulling location and, anyone inside that knows it exists.
- - No trail to trace and wipeout. If it is wiped out, Then it is clearly an inside job.
- - A pulling backup does mean the pulling system has access to the onsite backups.
- - - But the onsite backup can
Re: (Score:2)
Re: (Score:2)
Backup Architecture (Score:1)
Trivial, the right Backup Architecture is to have online backup that is done via something like remote btrfs snapshots (for zfs snapshots), and have those servers be secure. But, this does raise the interesting question, how do you know your appliance is secure? No patches in 20 years, and proven to be correct, with 30% market penetration or more... that might do it.
Frankly, I surprised we don't hear more of this type of total wipe more often. Makes for a great test case for the backup strategies that c
Re: (Score:2)
If you backup data instead of backing up the disks then it shouldn't be that hard to have append-only backups with very limited access permissions.
Then it also is pretty easy to do incremental offline backups of the changed data.
I wonder (Score:2)
I wonder which government officials used them.
Re: (Score:2)
I wonder which government officials used them.
Hillary, of course.
Doesn't sound like "hackers" (Score:2)
This sounds a lot like an internal job, more than external attack. Why risk getting logged on the way in, unless you are a disgruntled employee or competitor. Most likely an employee with unfavorable bonus.
Re: (Score:2)
It could easily happen if their sysadmins suck, everything is put together by hand, and somebody cracked the backup server. The backup server might have access to everything.
Re: (Score:2)
Demonstration attack (Score:2)
Replication != Backup (Score:3, Insightful)
Looks like ZFS replication may have been their backup plan? https://www.vfemail.net/design... [vfemail.net]
Recall an email (Score:1)
No backup - no pity (Score:2)
Seriously, what are these people doing?
Just recover it? (Score:3)
Damn, talk about annoying.
Oh! So they do know where the data ended up. Just restore it! You know, like in the movies?
Re: (Score:2)
So they do know where the data ended up. Just restore it! You know, like in the movies?
"Computer: enhance!"
Just do a restore from Wikileaks. (Score:3)
No backups? Really? (Score:2)
So they have no current backups at all? Seriously?
It's so easy to do these days that there's no good excuse not to. Hell, use a secured AWS bucket and stash your backups there.
So, nobody does backups anymore? (Score:2)
Or, they do backups, but keep all the copies online? For an app connected to the raw internet? And someone thought this was a good idea?
Not a "hacker," A disgruntled insider. (Score:2)
FAQ is a lie! (Score:1)
From the FAQ
> What is your backup strategy / data retention policy?
> VFEmail feels it's important to provide a long-term, stable, environment for our users. In that effort, we perform nightly backups to an offsite host from all on-site and off-site mail storage locations. This backup runs at 12am CST (-0600) and contains all user data.
> 3rd party storage of user data is generally not wanted by privacy-conscious users. If you fall into that category, you will want to use POP3 and download your mail
Slogan : Making email safe for the masses! (Score:2)
It's so safe that now even NSA, FBI, ... cannot have access to it! Nice job!
Re: (Score:2)
It depends on the definition of "criminal".
Not up to Starfleet standards (Score:2)
No secondary backups? Talk about amateurs.