Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×
Security Privacy

Hackers Wipe US Servers of Email Provider VFEmail (zdnet.com) 157

Posted by msmash from the attack-and-destroy dept.
Hackers have breached the severs of email provider VFEmail.net and wiped the data from all its US servers, destroying all US customers' data in the process. From a report: The attack took place yesterday, February 11, and was detected after the company's site and webmail client went down without notice. "At this time, the attacker has formatted all the disks on every server," the company said yesterday. "Every VM is lost. Every file server is lost, every backup server is lost. This was more than a multi-password via SSH exploit, and there was no ransom. Just attack and destroy," VFEmail said. The company's staff is now working to recover user emails, but as things stand right now, all data for US customers appears to have been deleted for good and gone into /dev/null.
This discussion has been archived. No new comments can be posted.

Hackers Wipe US Servers of Email Provider VFEmail More

Hackers Wipe US Servers of Email Provider VFEmail

Comments Filter:

  • There were NO offsite backups????? (Score:5, Insightful)

    by sconeu ( 64226 ) on Tuesday February 12, 2019 @03:54PM (#58111534) Homepage Journal

    No offsite backups? No tapes????

    Who designed the disaster plan for these guys?

    • No offsite backups? No tapes????

      Who designed the disaster plan for these guys?

      Same geniuses as Wells Fargo?

    • Re: (Score:3)

      by spudnic ( 32107 )

      It's all in their private cloud, of course!

    • You mean offline.

      Nothing happened to any particular location.

      • Re:There were NO offsite backups????? (Score:4, Insightful)

        by rickb928 ( 945187 ) on Tuesday February 12, 2019 @05:26PM (#58112076) Homepage Journal

        It *is* a PITA to put a tape in your bag, open up the fireproof safe at home, throw it in, get the *correct* one out, put it in your bag, and remember the next day to put that where it needs to be. And repeat. /s

        I did that for years. And I slept a little better.

    • Re: (Score:3)

      by lgw ( 121541 )

      No offsite backups? No tapes????

      Who designed the disaster plan for these guys?

      The plan was a disaster - mission complete!

      An online copy is not a backup, guys. It can be a great cache of a backup, but it's not a backup. Who still doesn't know this?

      • Re: (Score:3)

        by pnutjam ( 523990 )
        At the very least, you should be using a 2nd cloud service for backup, like rsync.net or those guys that are always releasing hard drive stats, backblaze.

    • No backup can be a feature (Score:5, Interesting)

      by b0s0z0ku ( 752509 ) on Tuesday February 12, 2019 @04:03PM (#58111630)

      That can be both a bug and a feature. No backups mean that there's no cache of deleted emails. Some users may want the ability to truly delete data, not have it able to "appear" due to legal proceedings 5 years from now.

      I'd say it's on the users to back up their email using a client that locally caches IMAP folders or downloads via POP3.

      • Re:No backup can be a feature (Score:4, Insightful)

        by Aighearach ( 97333 ) on Tuesday February 12, 2019 @05:09PM (#58111976)

        It would seem more practical to just limit the stored backups to the last n copies, like you do with rotated log files.

        If it can only come back for two weeks or something, that is sufficient for most use cases.

      • Re: (Score:1)

        by ljw1004 ( 764174 )

        That can be both a bug and a feature. No backups mean that there's no cache of deleted emails. Some users may want the ability to truly delete data, not have it able to "appear" due to legal proceedings 5 years from now. I'd say it's on the users to back up their email using a client that locally caches IMAP folders or downloads via POP3.

        I used to do that, starting in 1993. But I've used so many different computers since then, so many different email clients. My archive got too big to fit conveniently on my computer's storage. So then I was stuck with a load of separate volumes of backups that were hard to search. I wrote software to merge volumes of archives when I upgraded to bigger disks, also to export them into other formats.

        In the end, it was too much work for an inadequate solution. Now I just pay $8/month for an Exchange365 account.

        • I don't really back up my mail. I copy one mailbox to two others via IMAP and POP3. Another mailbox I copy via IMAP. And then I also have a copy in my beloved (/s) Microsoft Mail. It's not really a backup, I think, just copies. And I have a glorious spam library dating back to the 90s in some of it, just too lazy to clean it up.

          If you remember spam from the 90s, you know why some of it I've had to delete.

    • Re: (Score:3, Interesting)

      by Anonymous Coward

      The business plan probably.

      If you do make backups, you are too expensive, certainly cannot compete, and will go out of business. No income for you.

      If you do not make backups, you may make a nice buck for a while before the thing explodes in your face. Hell, maybe you are lucky and it never explodes at all.
      Regardless, at least you will make money for a while. So this scenario is clearly the winner. Screw the damage to your future ex customers, that is not your problem.

    • A more important question is "Why were the backup servers accessible from the email servers?"

      A good network design has the backup servers isolated from the production servers. Only the ports need for backup should be allowed. Even if using a copy over SSH, it is possible to set it so the backup servers can access production, but block all access from production to the backup servers. I should never be able to gain console/terminal access on the backup servers from production.

      • It appears that the backup servers did not share any authentication and were accessed and destroyed separately.

      • Re:There were NO offsite backups????? (Score:4, Insightful)

        by rickb928 ( 945187 ) on Tuesday February 12, 2019 @05:41PM (#58112136) Homepage Journal

        Once you're in the front door, you're going through the system. Only offline backups can be trusted to 'be there'.

        And no offline copies of the VM environment? I think of those as especially precious. DO I want to rebuild those from scratch? Nope.

        • And no offline copies of the VM environment? I think of those as especially precious. DO I want to rebuild those from scratch? Nope.

          They probably didn't build them to begin with, odds are they did it all with someone else's containers and they had no clue what was actually running on those systems.

          • They probably didn't build them to begin with, odds are they did it all with someone else's containers and they had no clue what was actually running on those systems.

            ^^^^^THIS.

            Yep, they most likely took some base container, maybe modded it a bit, and threw it into service. They likely have no idea at all how it was configured or what was in it.

            People are always concerned with the data and forget about the infrastructure that it lives in.

    • Re: (Score:2, Funny)

      by Anonymous Coward

      This is why DevOps is a bad idea.

      • You want to explain that one?

        What the fuck does this have to do with DevOps? And besides, if their DevOps guy can find his ass without a flashlight and a map, he'd have some kind of disaster recovery plan, even if it was just daily scripted snapshots of the server VMs.

        Unless of course they were running on bare metal, in which case 2002 called and wants to introduce you to a product called VMware ESX.

    • "Keep Circulating the Tapes!"

    • No offsite backups? No tapes????

      Who designed the disaster plan for these guys?

      No, no.. The Admin E-mailed the backups to himself every night.... They are all in his inbox... Don't worry, he encrypted them.

    • What? Me worry? [bing.com]

    • Re: (Score:2)

      by brausch ( 51013 )

      It also implies that a rogue employee could have done this at any time.

      I ran a large credit union IS department for years and made sure that no one person, even me, could have pulled this off. Various on-line (but in-house at local and remote site) backups done minute to minute in most cases and off-line backups done daily. Various permissions required to access electronic data stores, and different people with physical access. Tapes taken offsite every day AND MOUNTED AND READ AND VERIFIED at the remote si

    • Re: (Score:2)

      by rossz ( 67331 )

      My guess, bean-counter type management after ignoring advice from the technical people.

    • Re: (Score:2)

      by bartle ( 447377 )

      I don't know if, in this case, that's a fair criticism. VFEmail is providing a realtime service and going offline for any length of time has very serious repercussions.

      Tape backups aren't going to have the users' most recent emails and it could take days to fully restore prior emails. From the users' perspective, this is extremely inconvenient and they're probably going to take their recovered emails and go elsewhere.

      To run an online service in today's world, particularly an email host, means continual upti

  • Backups? (Score:3)

    by byteherder ( 722785 ) on Tuesday February 12, 2019 @03:55PM (#58111540)
    Time to pull yesterday's backup tapes. You do have the tapes from yesterday, don't you?

    • Re: (Score:1)

      by Anonymous Coward

      Plot twist: the last remaining copy of the encryption key is backed up on the encrypted backup tapes.

      • Re: (Score:2)

        by zlives ( 2009072 )

        OH the bitcoins

      • Plot twist: the last remaining copy of the encryption key is backed up on the encrypted backup tapes.

        Yea, but it's "12345".... What idiot uses THAT as a combination?

        Remind me to change the combination on my luggage..

      • Re: (Score:2)

        by pnutjam ( 523990 )
        I backed up my work laptop with borg and misunderstood the password protected a key-file, not the archive....

        My other backup missed all my dot files....

        pour some out for the lost data...

  • nice!! (Score:3)

    by zlives ( 2009072 ) on Tuesday February 12, 2019 @03:55PM (#58111544)

    offsite tape backup is sounding good right about now

    • offsite tape backup is sounding good right about now

      Don't worry, the admin was E-mailing the backups to himself every night for safe keeping...

  • You mean just the online backup servers... (Score:5, Interesting)

    by SuperKendall ( 25149 ) on Tuesday February 12, 2019 @03:56PM (#58111548)

    Every file server is lost, every backup server is lost.

    So, that's the online backup servers, but what about the offline backups... there were offline backups, right? RIGHT???

    I am starting to wonder if I don't need to ask every single electronic service I interact with to put in writing what tighter backup policies are. I imagine my stuff on gmail servers is safe... but that is truly only my imagination, who can say for sure even they have offline backups (that can be restored from)??

    • Re: (Score:2)

      by jythie ( 914043 )
      Sounds like they are trying to restore data from something, so hopefully they had offline backups of some type.

    • Re: (Score:3)

      by jythie ( 914043 )
      Also, depending on how nasty they were being, they might have lurked long enough to poison the offline backups too. People tend to not actually check them till something goes wrong.
      • Also, they may only keep backups for a few days for security reasons -- i.e. they want their users to be able to "truly delete" data.

      • Re:You mean just the online backup servers... (Score:5, Insightful)

        by bobbied ( 2522392 ) on Tuesday February 12, 2019 @04:31PM (#58111760)

        Also, depending on how nasty they were being, they might have lurked long enough to poison the offline backups too. People tend to not actually check them till something goes wrong.

        AND, when they check, some 70% turn out to be insufficient or not restorable. Most turn out to be nearly useless for anything but giving you a warm fuzzy feeling as you trot them off to offsite storage.

        Having a backup plan is one thing, TESTING your backup plan is the next level.... However, revising your backup plan and TESTING your backups are restorable on a regular basis is the only way to know it will work when the chips are down. IF you don't do all this work, it's NOT really backed up, regardless of how many tapes you put into storage.

        • That's the grunt work you farm out to the intern.

          You DO have interns, right?

        • This was how I discovered backups weren't quite working right for a specific piece of closed source software (something like Sharepoint before there was a Sharepoint). The company had gone out of business so there was no support. I was tasked with learning this piece of unsupported software so my first step was to restore the product's proprietary backup file using the instructions provided by the vendor in order to create a dev environment. After working with this environment for a few weeks I realized

        • The backup for some of the VoIP servers I've seen at companies was actually an identical server with the exact same hardware. It would be kept up to date by restoring the backup of the operational server onto it. This served as both a backup in case of hardware failure, and a sanity check to confirm the software backups were working and restorable,

      • Unless they are a publicly traded corporation under Sarbanes-Oxley review.

        That's one of the things any competent audit will ask for evidence of - working backup restore.

      • Also, depending on how nasty they were being, they might have lurked long enough to poison the offline backups too. People tend to not actually check them till something goes wrong.

        Perhaps that's when the ransom request shall materialise.

    • They probably don't. Not in the sense we think of, e.g. tapes.

  • IMAP/POP3 provider... (Score:5, Interesting)

    by b0s0z0ku ( 752509 ) on Tuesday February 12, 2019 @04:00PM (#58111594)
    Thankfully, VFEmail was primarily an IMAP/POP3 provider. I suspect that the majority of its users had a local backup in the form of an email client with a local store...

    • Re:IMAP/POP3 provider... (Score:4, Informative)

      by chiefcrash ( 1315009 ) on Tuesday February 12, 2019 @04:36PM (#58111786)
      Which, hopefully they've been paying attention: the current state of recovery means if you reconnect your client to your new mailbox, all your local mail will be lost (according to an update on their website)
      • That's generally not true; most mail clients flag mail as deleted in the local archive. You have to run a "compact" on the archive in order to totally delete "deleted" messages.

  • Sounds like a cleanup operation (Score:5, Interesting)

    by misnohmer ( 1636461 ) on Tuesday February 12, 2019 @04:09PM (#58111654)

    Maybe someone needed an email to disappear to avoid public embarrassment or legal trouble.

    • Re: (Score:2, Insightful)

      by Anonymous Coward

      If you're in a tight enough spot that you need to contact some hackers to annihilate an email company then you also probably don't have the time to wait around while they figure out if they can even get into that email company to do the job.

      So:
      1. They were already in and held the sword of Damocles over this company's head for a long time without them even knowing it just waiting for someone to fork over enough money to make it worth their while to let the sword fall
      or
      2. They had help from an insider employe

    • Hillary? Is that you?

  • First onsite backup
    Second offsite backup that pulls, not pushes.
    - A push backup leaves a trace that there is a backup and to where it is being pushed.
    - - Just track the push and wipeout the backup as well.
    - A pull backup is only visible from the pulling location and, anyone inside that knows it exists.
    - - No trail to trace and wipeout. If it is wiped out, Then it is clearly an inside job.
    - - A pulling backup does mean the pulling system has access to the onsite backups.
    - - - But the onsite backup can

    • Re: (Score:2)

      by pnutjam ( 523990 )
      Unless your cleaning your logs, you'll see a pull backup authenticating to the system. This also means you have to trust your backup endpoint, I avoid pull backups for this reason, but a proper backup account shouldn't have access to trash your backup destination.
    • Not a terrible strategy. Mine (for a personal system with around 500GB of important data) is to rotate four backup devices - external HDDs, not tapes - daily (kept on my person), weekly (to office), monthly and yearly (both to safe deposit box near me). I also have copies of the most important files (kids' pics and videos mainly) with my in-laws in Europe. The backups are encrypted before being put on the external drives. I'm aware of a few flaws that I am working to address. (a) I'd have limited defen

  • Backup Architecture (Score:1)

    by Anonymous Coward

    Trivial, the right Backup Architecture is to have online backup that is done via something like remote btrfs snapshots (for zfs snapshots), and have those servers be secure. But, this does raise the interesting question, how do you know your appliance is secure? No patches in 20 years, and proven to be correct, with 30% market penetration or more... that might do it.

    Frankly, I surprised we don't hear more of this type of total wipe more often. Makes for a great test case for the backup strategies that c

    • If you backup data instead of backing up the disks then it shouldn't be that hard to have append-only backups with very limited access permissions.

      Then it also is pretty easy to do incremental offline backups of the changed data.

  • I wonder which government officials used them.

  • This sounds a lot like an internal job, more than external attack. Why risk getting logged on the way in, unless you are a disgruntled employee or competitor. Most likely an employee with unfavorable bonus.

    • It could easily happen if their sysadmins suck, everything is put together by hand, and somebody cracked the backup server. The backup server might have access to everything.

    • Re: (Score:2)

      by pnutjam ( 523990 )
      pen-tester screw-up...? oops
  • Sounds like some hacker(s) needed to demonstrate their operational efficacy to potential clients. Either that or just some too-edgy vandal wanted to burn something to the ground. Small probability: someone needed something specific wiped and needed there to be no fingerprints left behind.

  • Replication != Backup (Score:3, Insightful)

    by bodog ( 231448 ) on Tuesday February 12, 2019 @04:47PM (#58111844)

    Looks like ZFS replication may have been their backup plan? https://www.vfemail.net/design... [vfemail.net]

  • That's a terrible way to recall an email.

  • Seriously, what are these people doing?

  • Just recover it? (Score:3)

    by DontBeAMoran ( 4843879 ) on Tuesday February 12, 2019 @05:23PM (#58112056)

    ...as things stand right now, all data for US customers appears to have been deleted for good...

    Damn, talk about annoying.

    ...and gone into /dev/null.

    Oh! So they do know where the data ended up. Just restore it! You know, like in the movies?

  • Just do a restore from Wikileaks. (Score:3)

    by jfdavis668 ( 1414919 ) on Tuesday February 12, 2019 @07:02PM (#58112526)
    I'm sure they have a recent copy.

  • So they have no current backups at all? Seriously?

    It's so easy to do these days that there's no good excuse not to. Hell, use a secured AWS bucket and stash your backups there.

  • Or, they do backups, but keep all the copies online? For an app connected to the raw internet? And someone thought this was a good idea?

  • This reeks of internal job. Complete and total devastation with no apparent purpose? Its too comprehensive to be an advanced script kiddie or random attack and therefor also too good to be anything without purpose. But there is no apparent purpose, so it must be an inside job. The offline tapes were probably deleted too, and that requires very skillful cracking indeed!

  • FAQ is a lie! (Score:1)

    by Anonymous Coward

    From the FAQ

    > What is your backup strategy / data retention policy?
    > VFEmail feels it's important to provide a long-term, stable, environment for our users. In that effort, we perform nightly backups to an offsite host from all on-site and off-site mail storage locations. This backup runs at 12am CST (-0600) and contains all user data.
    > 3rd party storage of user data is generally not wanted by privacy-conscious users. If you fall into that category, you will want to use POP3 and download your mail

  • It's so safe that now even NSA, FBI, ... cannot have access to it! Nice job!

Slashdot Top Deals

Per buck you get more computing action with the small computer. -- R.W. Hamming

Close