Forgot your password?
typodupeerror
Security Data Storage Technology

Hacker Destroys Avsim.com, Along With Its Backups 780

Posted by timothy
from the giving-you-the-benefit-of-their-bad-childhoods dept.
el americano writes "Flight Simulator community website Avsim has experienced a total data loss after both of their online servers were hacked. The site's founder, Tom Allensworth, explained why 13 years of community developed terrains, skins, and mods will not be restored from backups: 'Some have asked whether or not we had back ups. Yes, we dutifully backed up our servers every day. Unfortunately, we backed up the servers between our two servers. The hacker took out both servers, destroying our ability to use one or the other back up to remedy the situation.'"
This discussion has been archived. No new comments can be posted.

Hacker Destroys Avsim.com, Along With Its Backups

Comments Filter:
  • One word (Score:3, Funny)

    by Anonymous Coward on Friday May 15, 2009 @01:18AM (#27961977)

    Owned.

    • Lies, damn lies. (Score:4, Informative)

      by BrokenHalo (565198) on Friday May 15, 2009 @02:22AM (#27962383)
      The admins' claim that they were backed up is nothing short of an outright lie. A dependency on rsync or any other mirroring technique alone is just plain negligent, when both servers are exposed to the world at large. As a bad analogy, it's like allowing someone to light two fuses with the same match.

      The only way to do backups properly is to have a complete set, offline, in a separate location.

      Sheesh. When will people learn?
      • Re:Lies, damn lies. (Score:5, Informative)

        by Gerzel (240421) * <brollyferret&gmail,com> on Friday May 15, 2009 @03:03AM (#27962591) Journal

        Remember kids if it isn't backed up to an off-line copy then it isn't backed up.

        • by RichardJenkins (1362463) on Friday May 15, 2009 @05:56AM (#27963543)

          Backups: Not hard to get right, just very easy to get wrong.

      • by trawg (308495) on Friday May 15, 2009 @04:23AM (#27963049) Homepage

        I wouldn't call it lies - I'd call it ignorance

  • by Anonymous Coward on Friday May 15, 2009 @01:19AM (#27961983)

    To any sysadmins and DBAs...

    Make sure you have offsite backups

    • by nemesisrocks (1464705) on Friday May 15, 2009 @01:25AM (#27962035) Homepage

      Make sure you have offsite backups

      In this case, even offline (as opposed to offsite) backups would have sufficed.

      Removable hard disks, DVDs -- hell, even tapes. These are all forms of backups that can't be compromised (well, easily) over the internets.

      • Re: (Score:3, Informative)

        From the article

        ... we backed up the servers between our two servers.

        Nope, backing up a server to another online server is not a backup, it's merely another online copy.

        • Re: (Score:3, Insightful)

          by Steffan (126616)

          ... we backed up the servers between our two servers.

          Nope, backing up a server to another online server is not a backup, it's merely another online copy.

          It's the difference between HA [High Availability] and DR [Disaster Recovery].

          Unfortunately, they suffered a disaster, not a 'mere' server failure.

          All that said, my condolences to the server admin / founder, and especially, to all of the contributors. Thirteen years is a lot of data.

          • by darkpixel2k (623900) <aaron@heyaaron.com> on Friday May 15, 2009 @02:19AM (#27962367) Homepage

            Thirteen years is a lot of data.

            Bah--it's not that bad. They actually have crude backups of all their terrain data. They just have to figure out how to restore from 'IRL' format.

        • by Dan541 (1032000) on Friday May 15, 2009 @08:17AM (#27964455) Homepage

          13 years of work lost!

          Suddenly those external hard drives and safe deposit box don't look so expensive.

    • by coryboehne (244614) * on Friday May 15, 2009 @01:30AM (#27962073)

      It's actually very difficult to truly destroy data, especially remotely. There is actually a reason the DoD spec. requires physical destruction of the media.

      Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.

      The article doesn't lead me to believe that he's tried very hard to get this data back.. Maybe somebody (not me) who cares about this resource, should offer an attempt at data recovery.. Just be sure to hurry, before they do something that will ensure you cannot recover the data.

      I've recovered data off of formatted HDD's, off of corrupted file systems, off of compact flash cards and other media (Really useful if you want to keep those photo's that someone thought was deleted, be aware of this people).

      It's amazing how most people seem to think deleted means gone.

      • Re: (Score:3, Interesting)

        Seriously, just load up an undelete program, or file restorer. Do a scan, and recover. This isn't rocket science..
      • Re: (Score:3, Interesting)

        by linzeal (197905)
        Rootkits nowadays come with disk wiping utility [wikia.com].
      • by Khashishi (775369) on Friday May 15, 2009 @01:47AM (#27962197) Journal

        multiple times? I'd like to see you recover something that has been overwritten once.

      • by jamesh (87723) on Friday May 15, 2009 @01:50AM (#27962207)

        Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.

        People keep repeating that mantra to each other, but is it really true? Getting data off a 'formatted' disk is pretty easy as a format rarely does more than write a few sectors at the start of the disk. Getting data off of a disk that has had 'dd if=/dev/random of=/dev/sda' done to it is a different matter altogether.

        There have been papers written about getting some data out of the inter-track space, and scraping it off the noise floor etc with electron microscopes, but as far as I have researched, nobody has actually done it.

        I put it to you that more people have had their kidney's stolen after meeting a pretty girl at a party than there have been disks recovered after being completely overwritten with random data.

        • by short (66530) on Friday May 15, 2009 @03:20AM (#27962689) Homepage

          'dd if=/dev/random of=/dev/sda'

          • Use /dev/urandom as /dev/random will immediately exhaust your kernel entropy pool and hangs to get more (or it is at least unusably slow). urandom is more than enough for this purpose.
          • There are no reports anyone would be even able to restore data after rewriting them with simple /dev/zero. OTOH rewriting by /dev/urandom and /dev/zero costs mostly the same so why to care if /dev/zero is enough.
          • cat /dev/something >/dev/sda is enough/easier on any Linux kernel, dd had to be used on some old commercial Unices nobody has seen for 30 years now.
          • by QuoteMstr (55051) <dan.colascione@gmail.com> on Friday May 15, 2009 @03:39AM (#27962787)

            pv [ivarch.com] < /dev/zero > /dev/device is pretty nifty too.

          • by jamesh (87723) on Friday May 15, 2009 @05:23AM (#27963347)

            There are no reports anyone would be even able to restore data after rewriting them with simple /dev/zero. OTOH rewriting by /dev/urandom and /dev/zero costs mostly the same so why to care if /dev/zero is enough.

            Well, yes. And in fact due to the way data is encoded (MFM, RLL, whatever they use these days) a zero bit of data in a sector does not necessarily correspond to a physical zero bit in a magnetic sense.

            And given that one of the theories about how to recover data is "subtract the 'perfect' waveform of the track from the actual waveform of the track, and the difference will be some indication of the data that was there previously", it doesn't matter if a single pass is random, all 1's, or all 0's. If you were doing multiple passes then random data would be better, but psuedorandom would probably suffice as long as it was different with each rewrite because the objective is to push the variations well under the noise floor.

            cat /dev/something >/dev/sda is enough/easier on any Linux kernel, dd had to be used on some old commercial Unices nobody has seen for 30 years now.

            When I was writing floppies under AIX about 10 years ago, 'dd' with a suitable block size was many times faster than 'cat'. Maybe it wouldn't have made a difference for a harddisk though.

      • bullshit (Score:3, Interesting)

        by QuantumG (50515) *

        Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.

        How about once? With zeros.

        http://16systems.com/zero.php [16systems.com]

        If you can retrieve you data from a drive after it has been dd'd with /dev/zero, you might be able to win this prize.

        If you happen to be in the situation described, chances are you're fucked.

      • Re: (Score:3, Insightful)

        by adamchou (993073)
        Well, besides that, this site sounds like a community where people share UGC. This means that although they might not have it centrally backed up, they still have all the UGC out there, somewhere. I'm sure they can recover a good portion of their original content. The forums will be a bit harder to replace. But all that knowledge is in someone's head.
      • by wumingzi (67100) on Friday May 15, 2009 @04:40AM (#27963157) Homepage Journal

        Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.

        The DoD spec is written as it is for a reason. Given a drive with confidential data on it, an unauthorized person attempting to access the drive does not need to get everything back to pristine condition. Even recovering a small part of the total data set can cause incalculable damage if it's the right small part. The value of sites like Avsim are in the whole rather than the sum of the parts.

        I've recovered data off of formatted HDD's, off of corrupted file systems, off of compact flash cards and other media (Really useful if you want to keep those photo's that someone thought was deleted, be aware of this people).

        There's a large dependency on what you're trying to recover off of. DOS/NTFS are fairly easy to do recovers from. The first character of the filename is zilched out and the rest of the data to find the file is left intact. UNIX/Linux filesystems are a bear. Once you hit "rm", you've lost the ref to your inode. Putting Humpty Dumpty together again at that point becomes nearly impossible because the record which shows where all the pieces are is lost to you. If you have known text from the file, and a good knowledge of how the filesystem works, you should be able to backtrack. Otherwise? God help you.

        There's also an issue of how the data is stored. A single-drive system is fairly straightforward. 2 drives are harder. Once you get into a SAN/NAS where data is spread over multiple drives, recovery of even a single file with known text becomes tricky. Multiple files? Unknown data? The only hope I would see at that point is to put a large segment of the Slashdot community on the problem and tell them a large trove of high-res pictures of Natalie Portman completely nekkid are stored within.

    • by Anonymous Coward on Friday May 15, 2009 @01:44AM (#27962177)

      this really is a pathetic situation. Everybody is hammering these guys for just mirroring their data and saying that they should have had off site backup.........true, they should have. What really is the issue here is that ASSHOLES feel the need to attack for the sake of attacking a site. It would be like me going out and punching random people in the face just because I can.

      We have to stand up for those that cannot stand up for themselves.

      People that destroy just because they can are completely USELESS...............and should be SHOT.

    • by geekmux (1040042) on Friday May 15, 2009 @07:32AM (#27964109)

      To any sysadmins and DBAs...

      Make sure you have offsite backups

      Any person in the IT community who was alive to remember the events of 9/11 should have learned a valuable IT lesson from that event.

      Repeat after me. I will not store my "offsite" backups in the other tower.

  • lesson is (Score:4, Informative)

    by PhrostyMcByte (589271) <phrosty@gmail.com> on Friday May 15, 2009 @01:19AM (#27961985) Homepage
    more than one backup. always! especially if two servers are running the same software, who says they won't both fail at the same time?
  • Three words: (Score:4, Insightful)

    by Girtych (1345935) on Friday May 15, 2009 @01:21AM (#27961997)
    Off. Site. Backups. Textbook example of why you need to secure your backup data in a secure, non-networked location.
  • by Anonymous Coward on Friday May 15, 2009 @01:22AM (#27962005)

    Reserved for people who don't do archival backups, don't secure their systems, and then try to blame their ineptitude on hackers.

    Do backups.
    Do security.
    Do restore from your backups to test them.
    Do not blame others when it's shown you failed steps 1-3.

  • yes we had backups (Score:5, Insightful)

    by frovingslosh (582462) on Friday May 15, 2009 @01:22AM (#27962007)
    They say they had backups, and put them on the Internet where any hacker could get to them, under the same security the originals were stored under. If that's all they cared about their data, I don't see why the Slashdot community should care any more than they did.
  • by OttoM (467655) on Friday May 15, 2009 @01:23AM (#27962009)
    You now will be escorted off-site.
  • Offsite backups? (Score:5, Interesting)

    by Anonymous Coward on Friday May 15, 2009 @01:23AM (#27962021)

    I realize that from quite a few people's perspectives, storing their backups in a separate building constitutes off site storage. I'd almost buy that strategy. Not in the same environment, network, city etc.

    These guys were stupid.

    The day after 9/11 I was in an elevator, and caught a snippet of conversation between 2 people that had business interests with a firm that was in the WTC. The comment I heard was 'their backups were in the other building'. Another company lost.

    You can never totally plan for every contingency, but you can insure yourself. I know many developers that take hard copies of their code (meaning on removable media) home just for this reason. I have seen sys admins do the same because they didn't trust their DR stratagy.

    This was avoidable. This isn't even about disaster recovery. It is about business continuity.

    You can't afford not to protect your data.

    • Re:Offsite backups? (Score:4, Interesting)

      by 4D6963 (933028) on Friday May 15, 2009 @03:53AM (#27962885)
      Yep, watching that show Stephen Fry in America he interview a nuclear bunker dweller who said that after 9/11 he was contacted by several companies to put servers in bunkers as they had lost lots in the towers.
  • by Todd Knarr (15451) on Friday May 15, 2009 @01:24AM (#27962029) Homepage

    Repeat after me: mirroring is not a backup. Backups are physically removed from the machine and stored where they can't be altered until they're needed for a restore. If they aren't removed from the machine, well, as we've just seen that only ends in tears. Observe their pain and learn from it!

  • by MrMista_B (891430) on Friday May 15, 2009 @01:27AM (#27962045)

    'Backed up between two servers'... that's not what a backup is.

    I'm... astonished at the level of incompetence here. A site with 13 years of work like this, and they didn't bother to backup anything at all?

    And now they're trying to handwave it away with 'oh uh, uh really folks, seriously, were really did have backups haha, between servers olol'.

    I don't think 'olol' is going to impress anyone whos work was just wiped out by their incompetence.

  • by AHuxley (892839) on Friday May 15, 2009 @01:31AM (#27962079) Homepage Journal
    When invaded their identities system was lost too.
    All they had was a back up copy that made it out.
    After the war they could go in and find what was tampered with. ie who got a false identity.
    Take your data home with you every night.
  • by Fallen Kell (165468) on Friday May 15, 2009 @01:31AM (#27962083)
    As the subject says. "Online" backups and replication are simply tools to try and minimize downtime. They are NOT a backup solution. They never were and never should be touted as one, just as this example shows. The only good backup is one that occurs frequently, is verified that it worked, and is stored in a secure location such as a fire-proof safe, and even better in two different fire-proof safes in two different locations, preferably more than 100 miles apart.
  • by fishnuts (414425) <fishnuts@arpa.org> on Friday May 15, 2009 @01:32AM (#27962091) Homepage

    Whoever did this must have willfully wanted to destroy the website and its content. Deleting data in this manner is far beyond vandalism or criminal mischief.

    I hope the perps get served by a judge who recognizes just how severely malicious this was, and that enough of the people who used the site can provide the files back to the owners and the community.

  • Some backup stories (Score:5, Interesting)

    by IntentionalStance (1197099) on Friday May 15, 2009 @01:38AM (#27962129)
    I worked for a computer bureaux in the 80's. We upgraded the operating system - very cool, the new release allowed larger files. We didn't, unfortunately, upgrade the backup utility to handle these larger files. Months go by - then there's a problem - whoops backups are useless - Luckily there's a physical audit trail so we we can pay for very large data entry exercise to get our client's data back.

    A couple of years later, I am in the pub with some mates and John turns up. I ask him how he's managed to finish work and get to the pub so early. "I did a fast backup" he said. I was interested so I asked him to explain. "Oh, it's easy, get the target tapes from the rack, rub out the old date, write the new date, put them back into rack and go to the pub"

    Worked for a large software shop in the 90's. I am part of a decent sized Oracle development (circa 50 devs). Ops decides that Oracles backup routines are too slow and 'optimize' them. Some weeks later - guess what - there's a problem and the backups are useless - No physical audit trail this time - the team has to redo all of there work - it was not good for the project budget, the team moral or the client
  • by Chris Tucker (302549) on Friday May 15, 2009 @01:40AM (#27962143) Homepage

    ...the thieves and vandals who steal data and wreck servers.

    THIEVES AND VANDALS.

    Not "hackers".

    What was done was not hacking. It was vandalism. Plain and simple.

    Hackers create. Vandals destroy. Thieves steal.

    I'm surprised that this needs to be explained to the Slashdot community.

  • Real men... (Score:5, Informative)

    by hugetoon (766694) on Friday May 15, 2009 @01:51AM (#27962215)

    "Only wimps use tape backup: _real_ men just upload their important stuff
    on ftp, and let the rest of the world mirror it ;)"
                                                        Linus Torvalds Jul 20 1996, 3:00 am

  • by mlts (1038732) * on Friday May 15, 2009 @01:51AM (#27962227)

    This is a lesson every system administrator worth his or her salt learns over the long haul. You might back up dutifully, test restore, and have a well done system of ensuring backups are rotated correctly. Then you find out the tape drive you use is miscalibrated so only it can read your backup tapes, or you find the backup software you use on a daily basis is not in production, or the latest version has no support for the backlevel formats.

    I have found that in a production environment, you really need multiple methods for backup if at all possible:

    The first level is a dedicated backup server. This machine is locked down to the best of your abilities, and firewalled from the network, only allowing critical ports such as what the backup software uses, and perhaps ssh or RDP (if a Windows box). This machine copies everything from the other servers onto a large disk array, then to tape. The tapes are then cycled offsite via a service like Iron Mountain. Of course, the tapes are encrypted, and corporate officers get a copy of the master keys.

    Why tapes? Because they can be set read only after they are dismounted, and no computer, no matter how infected can modify or delete the tape contents once this is done, outside of a reflash of the tape drive's BIOS. This is important because its not unheard of for someone to write a program that trashes backups over a time interval. Higher end tapes can be used as WORM media like DLT-ICE.

    I can't emphasize enough about securing the backup server, both physically and network-wise. If this box gets compromised, all your data is available. On Windows machines, I recommend using some form of disk encryption (Bitlocker if the machine has a TPM, TrueCrypt, etc) so if the backup server or an array gets physically stolen, the data is of no use to a thief. This is in addition to the backup program's encryption.

    After you have a central backup server installed, secured (security is paramount on this machine unless the backup program client can do encryption), and backups running, you focus on the other levels of backup.

    The next level of backup is on the local servers. Most operating systems have a method of backing up the computer. If you can do this with a server, fire off a snapshot backup every month or so. Most OS backup methods don't have encryption, so this backup should go directly to a tape safe or secured container in the data center. Optionally, you can install backup software locally that can encrypt. I like using the backup/restore utility the OS gives for an image every quarter, then using more secure software more often, so the OS backups can be stored in a tape safe or physically secure container. This way, if the third party backup software ends up inoperable, there is still a method of getting a machine up somehow, or putting it in a virtual machine for recovery purposes.

    Finally, after you have backup servers and a rotation, companies might consider offsite cloud backup services like Mozy. Mozy offers use of keyfiles so all data is stored encrypted (encrypted on the client end). Of course, making sure the encryption key is stored safely is paramount, and the cost of storing a large backup in Mozy's cloud may be prohibitive. However, if worse comes to worst and your site is completely knocked out, as well as the offsite backup site, it may be thing that keeps your business up.

    Of course, scale this up or down as per your company's needs. A smaller business can get by using Mozy and a Windows Server 2008 box running Bitlocker, a network backup program with encryption such as Retrospect or Backup Exec, and using external drives every month to copy backup sets from the main ones to store offsite.

    A larger business might see about a true backup fabric system sold by IBM (TSM), EMC (Networker), or Microsoft's solution.

    The key is to not just have some built in redundancy so if one backup method is not usable, you have another, even if the backups are older, but to be able to do this in a manner that doesn't add too much time and equipment expense.

    • by inKubus (199753) on Friday May 15, 2009 @02:17AM (#27962351) Homepage Journal

      And for those who don't like to pay $10000 for backup software, there's Bacula [bacula.org]. Couple that with an LTO-4 drive (~1000) and LTO-4 tapes (800GB uncompressed, ~60/piece) and you're set. Rsync.net is a decent, cheap online provider for those gaps when you haven't rotated tapes.

      Bacula is pretty sweet because it lets you backup to disk volumes and then you can schedule a roll to tape. So you can just back everything up incrementally to a disk volume and then copy those backups to tape, and then run rsync on the disk volumes to have an offsite, online backup. When recovering, you ask to recover from whatever's available. If you keep enough disk storage around (and there's really no reason not to) you can recover to any date in the past. In the event of a disaster your tapes come into play.

      Now with drives so cheap the temptation is to buy a external hard drive and use that. But tapes have a long history, guaranteed backwards compatibility (planned anyway, LTO drives have to R/W the previous generation and Read 2 generations back), last longer than moving drives, are simpler, lighter, more robust and more portable. Not that I wouldn't keep a external around to dump desktops but tape is the DR standard.

  • by droidsURlooking4 (1543007) on Friday May 15, 2009 @02:05AM (#27962309)
    I kept them in my other pocket.
  • by jeric23 (1154589) on Friday May 15, 2009 @02:16AM (#27962345)

    A public viewing will be available at:

    http://web.archive.org/web/20080116064652/http://www.avsim.com/ [archive.org]

    No date has been set for the funeral.

  • by obarthelemy (160321) on Friday May 15, 2009 @03:11AM (#27962645)

    - tested
    - offline
    - off-site
    - several times

    anything else is "high-availability", not "backup".

  • by IvanTheNotSoBad (977004) on Friday May 15, 2009 @04:12AM (#27962987)
    So they had no real backup strategy....but what happened to them REALLY REALLY sucks. It really irks me seeing so many comments saying these "retards" had it coming to them.

    Listen folks....we're talking about a couple of guys who spent their free time creating a website. They're not making any real money out of this (in fact, they all have regular day jobs).

    They've been advertising for a Tech Manager (non-paid) for quite a quite so time now. They did get one recently...but it turns out the guy harvested the emails from the systems and sent out a bunch of spam. He has since been fired.Even though the avsim folks aren't saying it was him who hacked and destroyed their site, it's quite hard not to think it was him.

    It's been quite a blow to the flightsim community and I have noticed a lot of IT folks are offering help.....I just haven't seen a single one on this thread.
    • by An dochasac (591582) on Friday May 15, 2009 @06:11AM (#27963615)
      Mod parent up. These guys made mistakes, but well paid admins for enormous organizations make these same mistakes. (Bush's email anyone ;-) We should be more interested in informing and helping than in criticizing and 'persecuting(sic)'. When I first started in IT, I brought a hard drive back which contained important data for an Aids research clinic. I suggested that they make sure to do a backup now. I felt for them because the state of the art PC tape backup technology in 1988 was so slow, expensive and prone to eat tapes that I'd have almost suggested swapping out a 2nd MFM drive every day. A few weeks later I got a call, they'd lost their data again and this time there wasn't much I could do. Real men backup their data to slashdot. I hope you don't mind if I use this thread. beegin 665 mydailybackup.uue M27-N)W0@=&AIR!A(&=R96%T(&)A8VMU"$*27-N)W0@=&AIR!A(&=R96%T )(&)A8VMU"$* end

% APL is a natural extension of assembler language programming; ...and is best for educational purposes. -- A. Perlis

Working...