Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!


Forgot your password?
Security Data Storage Technology

Hacker Destroys Avsim.com, Along With Its Backups 780

Posted by timothy
from the giving-you-the-benefit-of-their-bad-childhoods dept.
el americano writes "Flight Simulator community website Avsim has experienced a total data loss after both of their online servers were hacked. The site's founder, Tom Allensworth, explained why 13 years of community developed terrains, skins, and mods will not be restored from backups: 'Some have asked whether or not we had back ups. Yes, we dutifully backed up our servers every day. Unfortunately, we backed up the servers between our two servers. The hacker took out both servers, destroying our ability to use one or the other back up to remedy the situation.'"
This discussion has been archived. No new comments can be posted.

Hacker Destroys Avsim.com, Along With Its Backups

Comments Filter:
  • by Anonymous Coward on Friday May 15, 2009 @12:19AM (#27961983)

    To any sysadmins and DBAs...

    Make sure you have offsite backups

  • lesson is (Score:4, Informative)

    by PhrostyMcByte (589271) <phrosty@gmail.com> on Friday May 15, 2009 @12:19AM (#27961985) Homepage
    more than one backup. always! especially if two servers are running the same software, who says they won't both fail at the same time?
  • by Anonymous Coward on Friday May 15, 2009 @12:22AM (#27962005)

    Reserved for people who don't do archival backups, don't secure their systems, and then try to blame their ineptitude on hackers.

    Do backups.
    Do security.
    Do restore from your backups to test them.
    Do not blame others when it's shown you failed steps 1-3.

  • by nemesisrocks (1464705) on Friday May 15, 2009 @12:25AM (#27962035) Homepage

    Make sure you have offsite backups

    In this case, even offline (as opposed to offsite) backups would have sufficed.

    Removable hard disks, DVDs -- hell, even tapes. These are all forms of backups that can't be compromised (well, easily) over the internets.

  • by coryboehne (244614) * on Friday May 15, 2009 @12:30AM (#27962073)

    It's actually very difficult to truly destroy data, especially remotely. There is actually a reason the DoD spec. requires physical destruction of the media.

    Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.

    The article doesn't lead me to believe that he's tried very hard to get this data back.. Maybe somebody (not me) who cares about this resource, should offer an attempt at data recovery.. Just be sure to hurry, before they do something that will ensure you cannot recover the data.

    I've recovered data off of formatted HDD's, off of corrupted file systems, off of compact flash cards and other media (Really useful if you want to keep those photo's that someone thought was deleted, be aware of this people).

    It's amazing how most people seem to think deleted means gone.

  • by unlametheweak (1102159) on Friday May 15, 2009 @12:45AM (#27962185)

    From the article

    ... we backed up the servers between our two servers.

    Nope, backing up a server to another online server is not a backup, it's merely another online copy.

  • Real men... (Score:5, Informative)

    by hugetoon (766694) on Friday May 15, 2009 @12:51AM (#27962215)

    "Only wimps use tape backup: _real_ men just upload their important stuff
    on ftp, and let the rest of the world mirror it ;)"
                                                        Linus Torvalds Jul 20 1996, 3:00 am

  • by unlametheweak (1102159) on Friday May 15, 2009 @01:05AM (#27962307)

    Which reminds me. They could always use the WayBack Machine to (help in) retrieving their archives:
    http://web.archive.org/web/*/http://www.avsim.com/ [archive.org]

    Google Cache seems to archive only the most recent pages: []

  • by inKubus (199753) on Friday May 15, 2009 @01:17AM (#27962351) Homepage Journal

    And for those who don't like to pay $10000 for backup software, there's Bacula [bacula.org]. Couple that with an LTO-4 drive (~1000) and LTO-4 tapes (800GB uncompressed, ~60/piece) and you're set. Rsync.net is a decent, cheap online provider for those gaps when you haven't rotated tapes.

    Bacula is pretty sweet because it lets you backup to disk volumes and then you can schedule a roll to tape. So you can just back everything up incrementally to a disk volume and then copy those backups to tape, and then run rsync on the disk volumes to have an offsite, online backup. When recovering, you ask to recover from whatever's available. If you keep enough disk storage around (and there's really no reason not to) you can recover to any date in the past. In the event of a disaster your tapes come into play.

    Now with drives so cheap the temptation is to buy a external hard drive and use that. But tapes have a long history, guaranteed backwards compatibility (planned anyway, LTO drives have to R/W the previous generation and Read 2 generations back), last longer than moving drives, are simpler, lighter, more robust and more portable. Not that I wouldn't keep a external around to dump desktops but tape is the DR standard.

  • Lies, damn lies. (Score:4, Informative)

    by BrokenHalo (565198) on Friday May 15, 2009 @01:22AM (#27962383)
    The admins' claim that they were backed up is nothing short of an outright lie. A dependency on rsync or any other mirroring technique alone is just plain negligent, when both servers are exposed to the world at large. As a bad analogy, it's like allowing someone to light two fuses with the same match.

    The only way to do backups properly is to have a complete set, offline, in a separate location.

    Sheesh. When will people learn?
  • by Anonymous Coward on Friday May 15, 2009 @01:40AM (#27962477)

    People always dis tapes. However, enterprise grade tapes are designed from the ground up, chemically, physically, electrically, and mechanically for long term data storage. I say enterprise grade because there is a difference between a tape format like DLT and LTO which was designed from the ground up as a high end data storage medium versus a tape format like the ones which were adapted from video or audiotapes where longetivity takes a back seat to economy.

    I drop a tape, check its spindle, dust it off, its fine. I drop a hard disk, and there is a good chance that all the data on it is history.

    As for Bacula, I am always wary of it. Does it just back up files, or does it back up vital components that are not file related, such as the Registry, ACLs, ADFs, and other things?

  • by VeryLargeNumber (1394367) on Friday May 15, 2009 @01:55AM (#27962557) Homepage

    > I'd like to see you recover something that has been overwritten once.

    You can't do it at home, but professional data recovery service can. Usually you can guess the previous data by precisely measuring the magnetic levels. The old values will influence the resulting intensity. Roughly (I'm not expert!) works like this:

    was -- now -- result
    0 -- 1 -- 0.9
    1 -- 0 -- 0.1
    1 -- 1 -- 1.1
    0 -- 0 -- 0

    That is why you should have MULTIPLE overwrites with RANDOM data.

  • Re:Lies, damn lies. (Score:5, Informative)

    by Gerzel (240421) * <brollyferret@g[ ]l.com ['mai' in gap]> on Friday May 15, 2009 @02:03AM (#27962591) Journal

    Remember kids if it isn't backed up to an off-line copy then it isn't backed up.

  • by obarthelemy (160321) on Friday May 15, 2009 @02:11AM (#27962645)

    - tested
    - offline
    - off-site
    - several times

    anything else is "high-availability", not "backup".

  • by unlametheweak (1102159) on Friday May 15, 2009 @02:12AM (#27962649)

    Unfortunately, the main site content that was lost is the downloadable files, which aren't archived (since they're large.)

    Which is what I suspected (I'm a not Flight-sim enthusiast, so am not familiar with their site, but I presumed there were probably large binaries). They may at least be able to get back a significant part of their forums and text based articles however. It's a start.

  • Re:Lies, damn lies. (Score:3, Informative)

    by Darinbob (1142669) on Friday May 15, 2009 @02:17AM (#27962671)

    There are companies that will do this for you. You make the backups, put them in a lock box, and the company comes around once a week and and picks them up and drops off next week's lock box.

  • Re:Lies, damn lies. (Score:3, Informative)

    by SanityInAnarchy (655584) <ninja@slaphack.com> on Friday May 15, 2009 @02:23AM (#27962709) Journal

    I'm going to respectfully disagree, there.

    A dedicated backup box can be much more hardened than a general-purpose webserver, as the backup box pretty much has a job of storing and retrieving files.

    A solid system of incremental backups helps, too.

    Yes, taking it offline is great. Do that... maybe monthly, if that.

    This scenario sounds much more like someone confused "RAID" with "Backup". RAID (and other high-availability schemes) protects you from hardware failure. Backup protects you from more software failure and human error.

  • by crisco (4669) on Friday May 15, 2009 @02:23AM (#27962711) Homepage
    The [a href="http://16systems.com/zero.php"]Great Zero Challenge[/url] says otherwise. They're simply asking for the filename of one of the files on a drive that has been wiped once with zeros. Despite offering the challenge for over a year and actively speaking to data recovery companies, no one has taken them up on the offer.
  • by crisco (4669) on Friday May 15, 2009 @02:25AM (#27962721) Homepage
    Markup Fail! Great Zero Challenge [16systems.com]
  • by QuoteMstr (55051) <dan.colascione@gmail.com> on Friday May 15, 2009 @02:39AM (#27962787)

    pv [ivarch.com] < /dev/zero > /dev/device is pretty nifty too.

  • Re:Lies, damn lies. (Score:4, Informative)

    by mustafap (452510) on Friday May 15, 2009 @02:43AM (#27962805) Homepage

    >but how many people actually keep off-site backups for home use?

    er, I do. I have a 4GB memory stick that I sync with my back drive on my home PC and a PC at work.

    Once a month I burn a DVD.

  • by IvanTheNotSoBad (977004) on Friday May 15, 2009 @03:12AM (#27962987)
    So they had no real backup strategy....but what happened to them REALLY REALLY sucks. It really irks me seeing so many comments saying these "retards" had it coming to them.

    Listen folks....we're talking about a couple of guys who spent their free time creating a website. They're not making any real money out of this (in fact, they all have regular day jobs).

    They've been advertising for a Tech Manager (non-paid) for quite a quite so time now. They did get one recently...but it turns out the guy harvested the emails from the systems and sent out a bunch of spam. He has since been fired.Even though the avsim folks aren't saying it was him who hacked and destroyed their site, it's quite hard not to think it was him.

    It's been quite a blow to the flightsim community and I have noticed a lot of IT folks are offering help.....I just haven't seen a single one on this thread.
  • Re:Lies, damn lies. (Score:5, Informative)

    by magarity (164372) on Friday May 15, 2009 @03:35AM (#27963119)

    A dedicated backup box can be much more hardened
    What you've described is only marginally better than what these people did. A second server playing backup device, even if it's "much more hardened", whatever that means, is still an extremely lousy and ineffective backup. If lightening hits your building or arson or theft, your "it's hardened"! backup server is just as toasted as the primary. Backups MUST be to removable media that's kept off site and inactive.
    Otherwise you've done practically the same thing for data "backup" as the RAID does via disks, except with two servers.

  • by jamesh (87723) on Friday May 15, 2009 @04:23AM (#27963347)

    There are no reports anyone would be even able to restore data after rewriting them with simple /dev/zero. OTOH rewriting by /dev/urandom and /dev/zero costs mostly the same so why to care if /dev/zero is enough.

    Well, yes. And in fact due to the way data is encoded (MFM, RLL, whatever they use these days) a zero bit of data in a sector does not necessarily correspond to a physical zero bit in a magnetic sense.

    And given that one of the theories about how to recover data is "subtract the 'perfect' waveform of the track from the actual waveform of the track, and the difference will be some indication of the data that was there previously", it doesn't matter if a single pass is random, all 1's, or all 0's. If you were doing multiple passes then random data would be better, but psuedorandom would probably suffice as long as it was different with each rewrite because the objective is to push the variations well under the noise floor.

    cat /dev/something >/dev/sda is enough/easier on any Linux kernel, dd had to be used on some old commercial Unices nobody has seen for 30 years now.

    When I was writing floppies under AIX about 10 years ago, 'dd' with a suitable block size was many times faster than 'cat'. Maybe it wouldn't have made a difference for a harddisk though.

  • by Ginger Unicorn (952287) on Friday May 15, 2009 @04:48AM (#27963507)
    Surely all the people who've downloaded the downloadable content over the years can all band together and restore a large proportion of it?
  • by Kupfernigk (1190345) on Friday May 15, 2009 @04:53AM (#27963527)
    Data recovery was possible, and was not actually that hard, on older drives. The reason was the size of the bits, and the inaccuracy of the tracking servos. As a result, an overwrite would rarely be on exactly the same path as the original data. Mounting the disc in a special drive with precision tracking and more than one head meant that the overwritten data could be read by the leading head, and then used to generate a correction signal which was added (with the correct delay) to the signal coming from the trailing head which was on a different alignment and so was picking up more of the previous signal. We're talking raw signal here, not ones and zeroes.

    Tedious and expensive, but several people made a good living out of doing it (one guy I knew did it as a hobby and made over UKP100K one year.) However, as bits get smaller, servos get more accurate, and tracks get denser, the modus operandi just ceases to exist any more.

    Mind you, for security reasons I always dismantle old drives and bend the disks in half using a lump hammer. That, and the fact that hard drive magnets are just incredibly useful if you have a steel hulled boat and want convenient attachments for e.g. cable ties. They are powerful and very short range, and usually nickel plated. To buy a pair of equally useful magnets from hardware stores costs nearly as much as a drive.

  • by batkiwi (137781) on Friday May 15, 2009 @05:07AM (#27963579)

    Police forces do not recover data from overwritten disks.

    "Formatted" (quick format, destroying partitoin table) yes. Overwritten, no.

  • Re:Offsite backups? (Score:4, Informative)

    by mcvos (645701) on Friday May 15, 2009 @05:07AM (#27963583)

    They should be kept on a different part of the electricity grid, preferably in a differnt postcode.

    It all depends on what kind of disasters you want your data to survive. If you want it to survive nuclear war, you need off-shore backup. Preferably in a neutral country that won't get involved in the war.

    If you want your data to survive a Vogon constructor fleet, use off-planet backup. Recovering it from the brain of a single surviving human (if any) is going to be costly and painful.

  • by Zebedeu (739988) on Friday May 15, 2009 @06:23AM (#27964057)

    It's too late. That battle is over and the word is lost.

    Just like Kleenex (the company) had its trademark stolen from it by falling into common usage, so did the word "hacker" lose its original meaning.

  • Re:Public Viewing (Score:3, Informative)

    by Kirth (183) on Friday May 15, 2009 @06:48AM (#27964223) Homepage

    No, its not. Login/Password required. And Lame explanations why this should be necessary:
    http://web.archive.org/web/20080116064652/http://www.avsim.com/ [archive.org]

    So the content not only got lost because of a stupid backup-strategy, but because of an even dumber login-required-strategy.

    Linus said it: "Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it ;)" And thats precisly what avsim should have done.

  • Re:lesson is (Score:3, Informative)

    by jra (5600) on Friday May 15, 2009 @07:44AM (#27964705)

    No, the *actual* lesson -- and I'm having exactly this same discussion this week in the comments at This Is True, oddly -- that *SPINNING MAGNETIC STORAGE IS NOT A "BACKUP"*.

    If a processor can reach it, it's not a backup.

    If the same fire can consume both the computer and the "backup", it's not a backup.

    DLT or LTO magtape, and move it out of the building, folks.

    I used to be even just the least little bit more generous on this, but given the prices on used DLT-4 drives, not anymore. If you're not backing up on tape at least half an inch wide, you're not backing up, and quit lying to yourself.

    It sucks to be That Guy... but perhaps he'll save hundreds of other sites in his catastrophe...

  • Re:Too Risky (Score:3, Informative)

    by DJRumpy (1345787) on Friday May 15, 2009 @08:01AM (#27964879)
    I didn't say you had to ship off hourly tapes. What hat did you pull that out of? You can use a mirror for minor recovery. We're talking about DR here, not a simple restore of an hourly type data request. The entire site for these folks is gone, not some data set for a transaction 3 hours ago, but everything.

    As to tapes getting lost in transit, that happens very rarely given the tracking techniques in use by folks like FedEx and UPS. Even so, you wouldn't have only a single set of tapes with all of your data on it, you would have an established rotation of data. Every company I have worked for uses this method. Some used daily, some weekly, some monthly, etc, but all shipped tapes off site at regular routines and cycled them out yearly, or every 7 years depending on the type of data and retention requirements.
  • Re:Lies, damn lies. (Score:4, Informative)

    by Ephemeriis (315124) on Friday May 15, 2009 @08:29AM (#27965179)

    I'm going to respectfully disagree, there.

    A dedicated backup box can be much more hardened than a general-purpose webserver, as the backup box pretty much has a job of storing and retrieving files.

    A solid system of incremental backups helps, too.

    Yes, taking it offline is great. Do that... maybe monthly, if that.

    This scenario sounds much more like someone confused "RAID" with "Backup". RAID (and other high-availability schemes) protects you from hardware failure. Backup protects you from more software failure and human error.


    What if your building burns down? What if some minor fire triggers the sprinklers? What if you get struck by lightning? What if an employee goes postal and takes a sledgehammer to all the electronics? What if a tree falls on the power lines and sends a giant surge through your wiring? What if someone breaks in and steals all the computers?

    It isn't a backup unless it leaves the site.

    Of course you could put your live backup box on the other end of some fiber in another state... That's physically off-site... But as long as it is up and running you have to worry about it as well. Hardened or not, it could get hacked. Or it could get a virus. Or some random glitch could corrupt the data on disk. Or its motherboard/HDD/CPU/whatever could die.

    It isn't a backup unless it is offline.

    And then there's the question of whether the thing actually works... You can have all the backups in the world, but if they're all corrupt it won't do you any good. You'll be restoring broken garbage to your replacement server.

    It isn't a backup unless it has been verified.

    What all of this comes down to is some kind of relatively portable media. Tapes, removable HDDs, CDs, DVDs, whatever. You want something that can leave the building on a daily basis. You want pretty much all your media to be out of the building. Bring in just what you need to run today's backup, and then take it out of the building as soon as that is done. Preferably to someplace relatively remote and safe... A safety deposit box is great. Or if someone has a safe at home. Or if you've got a branch-office or something.

  • by funkyjunkman (721687) on Friday May 15, 2009 @08:31AM (#27965203)
    There is no reason for the DoD spec other than paranoia.

    Check out this article from Seagate Recovery Services [actionfront.com]

    It has been suggested that an electron microscope could be used to read and interpret any patterns that were not fully overwritten by the process. Theoretically this can be done - but in practice it is little more than a myth.

    If data could be recovered at the rate of 1 bit per second - this process would take 9,259 days (or over 25 years) to recover 100 MB of information. This is assuming that you could read back and interpret each bit correctly, for example on data that has never been overwritten. If you are trying to read "traces" of data that were previously written there, in the most likely scenario you may be able to correctly recover, interpret and identify 30-40 percent of the signals.


    A "10101011" pattern may come back as "?010?01?" and every single character on the drive would be scrambled in a similar manner. The mathematical probability of decrypting such a puzzle into usable data is infinitesimal.

    It could be claimed that data can be recovered from any drive in the world with a guaranteed success rate of 50% "at the bit level". This sounds interesting until you consider that if you overwrote the entire surface of the drive with either all "0" or all "1" and since the original drive contained nothing but patterns of binary ones and zeros - half the bits would be correct - but obviously no data could be recovered.

    In conclusion, overwritten data cannot be read back or recovered by any current disk drive technology or laboratory technique.
  • by FooRat (182725) on Friday May 15, 2009 @09:12AM (#27965937)

    So if it was a minor natural disaster that destroyed the data, tell me which asshole do you shoot?

    Sorry, but anyone who doesn't properly back up 13 years of data is a bloody idiot, and yes it is their fault, because if you are in charge of that much data, it is your job and responsibility to do proper backups. It doesn't even take a genius to think up a few scary "what if" scenarios, nor does it take more than a few seconds, and it only takes a few minutes of Googling to learn the obvious basics.

    In fact, it is people like this who *purposely* tempt fate who should be held criminally negligent, especially if it's a business.

    An analogy might be a hospital that decides to tempt fate by not having generators. If you go in for some complex surgery, and you die because the power cuts out and there were no backup generators, you would say it's the hospital's fault, regardless of whether the power cut was caused by natural disaster or somebody malicious ... because a hospital should anticipate such things, and, like backups, the cost of anticipating and installing generators is miniscule compared to the disasterous alternative. To throw your hands up in the air and say "oh well, sh-t just happens that we can't control for, and people who damage electricity cables should be shot" is just a third-world mentality ... there's a reason hospitals have generators. The difference between animals and evolved man, is that man is capable of anticipating his potential futures and adapting his environment to mitigate accordingly. Animals sit and wait for bad stuff to happen, and whine about how it "shouldn't have happened" when it does.

  • Re:Lies, damn lies. (Score:3, Informative)

    by EvilBudMan (588716) on Friday May 15, 2009 @09:26AM (#27966161) Journal

    --Even so, most such incidents destroy small businesses completely just because they don't manage to get people back working in time.--

    Been there done that. Speed is very important. All the insurance in the world will not help you if you can't get back up fast enough, but if you do you will have a crew that has a work their ass off mentality for a few years after that and then you will do well. Then every one gets lazy again and something happens to remind them.

  • Re:Lies, damn lies. (Score:2, Informative)

    by hendrikboom (1001110) on Friday May 15, 2009 @09:48AM (#27966579)
    Testing backups is nontrivial art, too. I once created a magnetic-tape backup of critical files, then later in the day went to the trouble of reading the tape. It read just fine. A month later, when I needed it, I tried reading it. It turns out all the blocks had been truncated, apparently because I had forgotten to specify some obscure parameter when writing the tape. Why had it read back correctly the same day? Because the OS had obligingly cached the entire tape contents on disk in case I wanted to mount it again later.
  • Re:Too Risky (Score:3, Informative)

    by turbidostato (878842) on Friday May 15, 2009 @03:44PM (#27972353)

    "The main fault here was that they had fail-over and called it backups."


    "There is no one dogmatic way to look at backups. If you think there is, good luck finding a job in 10 years when conditions have changed."

    Wrong. Conditions have not changed in the last 35 years and I don't see them changing on the foreseable future. Technical conditions and abilities will change, true, but the essence of the work that has to be achieved won't change the same a mathematical theorem doesn't change.

    What a backup strategy is (short version):
    * A means to recover from a failure.
    It's obvious Tom Allensworth's strategy is a failure and it was obvious it was a failure from the very beginnig (it has been a hacker, but what if it were a virus or a worm, or a human failure deleting some critical files and then the deletion being replicated? Same result).

    What makes a minimal backup strategy (any less than this and your "solution" is not entitled to be called "backup strategy"):
    * There has to be no less than two complete data sets non connected with the systems being protected.
    * There has to be no less than one complete data set off-sited from were the systems being protected "live in".
    * There has to be no less than one current copy of the documentation needed to redeploy from barebones the protected systems off-sited from the facilities were the systems being protected "live in".
    * At the very least two people -the backup responsible and her direct superior, have to know where the above mentioned documentation lives and they must have the ability to recover it.

    Some side notes:
    * The last two points are not needed on a lone star-driven system, only on company-style ones. If there's only one person which will benefit from the data (i.e.: your personal data or a single-person bussiness') is good enough if only you know how to recover the data -it can even be only anotated "on you head" and not in paper, although you still would be better if in paper: memory fails with time.
    * The above point-set is not absolutly "failure-proof" and some common sense should be applied (if your system is likely to be attacked, you'd better have more datasets splitted over longer time ranges; if the backup admin and her superior tend to go together there's the risk you lose them both at a time, and so your ability to recover out of -now unknown to exist, documentation, etc.) but they are the bare minimum.
    * Last but not least, backups have exactly ZERO value. Recovering from backups when need arises is the valuable part, so test your recovery procedures, once and again and again. And let it do the test your less knowledgeable/capable/valuable people: maybe when the need arises that will be all you have.

Nothing succeeds like the appearance of success. -- Christopher Lascl