Slashdot Log In
Hacker Destroys Avsim.com, Along With Its Backups
Posted by
timothy
on Fri May 15, 2009 12:17 AM
from the giving-you-the-benefit-of-their-bad-childhoods dept.
from the giving-you-the-benefit-of-their-bad-childhoods dept.
el americano writes "Flight Simulator community website Avsim has experienced a total data loss after both of their online servers were hacked. The site's founder, Tom Allensworth, explained why 13 years of community developed terrains, skins, and mods will not be restored from backups: 'Some have asked whether or not we had back ups. Yes, we dutifully backed up our servers every day. Unfortunately, we backed up the servers between our two servers. The hacker took out both servers, destroying our ability to use one or the other back up to remedy the situation.'"
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
This should be a lesson... (Score:5, Informative)
To any sysadmins and DBAs...
Make sure you have offsite backups
Re:This should be a lesson... (Score:5, Informative)
Make sure you have offsite backups
In this case, even offline (as opposed to offsite) backups would have sufficed.
Removable hard disks, DVDs -- hell, even tapes. These are all forms of backups that can't be compromised (well, easily) over the internets.
Parent
Re:This should be a lesson... (Score:5, Funny)
Thirteen years is a lot of data.
Bah--it's not that bad. They actually have crude backups of all their terrain data. They just have to figure out how to restore from 'IRL' format.
Parent
Re:This should be a lesson... (Score:5, Funny)
Why would you put that on your resume?
Parent
Re:This should be a lesson... (Score:5, Interesting)
Oh, he gets my sympathies. I've had cases where I was specifically told "that is a scratch server: do not back it up, no one is supposed to keep real data on it". And when it crashed, my employers were very fortunate indeed that I'd completely ignored this and quietly been backing it up with my reserve, emergency tape drive, partly to make sure it kept working, partly to test out new backup tools, and partly because I knew staff would ignore this and use the big lump of spare storage for convenient archival space. My employer was actually angry at me for doing so, but the QA department was very, very, very grateful indeed.
The lesson is more subtle than some of us might realize, though. Never rely on a _single_ method of backup or data storage, because any factor that ruins that backup can ruin all copies of it. This is true for backup systems that use proprietary format, or a failed tape drive that's been screwing up backups for the last year (which I've seen happen with old mag-tape media). And I _love_ online backups: you can make the data accessible via NFS or CIFS or other file-sharing to people to recover the files they just accidentally deleted, without having to invest in a very, very expensive NetApp or similar file server. But oh, dear, I've also seen what happens when someone screws up the backup tools and deletes all the copies at the same time.
Parent
Re:This should be a lesson... (Score:5, Informative)
It's actually very difficult to truly destroy data, especially remotely. There is actually a reason the DoD spec. requires physical destruction of the media.
Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.
The article doesn't lead me to believe that he's tried very hard to get this data back.. Maybe somebody (not me) who cares about this resource, should offer an attempt at data recovery.. Just be sure to hurry, before they do something that will ensure you cannot recover the data.
I've recovered data off of formatted HDD's, off of corrupted file systems, off of compact flash cards and other media (Really useful if you want to keep those photo's that someone thought was deleted, be aware of this people).
It's amazing how most people seem to think deleted means gone.
Parent
Re:This should be a lesson... (Score:5, Insightful)
multiple times? I'd like to see you recover something that has been overwritten once.
Parent
Re:overwritten once CAN be recovered (Score:5, Informative)
Parent
Re:overwritten once CAN be recovered (Score:5, Insightful)
I've never seen *any* evidence or heard of *any* occasion that such a recovery, even from a only-once-zeroed drive was done.
Now the point is, one could say "of cooourse not, guys that can do this won't do it for peanuts, besides they're secres service" etc etc. But the point is: even if it's secret service and really expensive, at leas *some* news about it should have hit the public -- after all, this myth has been around for several years (a decade?) now.
I'd still even like to hear from a success story. Or even find a company that advertizes "We can (partly?) recover your zero'ed data -- it's going to cost a fortune, an arm and a leg, but we can." Haven't seen that one either yet. Not a commercial, not an offer, nothing... besides legends.
Parent
Re:overwritten once CAN be recovered (Score:5, Insightful)
I am a computer forensics expert. I search for deleted data for a living, and I testify in court as to what can be done.
Unfortunately you are wrong about recovering data that has been overwritten by using magnetic magic.
That is an urban legend that has been disproven. Maybe 20 years ago using low density MFM drives it was theoretically possible, but now it is not. Maybe the NSA has some tech they reversed engineered from an Area-51 UFO to do this, but I've never seen or heard of it.
Even Gutmann has recanted his 38 wipes recommendation.
Now don't mistake overwritten data for deleted data. When data is deleted it is NOT overwritten. When a hard drive is re-formatted almost nothing is over-written. When a file is overwritten with zeros or random bytes there are probably 10 more copies of that file and previous versions of that file floating around in unallocated sectors, swap space, file slack, hibernation files, etc.
But what IS overwritten is gone.
Parent
You are correct about drive age (Score:5, Informative)
Tedious and expensive, but several people made a good living out of doing it (one guy I knew did it as a hobby and made over UKP100K one year.) However, as bits get smaller, servos get more accurate, and tracks get denser, the modus operandi just ceases to exist any more.
Mind you, for security reasons I always dismantle old drives and bend the disks in half using a lump hammer. That, and the fact that hard drive magnets are just incredibly useful if you have a steel hulled boat and want convenient attachments for e.g. cable ties. They are powerful and very short range, and usually nickel plated. To buy a pair of equally useful magnets from hardware stores costs nearly as much as a drive.
Parent
Re:This should be a lesson... (Score:5, Insightful)
People keep repeating that mantra to each other, but is it really true? Getting data off a 'formatted' disk is pretty easy as a format rarely does more than write a few sectors at the start of the disk. Getting data off of a disk that has had 'dd if=/dev/random of=/dev/sda' done to it is a different matter altogether.
There have been papers written about getting some data out of the inter-track space, and scraping it off the noise floor etc with electron microscopes, but as far as I have researched, nobody has actually done it.
I put it to you that more people have had their kidney's stolen after meeting a pretty girl at a party than there have been disks recovered after being completely overwritten with random data.
Parent
Re:This should be a lesson... (Score:5, Interesting)
Parent
Re:This should be a lesson... (Score:5, Informative)
pv [ivarch.com] < /dev/zero > /dev/device is pretty nifty too.
Parent
Re:This should be a lesson... (Score:5, Interesting)
Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.
The DoD spec is written as it is for a reason. Given a drive with confidential data on it, an unauthorized person attempting to access the drive does not need to get everything back to pristine condition. Even recovering a small part of the total data set can cause incalculable damage if it's the right small part. The value of sites like Avsim are in the whole rather than the sum of the parts.
I've recovered data off of formatted HDD's, off of corrupted file systems, off of compact flash cards and other media (Really useful if you want to keep those photo's that someone thought was deleted, be aware of this people).
There's a large dependency on what you're trying to recover off of. DOS/NTFS are fairly easy to do recovers from. The first character of the filename is zilched out and the rest of the data to find the file is left intact. UNIX/Linux filesystems are a bear. Once you hit "rm", you've lost the ref to your inode. Putting Humpty Dumpty together again at that point becomes nearly impossible because the record which shows where all the pieces are is lost to you. If you have known text from the file, and a good knowledge of how the filesystem works, you should be able to backtrack. Otherwise? God help you.
There's also an issue of how the data is stored. A single-drive system is fairly straightforward. 2 drives are harder. Once you get into a SAN/NAS where data is spread over multiple drives, recovery of even a single file with known text becomes tricky. Multiple files? Unknown data? The only hope I would see at that point is to put a large segment of the Slashdot community on the problem and tell them a large trove of high-res pictures of Natalie Portman completely nekkid are stored within.
Parent
Re:This should be a lesson... (Score:5, Funny)
I hear it's murder. ;)
Parent
Re:This should be a lesson... (Score:5, Insightful)
this really is a pathetic situation. Everybody is hammering these guys for just mirroring their data and saying that they should have had off site backup.........true, they should have. What really is the issue here is that ASSHOLES feel the need to attack for the sake of attacking a site. It would be like me going out and punching random people in the face just because I can.
We have to stand up for those that cannot stand up for themselves.
People that destroy just because they can are completely USELESS...............and should be SHOT.
Parent
Re:This should be a lesson... (Score:5, Insightful)
How about we just shoot all hackers?
I'm not sure how that will protect against data loss from equipment failure, natural disaster, fire, software failure, solar flares, Secret Service, or really anything other than hackers.
Offsite, offline backups aren't a good idea solely to protect against hackers. They're a good idea to protect against data loss in general.
Parent
Re:This should be a lesson... (Score:5, Funny)
How about we start shooting people who can't recognize jokes. Sheesh.
Then who would mod for slashdot?
Parent
Re:This should be a lesson... (Score:5, Insightful)
What, you mean like this guy? [archive.org] You probably wouldn't even have the browser you're using right now if it weren't for that particular, uh. hacker.
Parent
Re:This should be a lesson... (Score:5, Informative)
Which reminds me. They could always use the WayBack Machine to (help in) retrieving their archives:
http://web.archive.org/web/*/http://www.avsim.com/ [archive.org]
Google Cache seems to archive only the most recent pages:
http://74.125.95.132/search?q=cache%3Ahttp%3A%2F%2Fwww.avsim.com%2F&submit2=Google [74.125.95.132]
Parent
Re:This should be a lesson... (Score:5, Insightful)
In fact, it seems so utterly stupid that I get furious every time I hear some thoughtless moron spout "Punish the hackers".
A little blame needs to come from all areas. Not every website or messageboard is run by someone with a CS degree with a minor in website security. A break-in of a government site or large corporate site is one thing, a family website another. This site is probably somewhere in between.
Saying it isn't the hackers fault that improper mehtods were used to secure a site is like saying it isn't the muggers fault that the lady's handbag was so easy to steal.
Parent
Re:This should be a lesson... (Score:5, Funny)
So, when rats attack your cellar, you pick the most intelligent and ask for advice?
Yes, he said don't worry about it and go back to posting on Slashdot.
Parent
Re:This should be a lesson... (Score:5, Funny)
Oh, we can make a good car analogy out of this: Having a backup car in case your primary car crashes is a great idea (if you can afford it). Except that instead of keeping their backup car locked in the garage, these people attached their backup car to their primary car with a tow bar and dragged it around everywhere they went. When the primary car crashed, the backup ran into it a fraction of a second later. Now they're sad that their backup car is dead too and are somehow suprised they don't have anything to drive.
Parent
yes we had backups (Score:5, Insightful)
Re:yes we had backups (Score:5, Funny)
Wait, we have to care? I thought we were supposed to point and laugh...
Parent
Offsite backups? (Score:5, Interesting)
I realize that from quite a few people's perspectives, storing their backups in a separate building constitutes off site storage. I'd almost buy that strategy. Not in the same environment, network, city etc.
These guys were stupid.
The day after 9/11 I was in an elevator, and caught a snippet of conversation between 2 people that had business interests with a firm that was in the WTC. The comment I heard was 'their backups were in the other building'. Another company lost.
You can never totally plan for every contingency, but you can insure yourself. I know many developers that take hard copies of their code (meaning on removable media) home just for this reason. I have seen sys admins do the same because they didn't trust their DR stratagy.
This was avoidable. This isn't even about disaster recovery. It is about business continuity.
You can't afford not to protect your data.
Sigh. Mirror != backup (Score:5, Insightful)
Repeat after me: mirroring is not a backup. Backups are physically removed from the machine and stored where they can't be altered until they're needed for a restore. If they aren't removed from the machine, well, as we've just seen that only ends in tears. Observe their pain and learn from it!
Some backup stories (Score:5, Interesting)
A couple of years later, I am in the pub with some mates and John turns up. I ask him how he's managed to finish work and get to the pub so early. "I did a fast backup" he said. I was interested so I asked him to explain. "Oh, it's easy, get the target tapes from the rack, rub out the old date, write the new date, put them back into rack and go to the pub"
Worked for a large software shop in the 90's. I am part of a decent sized Oracle development (circa 50 devs). Ops decides that Oracles backup routines are too slow and 'optimize' them. Some weeks later - guess what - there's a problem and the backups are useless - No physical audit trail this time - the team has to redo all of there work - it was not good for the project budget, the team moral or the client
Real men... (Score:5, Informative)
"Only wimps use tape backup: _real_ men just upload their important stuff ;)"
on ftp, and let the rest of the world mirror it
Linus Torvalds Jul 20 1996, 3:00 am
Re:Real men... (Score:5, Interesting)
Parent
Hindsight is always 20/20 (Score:5, Insightful)
This is a lesson every system administrator worth his or her salt learns over the long haul. You might back up dutifully, test restore, and have a well done system of ensuring backups are rotated correctly. Then you find out the tape drive you use is miscalibrated so only it can read your backup tapes, or you find the backup software you use on a daily basis is not in production, or the latest version has no support for the backlevel formats.
I have found that in a production environment, you really need multiple methods for backup if at all possible:
The first level is a dedicated backup server. This machine is locked down to the best of your abilities, and firewalled from the network, only allowing critical ports such as what the backup software uses, and perhaps ssh or RDP (if a Windows box). This machine copies everything from the other servers onto a large disk array, then to tape. The tapes are then cycled offsite via a service like Iron Mountain. Of course, the tapes are encrypted, and corporate officers get a copy of the master keys.
Why tapes? Because they can be set read only after they are dismounted, and no computer, no matter how infected can modify or delete the tape contents once this is done, outside of a reflash of the tape drive's BIOS. This is important because its not unheard of for someone to write a program that trashes backups over a time interval. Higher end tapes can be used as WORM media like DLT-ICE.
I can't emphasize enough about securing the backup server, both physically and network-wise. If this box gets compromised, all your data is available. On Windows machines, I recommend using some form of disk encryption (Bitlocker if the machine has a TPM, TrueCrypt, etc) so if the backup server or an array gets physically stolen, the data is of no use to a thief. This is in addition to the backup program's encryption.
After you have a central backup server installed, secured (security is paramount on this machine unless the backup program client can do encryption), and backups running, you focus on the other levels of backup.
The next level of backup is on the local servers. Most operating systems have a method of backing up the computer. If you can do this with a server, fire off a snapshot backup every month or so. Most OS backup methods don't have encryption, so this backup should go directly to a tape safe or secured container in the data center. Optionally, you can install backup software locally that can encrypt. I like using the backup/restore utility the OS gives for an image every quarter, then using more secure software more often, so the OS backups can be stored in a tape safe or physically secure container. This way, if the third party backup software ends up inoperable, there is still a method of getting a machine up somehow, or putting it in a virtual machine for recovery purposes.
Finally, after you have backup servers and a rotation, companies might consider offsite cloud backup services like Mozy. Mozy offers use of keyfiles so all data is stored encrypted (encrypted on the client end). Of course, making sure the encryption key is stored safely is paramount, and the cost of storing a large backup in Mozy's cloud may be prohibitive. However, if worse comes to worst and your site is completely knocked out, as well as the offsite backup site, it may be thing that keeps your business up.
Of course, scale this up or down as per your company's needs. A smaller business can get by using Mozy and a Windows Server 2008 box running Bitlocker, a network backup program with encryption such as Retrospect or Backup Exec, and using external drives every month to copy backup sets from the main ones to store offsite.
A larger business might see about a true backup fabric system sold by IBM (TSM), EMC (Networker), or Microsoft's solution.
The key is to not just have some built in redundancy so if one backup method is not usable, you have another, even if the backups are older, but to be able to do this in a manner that doesn't add too much time and equipment expense.
Re:Hindsight is always 20/20 (Score:5, Informative)
And for those who don't like to pay $10000 for backup software, there's Bacula [bacula.org]. Couple that with an LTO-4 drive (~1000) and LTO-4 tapes (800GB uncompressed, ~60/piece) and you're set. Rsync.net is a decent, cheap online provider for those gaps when you haven't rotated tapes.
Bacula is pretty sweet because it lets you backup to disk volumes and then you can schedule a roll to tape. So you can just back everything up incrementally to a disk volume and then copy those backups to tape, and then run rsync on the disk volumes to have an offsite, online backup. When recovering, you ask to recover from whatever's available. If you keep enough disk storage around (and there's really no reason not to) you can recover to any date in the past. In the event of a disaster your tapes come into play.
Now with drives so cheap the temptation is to buy a external hard drive and use that. But tapes have a long history, guaranteed backwards compatibility (planned anyway, LTO drives have to R/W the previous generation and Read 2 generations back), last longer than moving drives, are simpler, lighter, more robust and more portable. Not that I wouldn't keep a external around to dump desktops but tape is the DR standard.
Parent
Of course I have an extra set of keys.. (Score:5, Funny)
Public Viewing (Score:5, Funny)
A public viewing will be available at:
http://web.archive.org/web/20080116064652/http://www.avsim.com/ [archive.org]
No date has been set for the funeral.
The people running the site ARE NOT IT Admins (Score:5, Informative)
Listen folks....we're talking about a couple of guys who spent their free time creating a website. They're not making any real money out of this (in fact, they all have regular day jobs).
They've been advertising for a Tech Manager (non-paid) for quite a quite so time now. They did get one recently...but it turns out the guy harvested the emails from the systems and sent out a bunch of spam. He has since been fired.Even though the avsim folks aren't saying it was him who hacked and destroyed their site, it's quite hard not to think it was him.
It's been quite a blow to the flightsim community and I have noticed a lot of IT folks are offering help.....I just haven't seen a single one on this thread.
Re:The people running the site ARE NOT IT Admins (Score:5, Insightful)
Parent
Re:the web is ephemeral (Score:5, Funny)
Maybe future historians will consider this a dark age, whose intellectual production was lost.
Please don't say our treasured facebook, twitter, slashdot posts, wikipedia revision wars and v1agra spam may not be preserved for posterity.
I'm not yet convinced that information that today exists only on the internet is really meant for eternity :)
Parent
Eternity (Score:5, Funny)
Only goatse is eternal. The rest is being used to seed a randomness generator somewhere.
Parent
Re:the web is ephemeral (Score:5, Funny)
Wikipedia revision wars will be a GOLDMINE for future archeologist.
Think about just how much they reveal about a certain topic.
Such as the difference of opinion about the color [wikipedia.org] variations [wikipedia.org] of the carrot [wikipedia.org] !
Parent
Re:the web is ephemeral (Score:5, Insightful)
nonsense...
completely inaccurate guestimation, but probably only about 1% of anything carved in stone, is still decipherable or even exists, same with scrolls, otherwise we'd be littered with 2000 year old shopping lists, love letters, etc, how many notebooks (the paper kind) have you gone through during school, as journals, boredom... still have them all?
Hell, we probably only have about 1% of the stuff that was written down 100 years ago, probably only about 3% of the buildings, 0.3% of the cars, 2% of the paintings...etc...etc... most of the ancient books we have, are copies of copies of copies, and we can do that with magnitudes of efficiency now, not to mention recovery, hard drive gets erased, it's easier to get the data back than a scroll that's been erased, or a stone.
If even 0.1% of what we have on the internet right now exists in 500 years, it'll still probably be more than everything we have in stone, scrolls, and print right now...
With the various sorts of "Library of Congress" out there, if you had the chance to peruse and take/read whatever you wanted, you'd probably only find 0.5% of it interesting anyways, much like what's on the internet.
Parent
Re:Really? (Score:5, Funny)
Anyone who hacks a flight-sim sight has no life and really needs to get laid.
Coming from a slashdotter that is pretty rough.
Parent
Re:There's a special place in hell... (Score:5, Insightful)
Parent
Re:Lies, damn lies. (Score:5, Informative)
Remember kids if it isn't backed up to an off-line copy then it isn't backed up.
Parent
Re:Lies, damn lies. (Score:5, Insightful)
Backups: Not hard to get right, just very easy to get wrong.
Parent
Re:Lies, damn lies. (Score:5, Insightful)
And I'm going to respectfully disagree with you too.
For most small businesses cash flow is critical. If you don't have a record of who paid you in the last month then you can't invoice the rest and you are dead. Your repeat customers will spot duplicate invoices and probably just block payments until it's all sorted out. The attack that you are defending against is either a fire which destroys your office or a burglary which steals all your computers in the night, including the backup box, taking the backups just because they happen to be there.
You need off site backups on a different, non internet-connected medium no less often than once a week. That is the maximum time for which it is acceptable (we are talking about disaster recovery here; "acceptable" has a different meaning from normal) to re-invoice people who have already paid you. Even so, most such incidents destroy small businesses completely just because they don't manage to get people back working in time. This just gives you a fighting chance if you have a nice and understanding bank manager and do a little more disaster planning. It is astounding how much difference spending four hours just thinking about it can make (e.g. you know the number of the temporary office providers, you know which people in your office can work from home and you realise everybody in your company should have a mobile phone, especially the receptionist).
And finally; if you haven't tried restoring from it, it isn't a backup.
Parent
Re:Lies, damn lies. (Score:5, Insightful)
"And finally; if you haven't tried restoring from it, it isn't a backup."
That, my friend, need to be carved on a marble plate and hung over the door to every datacenter.
Parent
Re:Lies, damn lies. (Score:5, Informative)
A dedicated backup box can be much more hardened
What you've described is only marginally better than what these people did. A second server playing backup device, even if it's "much more hardened", whatever that means, is still an extremely lousy and ineffective backup. If lightening hits your building or arson or theft, your "it's hardened"! backup server is just as toasted as the primary. Backups MUST be to removable media that's kept off site and inactive.
Otherwise you've done practically the same thing for data "backup" as the RAID does via disks, except with two servers.
Parent
Re:Lies, damn lies. (Score:5, Funny)
lightening hits your building or arson or theft
I thought lightening came under theft
Parent
Re:Lies, damn lies. (Score:5, Insightful)
If by 'dedicated backup box' you mean two offsite machines both of which are themselves in highly secure and robust sites I could be convinced that it's possible to build an effective backup strategy around them.
Our backup strategy for the office (files/databases) is to have a single, off-site 'consolidation server' which we dump transaction logs to real time (with full database dump overnight), and make incremental backups of files every hour throughout the day rsyncing the full current file overnight. Then, this machine is itself backed up using a full weekly/daily differential tape backups.
I get shivers how everyone talks about backup strategies but not restore strategies as if the data fairy will wave a wand to restore your backups when it all goes tits up. We have a regularly rehearsed backup strategy. If we're in the same office, we attempt a pull down from the remote consolidation server. If that is down, or we are in another office etc. we put a copy of the encrypted tape backups on USB drives, courier them over and restore them here.
My biggest bugbear is that the remote consolidation server is not encrypted - we have to trust the hosting partner. We could not find an acceptable method that didn't involve remote plaintext data existing.
We spend about £12k (or about half a junior IT FTE) a year on backups and there is not a single day where I do not worry and personally check that they're working correctly.
Does anyone see any holes or room for improvement? Would be very happy for suggestions to improve.
Parent