Hacker Destroys Avsim.com, Along With Its Backups 780
el americano writes "Flight Simulator community website Avsim has experienced a total data loss after both of their online servers were hacked. The site's founder, Tom Allensworth, explained why 13 years of community developed terrains, skins, and mods will not be restored from backups: 'Some have asked whether or not we had back ups. Yes, we dutifully backed up our servers every day. Unfortunately, we backed up the servers between our two servers. The hacker took out both servers, destroying our ability to use one or the other back up to remedy the situation.'"
the web is ephemeral (Score:1, Interesting)
That really sucks - I used to use that site all the time back when I was into sims, and even contributed some TerraScene and other goodies.
This highlights the ephemeral nature of the web. Thousands of years ago, information was carved into rock, and we still have many of the originals. Then it was written onto scrolls, some of which survive today. Now it's on a disk, with a lifetime of a few years. Yes, they can be backed up... but the whole thing is very precarious. In 500 years how much of what people create today on sites like avsim will still exist? I predict basically none of it.
Maybe future historians will consider this a dark age, whose intellectual production was lost.
Offsite backups? (Score:5, Interesting)
I realize that from quite a few people's perspectives, storing their backups in a separate building constitutes off site storage. I'd almost buy that strategy. Not in the same environment, network, city etc.
These guys were stupid.
The day after 9/11 I was in an elevator, and caught a snippet of conversation between 2 people that had business interests with a firm that was in the WTC. The comment I heard was 'their backups were in the other building'. Another company lost.
You can never totally plan for every contingency, but you can insure yourself. I know many developers that take hard copies of their code (meaning on removable media) home just for this reason. I have seen sys admins do the same because they didn't trust their DR stratagy.
This was avoidable. This isn't even about disaster recovery. It is about business continuity.
You can't afford not to protect your data.
Learn from Kuwait too (Score:3, Interesting)
All they had was a back up copy that made it out.
After the war they could go in and find what was tampered with. ie who got a false identity.
Take your data home with you every night.
Some backup stories (Score:5, Interesting)
A couple of years later, I am in the pub with some mates and John turns up. I ask him how he's managed to finish work and get to the pub so early. "I did a fast backup" he said. I was interested so I asked him to explain. "Oh, it's easy, get the target tapes from the rack, rub out the old date, write the new date, put them back into rack and go to the pub"
Worked for a large software shop in the 90's. I am part of a decent sized Oracle development (circa 50 devs). Ops decides that Oracles backup routines are too slow and 'optimize' them. Some weeks later - guess what - there's a problem and the backups are useless - No physical audit trail this time - the team has to redo all of there work - it was not good for the project budget, the team moral or the client
Re:This should be a lesson... (Score:3, Interesting)
Re:This should be a lesson... (Score:3, Interesting)
bullshit (Score:3, Interesting)
Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.
How about once? With zeros.
http://16systems.com/zero.php [16systems.com]
If you can retrieve you data from a drive after it has been dd'd with /dev/zero, you might be able to win this prize.
If you happen to be in the situation described, chances are you're fucked.
Re:This should be a lesson... (Score:2, Interesting)
Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.
A simple dd command with one run of 0's will permanently delete the data on a disk. Once upon a time it may have been possible to read the data after a single write but it is no longer possible. This challenge has been standing for quite some time [16systems.com] and even though this is not proof of my assertion I am certain the multiple passes of writes thing is complete garbage.
Re:bullshit (Score:4, Interesting)
Re:Lies, damn lies. (Score:3, Interesting)
For a medium-to-large business, I wholeheartedly agree with you.
However, what would be a good policy for small business (sole proprietorships or only a few people) or individuals? Not everyone can afford properly secured offline remote backups. The best effort that the average individual can do is set up a cron job rsync to a remote server if he/she has one and then do a few local rsyncs for redundancy every few hours. (this is what I do)
Re:Real men... (Score:5, Interesting)
Re:This should be a lesson... (Score:5, Interesting)
Re:Offsite backups? (Score:4, Interesting)
Re:Some backup stories (Score:2, Interesting)
Months go by - then there's a problem - whoops backups are useless - Luckily there's a physical audit trail so we we can pay for very large data entry exercise to get our client's data back.
I worked for an outfit where they couldn't get authorization for a backup server. Dumb little me added up the cost of a few days idle time for the small group that used the server. Simple subtraction said, "Get the damned server." So they did.
I built the new server (OS/2) according to my manager's specs. It worked OK. I said, "You know, we'll never have a better chance to test out out disaster recovery. Let's scrape the HDs in this thing and restore it from the backup." But oh no, the smart son of a bitch I worked for decided the old server might be failing, so we had to put the new one online right away.
Nice going, asshole -- some time later (don't remember why -- this was eight years ago), we lost the new server. Get backup tapes, run restore. Bring up system -- ha, ha -- ACLs not restored by the proprietary backup software we were using. (It worked fine for file restores, just not a full system restore.)
We had to re-enter all of the user database and individual permissions by hand. Nice going, hot-shit IBM-trained systems dude manager. You're just lucky it was mainly a file server for our department, not one of the large networks managed by the real network guys in the company.
This asshole was so lame that he didn't even take advantage of the OS/2 facility which logged all the build options for later re-use, including making tweaks so you could build and modify systems quickly when you wanted to test out various build options. It was like a fucking high end European luxury car -- each machine was hand built.
Re:Lies, damn lies. (Score:2, Interesting)
You only have 4GB of irreplaceable data?
Just my family photos/videos archive broke the 2TB boundary this year, and that doesn't include the 1TB of archive media from my personal projects (images, old versions of personal websites, video montages, etc).
I think having a normally off, seldom used mirror of my 3TB of data the best backup solution I can muster.
Re:This should be a lesson... (Score:5, Interesting)
Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.
The DoD spec is written as it is for a reason. Given a drive with confidential data on it, an unauthorized person attempting to access the drive does not need to get everything back to pristine condition. Even recovering a small part of the total data set can cause incalculable damage if it's the right small part. The value of sites like Avsim are in the whole rather than the sum of the parts.
I've recovered data off of formatted HDD's, off of corrupted file systems, off of compact flash cards and other media (Really useful if you want to keep those photo's that someone thought was deleted, be aware of this people).
There's a large dependency on what you're trying to recover off of. DOS/NTFS are fairly easy to do recovers from. The first character of the filename is zilched out and the rest of the data to find the file is left intact. UNIX/Linux filesystems are a bear. Once you hit "rm", you've lost the ref to your inode. Putting Humpty Dumpty together again at that point becomes nearly impossible because the record which shows where all the pieces are is lost to you. If you have known text from the file, and a good knowledge of how the filesystem works, you should be able to backtrack. Otherwise? God help you.
There's also an issue of how the data is stored. A single-drive system is fairly straightforward. 2 drives are harder. Once you get into a SAN/NAS where data is spread over multiple drives, recovery of even a single file with known text becomes tricky. Multiple files? Unknown data? The only hope I would see at that point is to put a large segment of the Slashdot community on the problem and tell them a large trove of high-res pictures of Natalie Portman completely nekkid are stored within.
Re:This should be a lesson... (Score:4, Interesting)
Assuming we are talking about a disk that has been entirely written with zero's or random data, eg a deliberate attempt to render the disk unreadable - citation needed (or are you just repeating something you heard from someone who heard it from someone else?)
The 'black boxes' are designed to ensure the survival of the internal medium, so it's no surprise that the data is recoverable (don't they use analogue tape on a loop? or is that just for the voice recorder? or is my knowledge way out of data :)
When inter track spacings were wider and density in other dimensions was lower (20 years ago?) it was possible to recover data after a complete write with zero's, but not now.
While Wikipedia isn't the definitive answer on anything, it clearly states in several places that a single pass of the entire disk is enough to erase the disk with no chance of recovery.
Sounds awfully like an urban legend. Are the illuminati involved somehow? :p
Re:overwritten once CAN be recovered (Score:2, Interesting)
Earlier this year we had a hard drive failure, and we really wanted the data back badly (money isn't important).
So, off it went to a 'professional recovery' service. A couple thousand bucks later, they were able to image some portion of the drive, and handed us the files they had recovered.
The number of files they were able to recover was pitiful. It was as if they imaged the disk and ran a simple undelete program (ext2) or something.
Is this normal? Are there any guys out there that dig a little deeper than this?
I spent 2 weeks writing my own recovery software that carved the data out of the drive image, and saved 10 times the number of files they were able to. If I can do it, why can't they? Are there any recovery experts that actually compare samples of the data to be recovered (in our case, our own format binary data files, not similar to anything else) and make an effort to carve the files out, instead of relying on whatever recoverable filesystem information is available? (yeah, without being able to rely on ext2 filesystem information, you have to make certain assumptions..)
Re:Lies, damn lies. (Score:3, Interesting)
This is pretty much what my company does. There are only 5 full time staff so things like tape backup procedures are too expensive for our needs. We do have a lot of data though.
We have everything on a NAS running RAID 5, at the end of each day anything that has been changed that day gets written to a DVD, which goes offsite. Normally chucked into someones bag on the way out of the door, but the DVDs are only for quick file recovery so not crucial.
At the end of each week we do a complete backup of the NAS onto a 2TB external drive (which is actually 2 x 1TB drives running JBOD in an enclosure). That goes offsite, then at the end of the next week a second drive is used for the offsite. So we always have 2 copies of everything offsite max 1week or 2weeks old respectively.
Not a completely fool proof system but good enough to give me peace of mind with respect to hardware failure, theft, fire and penetration of the office network.
Re:This should be a lesson... (Score:5, Interesting)
Oh, he gets my sympathies. I've had cases where I was specifically told "that is a scratch server: do not back it up, no one is supposed to keep real data on it". And when it crashed, my employers were very fortunate indeed that I'd completely ignored this and quietly been backing it up with my reserve, emergency tape drive, partly to make sure it kept working, partly to test out new backup tools, and partly because I knew staff would ignore this and use the big lump of spare storage for convenient archival space. My employer was actually angry at me for doing so, but the QA department was very, very, very grateful indeed.
The lesson is more subtle than some of us might realize, though. Never rely on a _single_ method of backup or data storage, because any factor that ruins that backup can ruin all copies of it. This is true for backup systems that use proprietary format, or a failed tape drive that's been screwing up backups for the last year (which I've seen happen with old mag-tape media). And I _love_ online backups: you can make the data accessible via NFS or CIFS or other file-sharing to people to recover the files they just accidentally deleted, without having to invest in a very, very expensive NetApp or similar file server. But oh, dear, I've also seen what happens when someone screws up the backup tools and deletes all the copies at the same time.
Re:These aren't hackers (Score:2, Interesting)
Plus there's no telling if the hacker knew the impact data deletion would have...
There's a difference between:
"MWUAHAHAHA, I'm destroying 13 years of work."
and
"MWUAHAHAHA, it'll be fun watching them restore from tape"
Re:Lies, damn lies. (Score:4, Interesting)
At least one. I've been doing this ever since one of my colleagues, who had been working on a book for a year and a half, had his house burgled and the thief took his computer AND his backup system AND the lockbox in which he kept backup tapes (it was a long time ago). He thought he was doing everything right.
I had to help him scan several hundred pages from several sets of loose pages and chapters from a marked-up stale manuscript, and this was back in a day when scanners and OCR weren't all that great.
I saw him suffer so badly, and was so shaken in his faith in technology (he's an English teacher, not really a technical adept) that we devised a simple system of rotating tapes (later external hard drives, now 8gig flash drives) that we used to keep for each other (he's moved away now).
If I'm working on something that I absolutely cannot lose, I'll occasionally put it on a flash drive and give it to my wife. That's my safest keeping. I don't ask her where she keeps it, but sometimes when I get it back it has a faint aroma of the Pacific Ocean.
Re:This should be a lesson... (Score:4, Interesting)
Surely all the people who've downloaded the downloadable content over the years can all band together and restore a large proportion of it?
I also thought: developers and user must have downloaded/synced great deal of source (binaries for sure :-).
So why not scramble it back together? As for the web site, perhaps pages are cached somewhere on the web, as some people already pointed out.
I've visited their site about a month or so ago, and they did really impressive work, they should try to restore it.
Re:The people running the site ARE NOT IT Admins (Score:2, Interesting)
Seriously: buying a LTO drive and sending media to IronMountain is a fantastic idea, but this community sites like this aren't a business. They're, say, 10,000 devout users hitting a virtual machine or two, and the admins are _already_ dropping a couple hundred per month on the hosting. Where does the money come from? Where does the _time_ come from? Whoever should've been testing avsim's backups was probably also moderating forums, working on the site, and working a day job.
Free community sites like this are great, great part of internet, and the people who run them are pouring their own time and money into something they love. And unless you want to run a free offsite backup service, the best you can do is to warn people what can happen, show them what a reasonably solid backup strategy looks like, and hope that no dickheads trash their site.
Re:Too Risky (Score:4, Interesting)
Obviously you've never had to back up about 8TB of data, with about 200GB of incremental changes weekly. We manufacture products with custom artwork, so we go through a lot of larger art files every week. You have any idea how much fun it would be to split up that much data across LTO-3 tapes? And then do incrementals? And then test it regularly? Our stateside IT staff that would be responsible for the backups consists of one other person aside from myself. We don't have the time to maintain a tape-based system. We do a combination of offsite, online backups...some to our facilities in the Philippines in China for both backup and operating purposes, and some to our 'local' IPSs for purely backup purposes. We do keep an additional copy our 40GB SQL database for our ERP system backed up weekly to a portable hard drive that we switch with one in our safety deposit box each week.
While tapes have been 'standard business practice' for years, the technology is lagging behind and is becoming inadequeate, especially for large businesses (you think Google, Microsoft, Amazon, etc. back everything up on tapes?), but also smaller businesses, too.