Hacker Destroys Avsim.com, Along With Its Backups 780
el americano writes "Flight Simulator community website Avsim has experienced a total data loss after both of their online servers were hacked. The site's founder, Tom Allensworth, explained why 13 years of community developed terrains, skins, and mods will not be restored from backups: 'Some have asked whether or not we had back ups. Yes, we dutifully backed up our servers every day. Unfortunately, we backed up the servers between our two servers. The hacker took out both servers, destroying our ability to use one or the other back up to remedy the situation.'"
One word (Score:3, Funny)
Owned.
Lies, damn lies. (Score:4, Informative)
The only way to do backups properly is to have a complete set, offline, in a separate location.
Sheesh. When will people learn?
Re:Lies, damn lies. (Score:5, Informative)
Remember kids if it isn't backed up to an off-line copy then it isn't backed up.
Re:Lies, damn lies. (Score:5, Insightful)
Backups: Not hard to get right, just very easy to get wrong.
Re:Lies, damn lies. (Score:4, Informative)
>but how many people actually keep off-site backups for home use?
er, I do. I have a 4GB memory stick that I sync with my back drive on my home PC and a PC at work.
Once a month I burn a DVD.
Re:Lies, damn lies. (Score:4, Interesting)
At least one. I've been doing this ever since one of my colleagues, who had been working on a book for a year and a half, had his house burgled and the thief took his computer AND his backup system AND the lockbox in which he kept backup tapes (it was a long time ago). He thought he was doing everything right.
I had to help him scan several hundred pages from several sets of loose pages and chapters from a marked-up stale manuscript, and this was back in a day when scanners and OCR weren't all that great.
I saw him suffer so badly, and was so shaken in his faith in technology (he's an English teacher, not really a technical adept) that we devised a simple system of rotating tapes (later external hard drives, now 8gig flash drives) that we used to keep for each other (he's moved away now).
If I'm working on something that I absolutely cannot lose, I'll occasionally put it on a flash drive and give it to my wife. That's my safest keeping. I don't ask her where she keeps it, but sometimes when I get it back it has a faint aroma of the Pacific Ocean.
Re:Lies, damn lies. (Score:4, Funny)
You only have 4GB of irreplaceable data?
Just my family photos/videos archive broke the 2TB boundary this year, and that doesn't include the 1TB of archive media from my personal projects (images, old versions of personal websites, video montages, etc).
I think having a normally off, seldom used mirror of my 3TB of data the best backup solution I can muster.
You only have 3TB of irreplaceable data? I'm currently up to 5PB, though half of that is my pr0n collection.
Re:Lies, damn lies. (Score:5, Funny)
That's not an archive, pal, that's evidence.
You wanna destroy that stuff, the sooner the better.
Re:Lies, damn lies. (Score:5, Funny)
Note to self: never, ever ask you about your hobbies.
Re:Lies, damn lies. (Score:5, Insightful)
And I'm going to respectfully disagree with you too.
For most small businesses cash flow is critical. If you don't have a record of who paid you in the last month then you can't invoice the rest and you are dead. Your repeat customers will spot duplicate invoices and probably just block payments until it's all sorted out. The attack that you are defending against is either a fire which destroys your office or a burglary which steals all your computers in the night, including the backup box, taking the backups just because they happen to be there.
You need off site backups on a different, non internet-connected medium no less often than once a week. That is the maximum time for which it is acceptable (we are talking about disaster recovery here; "acceptable" has a different meaning from normal) to re-invoice people who have already paid you. Even so, most such incidents destroy small businesses completely just because they don't manage to get people back working in time. This just gives you a fighting chance if you have a nice and understanding bank manager and do a little more disaster planning. It is astounding how much difference spending four hours just thinking about it can make (e.g. you know the number of the temporary office providers, you know which people in your office can work from home and you realise everybody in your company should have a mobile phone, especially the receptionist).
And finally; if you haven't tried restoring from it, it isn't a backup.
Re:Lies, damn lies. (Score:4, Insightful)
Off topic, the internet would be a much nicer place if all disagreements were presumed to be respectful until obviously indicated otherwise...
Re:Lies, damn lies. (Score:5, Insightful)
"And finally; if you haven't tried restoring from it, it isn't a backup."
That, my friend, need to be carved on a marble plate and hung over the door to every datacenter.
Re:Lies, damn lies. (Score:4, Insightful)
Too Risky (Score:5, Insightful)
If it's on the internet, then it is exposed.
Re:Too Risky (Score:4, Interesting)
Obviously you've never had to back up about 8TB of data, with about 200GB of incremental changes weekly. We manufacture products with custom artwork, so we go through a lot of larger art files every week. You have any idea how much fun it would be to split up that much data across LTO-3 tapes? And then do incrementals? And then test it regularly? Our stateside IT staff that would be responsible for the backups consists of one other person aside from myself. We don't have the time to maintain a tape-based system. We do a combination of offsite, online backups...some to our facilities in the Philippines in China for both backup and operating purposes, and some to our 'local' IPSs for purely backup purposes. We do keep an additional copy our 40GB SQL database for our ERP system backed up weekly to a portable hard drive that we switch with one in our safety deposit box each week.
While tapes have been 'standard business practice' for years, the technology is lagging behind and is becoming inadequeate, especially for large businesses (you think Google, Microsoft, Amazon, etc. back everything up on tapes?), but also smaller businesses, too.
Re:Lies, damn lies. (Score:5, Informative)
A dedicated backup box can be much more hardened
What you've described is only marginally better than what these people did. A second server playing backup device, even if it's "much more hardened", whatever that means, is still an extremely lousy and ineffective backup. If lightening hits your building or arson or theft, your "it's hardened"! backup server is just as toasted as the primary. Backups MUST be to removable media that's kept off site and inactive.
Otherwise you've done practically the same thing for data "backup" as the RAID does via disks, except with two servers.
Re:Lies, damn lies. (Score:5, Funny)
lightening hits your building or arson or theft
I thought lightening came under theft
Re:Lies, damn lies. (Score:5, Insightful)
If by 'dedicated backup box' you mean two offsite machines both of which are themselves in highly secure and robust sites I could be convinced that it's possible to build an effective backup strategy around them.
Our backup strategy for the office (files/databases) is to have a single, off-site 'consolidation server' which we dump transaction logs to real time (with full database dump overnight), and make incremental backups of files every hour throughout the day rsyncing the full current file overnight. Then, this machine is itself backed up using a full weekly/daily differential tape backups.
I get shivers how everyone talks about backup strategies but not restore strategies as if the data fairy will wave a wand to restore your backups when it all goes tits up. We have a regularly rehearsed backup strategy. If we're in the same office, we attempt a pull down from the remote consolidation server. If that is down, or we are in another office etc. we put a copy of the encrypted tape backups on USB drives, courier them over and restore them here.
My biggest bugbear is that the remote consolidation server is not encrypted - we have to trust the hosting partner. We could not find an acceptable method that didn't involve remote plaintext data existing.
We spend about £12k (or about half a junior IT FTE) a year on backups and there is not a single day where I do not worry and personally check that they're working correctly.
Does anyone see any holes or room for improvement? Would be very happy for suggestions to improve.
Re:Lies, damn lies. (Score:4, Informative)
I'm going to respectfully disagree, there.
A dedicated backup box can be much more hardened than a general-purpose webserver, as the backup box pretty much has a job of storing and retrieving files.
A solid system of incremental backups helps, too.
Yes, taking it offline is great. Do that... maybe monthly, if that.
This scenario sounds much more like someone confused "RAID" with "Backup". RAID (and other high-availability schemes) protects you from hardware failure. Backup protects you from more software failure and human error.
Wrong.
What if your building burns down? What if some minor fire triggers the sprinklers? What if you get struck by lightning? What if an employee goes postal and takes a sledgehammer to all the electronics? What if a tree falls on the power lines and sends a giant surge through your wiring? What if someone breaks in and steals all the computers?
It isn't a backup unless it leaves the site.
Of course you could put your live backup box on the other end of some fiber in another state... That's physically off-site... But as long as it is up and running you have to worry about it as well. Hardened or not, it could get hacked. Or it could get a virus. Or some random glitch could corrupt the data on disk. Or its motherboard/HDD/CPU/whatever could die.
It isn't a backup unless it is offline.
And then there's the question of whether the thing actually works... You can have all the backups in the world, but if they're all corrupt it won't do you any good. You'll be restoring broken garbage to your replacement server.
It isn't a backup unless it has been verified.
What all of this comes down to is some kind of relatively portable media. Tapes, removable HDDs, CDs, DVDs, whatever. You want something that can leave the building on a daily basis. You want pretty much all your media to be out of the building. Bring in just what you need to run today's backup, and then take it out of the building as soon as that is done. Preferably to someplace relatively remote and safe... A safety deposit box is great. Or if someone has a safe at home. Or if you've got a branch-office or something.
Re:Lies, damn lies. (Score:4, Insightful)
I wouldn't call it lies - I'd call it ignorance
Re:Lies, damn lies. (Score:4, Insightful)
They buy USB hard drives (at least six times the amount of data they have, split among at least three drives), rent a safety deposit box in a bank, and install rdiff-backup. Then they rotate the drives weekly -- at any point one drive is backing up their systems daily, two are stored at the bank. Complete incremental backup solution with offsite storage.
This should be a lesson... (Score:5, Informative)
To any sysadmins and DBAs...
Make sure you have offsite backups
Re:This should be a lesson... (Score:5, Informative)
Make sure you have offsite backups
In this case, even offline (as opposed to offsite) backups would have sufficed.
Removable hard disks, DVDs -- hell, even tapes. These are all forms of backups that can't be compromised (well, easily) over the internets.
Re: (Score:3, Informative)
From the article
... we backed up the servers between our two servers.
Nope, backing up a server to another online server is not a backup, it's merely another online copy.
Re: (Score:3, Insightful)
It's the difference between HA [High Availability] and DR [Disaster Recovery].
Unfortunately, they suffered a disaster, not a 'mere' server failure.
All that said, my condolences to the server admin / founder, and especially, to all of the contributors. Thirteen years is a lot of data.
Re:This should be a lesson... (Score:5, Funny)
Thirteen years is a lot of data.
Bah--it's not that bad. They actually have crude backups of all their terrain data. They just have to figure out how to restore from 'IRL' format.
Re:This should be a lesson... (Score:5, Funny)
Why would you put that on your resume?
Re:This should be a lesson... (Score:5, Interesting)
Oh, he gets my sympathies. I've had cases where I was specifically told "that is a scratch server: do not back it up, no one is supposed to keep real data on it". And when it crashed, my employers were very fortunate indeed that I'd completely ignored this and quietly been backing it up with my reserve, emergency tape drive, partly to make sure it kept working, partly to test out new backup tools, and partly because I knew staff would ignore this and use the big lump of spare storage for convenient archival space. My employer was actually angry at me for doing so, but the QA department was very, very, very grateful indeed.
The lesson is more subtle than some of us might realize, though. Never rely on a _single_ method of backup or data storage, because any factor that ruins that backup can ruin all copies of it. This is true for backup systems that use proprietary format, or a failed tape drive that's been screwing up backups for the last year (which I've seen happen with old mag-tape media). And I _love_ online backups: you can make the data accessible via NFS or CIFS or other file-sharing to people to recover the files they just accidentally deleted, without having to invest in a very, very expensive NetApp or similar file server. But oh, dear, I've also seen what happens when someone screws up the backup tools and deletes all the copies at the same time.
Re:This should be a lesson... (Score:5, Insightful)
13 years of work lost!
Suddenly those external hard drives and safe deposit box don't look so expensive.
Re:This should be a lesson... (Score:5, Informative)
It's actually very difficult to truly destroy data, especially remotely. There is actually a reason the DoD spec. requires physical destruction of the media.
Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.
The article doesn't lead me to believe that he's tried very hard to get this data back.. Maybe somebody (not me) who cares about this resource, should offer an attempt at data recovery.. Just be sure to hurry, before they do something that will ensure you cannot recover the data.
I've recovered data off of formatted HDD's, off of corrupted file systems, off of compact flash cards and other media (Really useful if you want to keep those photo's that someone thought was deleted, be aware of this people).
It's amazing how most people seem to think deleted means gone.
Re: (Score:3, Interesting)
Re:This should be a lesson... (Score:4, Insightful)
How much fuel? What type? How toxic is it? Does it require special handling? Does it require special tanking? What are the safety procedures? Are there boil-off rates, or evaporation rates? What type of oxidizer will the fuel require? How much will it require? How big of an engine? What type of nozzle? Will it withstand the heat of the exhaust? What materials will it be built from? How do you ignite the fuel? Is the rocket strong enough to withstand the launch stresses? What happens when it breaks the sound barrier? What size of payload will that much of that type of fuel running through that engine lift? Will it reach orbit? Will it reach the size and shape of orbit needed to put that payload where we want it? Will it be able to do that in a single stage or multiple stages? Will it need multiple engines? How many? What layout? Will the upper stage engines start in a vacuum? When does the staging event occur? Where will the spent stages land? Are they expendable, or do they need to be recovered? What sort of accelerations will the payload experience? Will they be gentle enough to put humans on top of the stack? Can the engines be throttled? What sort of failure modes does this rocket experience? Is there a way of detecting an imminent failure? Will there be enough time to trigger the Launch Abort System? What sort of guidance system will there be? How will you steer the rocket? Is it even dynamically stable? What happens to that stability if an engine fails? What happens to the center of mass as the fuel is expended? Does that affect the stability?
Yeah, rocket science is real easy.
Re:This should be a lesson... (Score:5, Funny)
Re: (Score:3, Interesting)
Re:This should be a lesson... (Score:5, Insightful)
multiple times? I'd like to see you recover something that has been overwritten once.
Re:overwritten once CAN be recovered (Score:5, Informative)
Re:overwritten once CAN be recovered (Score:4, Informative)
Re:overwritten once CAN be recovered (Score:5, Insightful)
I've never seen *any* evidence or heard of *any* occasion that such a recovery, even from a only-once-zeroed drive was done.
Now the point is, one could say "of cooourse not, guys that can do this won't do it for peanuts, besides they're secres service" etc etc. But the point is: even if it's secret service and really expensive, at leas *some* news about it should have hit the public -- after all, this myth has been around for several years (a decade?) now.
I'd still even like to hear from a success story. Or even find a company that advertizes "We can (partly?) recover your zero'ed data -- it's going to cost a fortune, an arm and a leg, but we can." Haven't seen that one either yet. Not a commercial, not an offer, nothing... besides legends.
Re:overwritten once CAN be recovered (Score:5, Insightful)
I am a computer forensics expert. I search for deleted data for a living, and I testify in court as to what can be done.
Unfortunately you are wrong about recovering data that has been overwritten by using magnetic magic.
That is an urban legend that has been disproven. Maybe 20 years ago using low density MFM drives it was theoretically possible, but now it is not. Maybe the NSA has some tech they reversed engineered from an Area-51 UFO to do this, but I've never seen or heard of it.
Even Gutmann has recanted his 38 wipes recommendation.
Now don't mistake overwritten data for deleted data. When data is deleted it is NOT overwritten. When a hard drive is re-formatted almost nothing is over-written. When a file is overwritten with zeros or random bytes there are probably 10 more copies of that file and previous versions of that file floating around in unallocated sectors, swap space, file slack, hibernation files, etc.
But what IS overwritten is gone.
You are correct about drive age (Score:5, Informative)
Tedious and expensive, but several people made a good living out of doing it (one guy I knew did it as a hobby and made over UKP100K one year.) However, as bits get smaller, servos get more accurate, and tracks get denser, the modus operandi just ceases to exist any more.
Mind you, for security reasons I always dismantle old drives and bend the disks in half using a lump hammer. That, and the fact that hard drive magnets are just incredibly useful if you have a steel hulled boat and want convenient attachments for e.g. cable ties. They are powerful and very short range, and usually nickel plated. To buy a pair of equally useful magnets from hardware stores costs nearly as much as a drive.
Re:This should be a lesson... (Score:5, Insightful)
People keep repeating that mantra to each other, but is it really true? Getting data off a 'formatted' disk is pretty easy as a format rarely does more than write a few sectors at the start of the disk. Getting data off of a disk that has had 'dd if=/dev/random of=/dev/sda' done to it is a different matter altogether.
There have been papers written about getting some data out of the inter-track space, and scraping it off the noise floor etc with electron microscopes, but as far as I have researched, nobody has actually done it.
I put it to you that more people have had their kidney's stolen after meeting a pretty girl at a party than there have been disks recovered after being completely overwritten with random data.
Re:This should be a lesson... (Score:5, Interesting)
Re:This should be a lesson... (Score:5, Informative)
pv [ivarch.com] < /dev/zero > /dev/device is pretty nifty too.
Re:This should be a lesson... (Score:4, Informative)
Well, yes. And in fact due to the way data is encoded (MFM, RLL, whatever they use these days) a zero bit of data in a sector does not necessarily correspond to a physical zero bit in a magnetic sense.
And given that one of the theories about how to recover data is "subtract the 'perfect' waveform of the track from the actual waveform of the track, and the difference will be some indication of the data that was there previously", it doesn't matter if a single pass is random, all 1's, or all 0's. If you were doing multiple passes then random data would be better, but psuedorandom would probably suffice as long as it was different with each rewrite because the objective is to push the variations well under the noise floor.
When I was writing floppies under AIX about 10 years ago, 'dd' with a suitable block size was many times faster than 'cat'. Maybe it wouldn't have made a difference for a harddisk though.
Re:This should be a lesson... (Score:4, Interesting)
Assuming we are talking about a disk that has been entirely written with zero's or random data, eg a deliberate attempt to render the disk unreadable - citation needed (or are you just repeating something you heard from someone who heard it from someone else?)
The 'black boxes' are designed to ensure the survival of the internal medium, so it's no surprise that the data is recoverable (don't they use analogue tape on a loop? or is that just for the voice recorder? or is my knowledge way out of data :)
When inter track spacings were wider and density in other dimensions was lower (20 years ago?) it was possible to recover data after a complete write with zero's, but not now.
While Wikipedia isn't the definitive answer on anything, it clearly states in several places that a single pass of the entire disk is enough to erase the disk with no chance of recovery.
Sounds awfully like an urban legend. Are the illuminati involved somehow? :p
Re:This should be a lesson... (Score:4, Informative)
Police forces do not recover data from overwritten disks.
"Formatted" (quick format, destroying partitoin table) yes. Overwritten, no.
bullshit (Score:3, Interesting)
Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.
How about once? With zeros.
http://16systems.com/zero.php [16systems.com]
If you can retrieve you data from a drive after it has been dd'd with /dev/zero, you might be able to win this prize.
If you happen to be in the situation described, chances are you're fucked.
Re:bullshit (Score:4, Interesting)
Re: (Score:3, Insightful)
Re:This should be a lesson... (Score:5, Interesting)
Unless you have overwritten the area on the physical disk that contained the data, multiple times, the data can still be recovered.
The DoD spec is written as it is for a reason. Given a drive with confidential data on it, an unauthorized person attempting to access the drive does not need to get everything back to pristine condition. Even recovering a small part of the total data set can cause incalculable damage if it's the right small part. The value of sites like Avsim are in the whole rather than the sum of the parts.
I've recovered data off of formatted HDD's, off of corrupted file systems, off of compact flash cards and other media (Really useful if you want to keep those photo's that someone thought was deleted, be aware of this people).
There's a large dependency on what you're trying to recover off of. DOS/NTFS are fairly easy to do recovers from. The first character of the filename is zilched out and the rest of the data to find the file is left intact. UNIX/Linux filesystems are a bear. Once you hit "rm", you've lost the ref to your inode. Putting Humpty Dumpty together again at that point becomes nearly impossible because the record which shows where all the pieces are is lost to you. If you have known text from the file, and a good knowledge of how the filesystem works, you should be able to backtrack. Otherwise? God help you.
There's also an issue of how the data is stored. A single-drive system is fairly straightforward. 2 drives are harder. Once you get into a SAN/NAS where data is spread over multiple drives, recovery of even a single file with known text becomes tricky. Multiple files? Unknown data? The only hope I would see at that point is to put a large segment of the Slashdot community on the problem and tell them a large trove of high-res pictures of Natalie Portman completely nekkid are stored within.
Re:This should be a lesson... (Score:5, Funny)
I hear it's murder. ;)
Re:This should be a lesson... (Score:5, Insightful)
this really is a pathetic situation. Everybody is hammering these guys for just mirroring their data and saying that they should have had off site backup.........true, they should have. What really is the issue here is that ASSHOLES feel the need to attack for the sake of attacking a site. It would be like me going out and punching random people in the face just because I can.
We have to stand up for those that cannot stand up for themselves.
People that destroy just because they can are completely USELESS...............and should be SHOT.
Addition to the lesson... (Score:5, Funny)
To any sysadmins and DBAs...
Make sure you have offsite backups
Any person in the IT community who was alive to remember the events of 9/11 should have learned a valuable IT lesson from that event.
Repeat after me. I will not store my "offsite" backups in the other tower.
Re:This should be a lesson... (Score:5, Insightful)
How about we just shoot all hackers?
I'm not sure how that will protect against data loss from equipment failure, natural disaster, fire, software failure, solar flares, Secret Service, or really anything other than hackers.
Offsite, offline backups aren't a good idea solely to protect against hackers. They're a good idea to protect against data loss in general.
Re:This should be a lesson... (Score:5, Funny)
How about we start shooting people who can't recognize jokes. Sheesh.
Then who would mod for slashdot?
Re:This should be a lesson... (Score:5, Insightful)
What, you mean like this guy? [archive.org] You probably wouldn't even have the browser you're using right now if it weren't for that particular, uh. hacker.
Re:This should be a lesson... (Score:5, Informative)
Which reminds me. They could always use the WayBack Machine to (help in) retrieving their archives:
http://web.archive.org/web/*/http://www.avsim.com/ [archive.org]
Google Cache seems to archive only the most recent pages:
http://74.125.95.132/search?q=cache%3Ahttp%3A%2F%2Fwww.avsim.com%2F&submit2=Google [74.125.95.132]
Re:This should be a lesson... (Score:4, Interesting)
Surely all the people who've downloaded the downloadable content over the years can all band together and restore a large proportion of it?
I also thought: developers and user must have downloaded/synced great deal of source (binaries for sure :-).
So why not scramble it back together? As for the web site, perhaps pages are cached somewhere on the web, as some people already pointed out.
I've visited their site about a month or so ago, and they did really impressive work, they should try to restore it.
Re: (Score:3, Funny)
But then who's going to take out the Gibson?
Re: (Score:3, Insightful)
Or pay them to find shit like this before someone does this.
The logic behind "Destroy your only resource that can work to actually help you fix the holes that will be exploited by foreign hackers or terrorists" is completely beyond me.
In fact, it seems so utterly stupid that I get furious every time I hear some thoughtless moron spout "Punish the hackers". Suggesting they should be killed? I'd personally sooner keep those intelligent if misguided people--being the only ones that are really going to be use
Re:This should be a lesson... (Score:5, Insightful)
In fact, it seems so utterly stupid that I get furious every time I hear some thoughtless moron spout "Punish the hackers".
A little blame needs to come from all areas. Not every website or messageboard is run by someone with a CS degree with a minor in website security. A break-in of a government site or large corporate site is one thing, a family website another. This site is probably somewhere in between.
Saying it isn't the hackers fault that improper mehtods were used to secure a site is like saying it isn't the muggers fault that the lady's handbag was so easy to steal.
Re:This should be a lesson... (Score:4, Funny)
So, when rats attack your cellar, you pick the most intelligent and ask for advice?
Or just kill them by anything at hand.
Re:This should be a lesson... (Score:5, Funny)
So, when rats attack your cellar, you pick the most intelligent and ask for advice?
Yes, he said don't worry about it and go back to posting on Slashdot.
Re:This should be a lesson... (Score:5, Funny)
Oh, we can make a good car analogy out of this: Having a backup car in case your primary car crashes is a great idea (if you can afford it). Except that instead of keeping their backup car locked in the garage, these people attached their backup car to their primary car with a tow bar and dragged it around everywhere they went. When the primary car crashed, the backup ran into it a fraction of a second later. Now they're sad that their backup car is dead too and are somehow suprised they don't have anything to drive.
lesson is (Score:4, Informative)
Three words: (Score:4, Insightful)
There's a special place in hell... (Score:3, Informative)
Reserved for people who don't do archival backups, don't secure their systems, and then try to blame their ineptitude on hackers.
Do backups.
Do security.
Do restore from your backups to test them.
Do not blame others when it's shown you failed steps 1-3.
Re:There's a special place in hell... (Score:5, Insightful)
yes we had backups (Score:5, Insightful)
Re:yes we had backups (Score:5, Funny)
Wait, we have to care? I thought we were supposed to point and laugh...
Re:yes we had backups (Score:4, Insightful)
Absolutely, I mean, so what if those guys broke into your house and killed you and raped your mom *right in your own basement bedroom* ... y'know, you should have had better locks, and used them more consistently; y'know, if you'd really cared.
Sure, there are *much* better backup strategies; that having been said, somebody broke in and did a bunch of damage for shits and grins. They suck.
like the backups should have been (Score:4, Funny)
Offsite backups? (Score:5, Interesting)
I realize that from quite a few people's perspectives, storing their backups in a separate building constitutes off site storage. I'd almost buy that strategy. Not in the same environment, network, city etc.
These guys were stupid.
The day after 9/11 I was in an elevator, and caught a snippet of conversation between 2 people that had business interests with a firm that was in the WTC. The comment I heard was 'their backups were in the other building'. Another company lost.
You can never totally plan for every contingency, but you can insure yourself. I know many developers that take hard copies of their code (meaning on removable media) home just for this reason. I have seen sys admins do the same because they didn't trust their DR stratagy.
This was avoidable. This isn't even about disaster recovery. It is about business continuity.
You can't afford not to protect your data.
Re:Offsite backups? (Score:4, Interesting)
Re:Offsite backups? (Score:4, Informative)
They should be kept on a different part of the electricity grid, preferably in a differnt postcode.
It all depends on what kind of disasters you want your data to survive. If you want it to survive nuclear war, you need off-shore backup. Preferably in a neutral country that won't get involved in the war.
If you want your data to survive a Vogon constructor fleet, use off-planet backup. Recovering it from the brain of a single surviving human (if any) is going to be costly and painful.
Sigh. Mirror != backup (Score:5, Insightful)
Repeat after me: mirroring is not a backup. Backups are physically removed from the machine and stored where they can't be altered until they're needed for a restore. If they aren't removed from the machine, well, as we've just seen that only ends in tears. Observe their pain and learn from it!
So, they had NO backups? (Score:4, Insightful)
'Backed up between two servers'... that's not what a backup is.
I'm... astonished at the level of incompetence here. A site with 13 years of work like this, and they didn't bother to backup anything at all?
And now they're trying to handwave it away with 'oh uh, uh really folks, seriously, were really did have backups haha, between servers olol'.
I don't think 'olol' is going to impress anyone whos work was just wiped out by their incompetence.
Learn from Kuwait too (Score:3, Interesting)
All they had was a back up copy that made it out.
After the war they could go in and find what was tampered with. ie who got a false identity.
Take your data home with you every night.
And yet another example why you need real backups (Score:4, Insightful)
These aren't hackers (Score:4, Insightful)
Whoever did this must have willfully wanted to destroy the website and its content. Deleting data in this manner is far beyond vandalism or criminal mischief.
I hope the perps get served by a judge who recognizes just how severely malicious this was, and that enough of the people who used the site can provide the files back to the owners and the community.
Some backup stories (Score:5, Interesting)
A couple of years later, I am in the pub with some mates and John turns up. I ask him how he's managed to finish work and get to the pub so early. "I did a fast backup" he said. I was interested so I asked him to explain. "Oh, it's easy, get the target tapes from the rack, rub out the old date, write the new date, put them back into rack and go to the pub"
Worked for a large software shop in the 90's. I am part of a decent sized Oracle development (circa 50 devs). Ops decides that Oracles backup routines are too slow and 'optimize' them. Some weeks later - guess what - there's a problem and the backups are useless - No physical audit trail this time - the team has to redo all of there work - it was not good for the project budget, the team moral or the client
There's a perfectly good set of words for... (Score:4, Insightful)
...the thieves and vandals who steal data and wreck servers.
THIEVES AND VANDALS.
Not "hackers".
What was done was not hacking. It was vandalism. Plain and simple.
Hackers create. Vandals destroy. Thieves steal.
I'm surprised that this needs to be explained to the Slashdot community.
Real men... (Score:5, Informative)
"Only wimps use tape backup: _real_ men just upload their important stuff ;)"
on ftp, and let the rest of the world mirror it
Linus Torvalds Jul 20 1996, 3:00 am
Re:Real men... (Score:5, Interesting)
Hindsight is always 20/20 (Score:5, Insightful)
This is a lesson every system administrator worth his or her salt learns over the long haul. You might back up dutifully, test restore, and have a well done system of ensuring backups are rotated correctly. Then you find out the tape drive you use is miscalibrated so only it can read your backup tapes, or you find the backup software you use on a daily basis is not in production, or the latest version has no support for the backlevel formats.
I have found that in a production environment, you really need multiple methods for backup if at all possible:
The first level is a dedicated backup server. This machine is locked down to the best of your abilities, and firewalled from the network, only allowing critical ports such as what the backup software uses, and perhaps ssh or RDP (if a Windows box). This machine copies everything from the other servers onto a large disk array, then to tape. The tapes are then cycled offsite via a service like Iron Mountain. Of course, the tapes are encrypted, and corporate officers get a copy of the master keys.
Why tapes? Because they can be set read only after they are dismounted, and no computer, no matter how infected can modify or delete the tape contents once this is done, outside of a reflash of the tape drive's BIOS. This is important because its not unheard of for someone to write a program that trashes backups over a time interval. Higher end tapes can be used as WORM media like DLT-ICE.
I can't emphasize enough about securing the backup server, both physically and network-wise. If this box gets compromised, all your data is available. On Windows machines, I recommend using some form of disk encryption (Bitlocker if the machine has a TPM, TrueCrypt, etc) so if the backup server or an array gets physically stolen, the data is of no use to a thief. This is in addition to the backup program's encryption.
After you have a central backup server installed, secured (security is paramount on this machine unless the backup program client can do encryption), and backups running, you focus on the other levels of backup.
The next level of backup is on the local servers. Most operating systems have a method of backing up the computer. If you can do this with a server, fire off a snapshot backup every month or so. Most OS backup methods don't have encryption, so this backup should go directly to a tape safe or secured container in the data center. Optionally, you can install backup software locally that can encrypt. I like using the backup/restore utility the OS gives for an image every quarter, then using more secure software more often, so the OS backups can be stored in a tape safe or physically secure container. This way, if the third party backup software ends up inoperable, there is still a method of getting a machine up somehow, or putting it in a virtual machine for recovery purposes.
Finally, after you have backup servers and a rotation, companies might consider offsite cloud backup services like Mozy. Mozy offers use of keyfiles so all data is stored encrypted (encrypted on the client end). Of course, making sure the encryption key is stored safely is paramount, and the cost of storing a large backup in Mozy's cloud may be prohibitive. However, if worse comes to worst and your site is completely knocked out, as well as the offsite backup site, it may be thing that keeps your business up.
Of course, scale this up or down as per your company's needs. A smaller business can get by using Mozy and a Windows Server 2008 box running Bitlocker, a network backup program with encryption such as Retrospect or Backup Exec, and using external drives every month to copy backup sets from the main ones to store offsite.
A larger business might see about a true backup fabric system sold by IBM (TSM), EMC (Networker), or Microsoft's solution.
The key is to not just have some built in redundancy so if one backup method is not usable, you have another, even if the backups are older, but to be able to do this in a manner that doesn't add too much time and equipment expense.
Re:Hindsight is always 20/20 (Score:5, Informative)
And for those who don't like to pay $10000 for backup software, there's Bacula [bacula.org]. Couple that with an LTO-4 drive (~1000) and LTO-4 tapes (800GB uncompressed, ~60/piece) and you're set. Rsync.net is a decent, cheap online provider for those gaps when you haven't rotated tapes.
Bacula is pretty sweet because it lets you backup to disk volumes and then you can schedule a roll to tape. So you can just back everything up incrementally to a disk volume and then copy those backups to tape, and then run rsync on the disk volumes to have an offsite, online backup. When recovering, you ask to recover from whatever's available. If you keep enough disk storage around (and there's really no reason not to) you can recover to any date in the past. In the event of a disaster your tapes come into play.
Now with drives so cheap the temptation is to buy a external hard drive and use that. But tapes have a long history, guaranteed backwards compatibility (planned anyway, LTO drives have to R/W the previous generation and Read 2 generations back), last longer than moving drives, are simpler, lighter, more robust and more portable. Not that I wouldn't keep a external around to dump desktops but tape is the DR standard.
Of course I have an extra set of keys.. (Score:5, Funny)
Public Viewing (Score:5, Funny)
A public viewing will be available at:
http://web.archive.org/web/20080116064652/http://www.avsim.com/ [archive.org]
No date has been set for the funeral.
Data is NOT backed up until it is (Score:4, Informative)
- tested
- offline
- off-site
- several times
anything else is "high-availability", not "backup".
The people running the site ARE NOT IT Admins (Score:5, Informative)
Listen folks....we're talking about a couple of guys who spent their free time creating a website. They're not making any real money out of this (in fact, they all have regular day jobs).
They've been advertising for a Tech Manager (non-paid) for quite a quite so time now. They did get one recently...but it turns out the guy harvested the emails from the systems and sent out a bunch of spam. He has since been fired.Even though the avsim folks aren't saying it was him who hacked and destroyed their site, it's quite hard not to think it was him.
It's been quite a blow to the flightsim community and I have noticed a lot of IT folks are offering help.....I just haven't seen a single one on this thread.
Re:The people running the site ARE NOT IT Admins (Score:5, Insightful)
Re:Copying between servers is NOT backing up (Score:4, Insightful)
"I hope the same administrator will never again make the same mistake with backups."
He won't for this company, that is for sure.
Re: (Score:3, Insightful)
Honestly, how many man-hours and equipment do you really want to commit to backup? Do you really think it's worthwhile to get a tape system and regularly move tapes off-site for some community mods? Anyone can envision a system that is far more secure than this, but paying for it is another thing.
If the mods were good quality and downloaded often, the community should be able to act as a backup of sorts.
Re:the web is ephemeral (Score:5, Funny)
Maybe future historians will consider this a dark age, whose intellectual production was lost.
Please don't say our treasured facebook, twitter, slashdot posts, wikipedia revision wars and v1agra spam may not be preserved for posterity.
I'm not yet convinced that information that today exists only on the internet is really meant for eternity :)
Eternity (Score:5, Funny)
Only goatse is eternal. The rest is being used to seed a randomness generator somewhere.
Re:the web is ephemeral (Score:4, Insightful)
Wikipedia revision wars will be a GOLDMINE for future archeologist.
Think about just how much they reveal about a certain topic.
Re:the web is ephemeral (Score:5, Funny)
Wikipedia revision wars will be a GOLDMINE for future archeologist.
Think about just how much they reveal about a certain topic.
Such as the difference of opinion about the color [wikipedia.org] variations [wikipedia.org] of the carrot [wikipedia.org] !
Re:the web is ephemeral (Score:5, Insightful)
nonsense...
completely inaccurate guestimation, but probably only about 1% of anything carved in stone, is still decipherable or even exists, same with scrolls, otherwise we'd be littered with 2000 year old shopping lists, love letters, etc, how many notebooks (the paper kind) have you gone through during school, as journals, boredom... still have them all?
Hell, we probably only have about 1% of the stuff that was written down 100 years ago, probably only about 3% of the buildings, 0.3% of the cars, 2% of the paintings...etc...etc... most of the ancient books we have, are copies of copies of copies, and we can do that with magnitudes of efficiency now, not to mention recovery, hard drive gets erased, it's easier to get the data back than a scroll that's been erased, or a stone.
If even 0.1% of what we have on the internet right now exists in 500 years, it'll still probably be more than everything we have in stone, scrolls, and print right now...
With the various sorts of "Library of Congress" out there, if you had the chance to peruse and take/read whatever you wanted, you'd probably only find 0.5% of it interesting anyways, much like what's on the internet.
Re:Really? (Score:5, Funny)
Anyone who hacks a flight-sim sight has no life and really needs to get laid.
Coming from a slashdotter that is pretty rough.
Re: (Score:3, Funny)