Data Loss Bug In OS X 10.5 Leopard

An anonymous reader writes "Leopard's Finder has a glaring bug in its directory-moving code, leading to horrendous data loss if a destination volume disappears while a move operation is in progress. This author first came across it when Samba crashed while he was moving a directory from his desktop over to a Samba mount on his FreeBSD server."

That's silly.

Normally while moving you ensure the copy completed before deleting the original. Apple must be using some discount programmers.

Ah, the "outsourcing" coding model..

Advert on Amazon Mechanical Turk:
Write OS-X compatible application to Move a file between two filesystem devices..
Time Allotted:: 6 hours. Reward: $10.00..

Re:Ah, the "outsourcing" coding model..

Um... no. REALLY no. Please don't do that.

The syntax for "||" is:
If command 1 fails, do command 2 - otherwise exit (where you used "command1 || command2").

In this case, your command will either copy all the files from $from to $to or delete all files at $from.

What you probably meant is cp $from $to && rm -r $from, which only performs the second command if the first succeeded. This solution is far from perfect for reasons mentioned by other posters, but it's still significantly safer and more useful than yours...

Re:Ah, the "outsourcing" coding model..

And that's probably why you don't work for Apple!

Re:Ah, the "outsourcing" coding model..

rm $from Now where is my $5?

Re:That's silly.

Apple must be using some discount programmers

Of course not! Don't be a troll.

Everyone knows that Apple's products Just Work, and that's no different in this case. The files were moved just like you asked, and if you can't find them. well, that's not Apple's fault, is it? You don't blame the contractor who built you home when you lose your keys, do you?

In any case, you should be using Shadow Copy...er...Time Machine which would have protected you from going and losing track of your own files.

Temporal problem

In any case, you should be using Shadow Copy...er...Time Machine which would have protected you from going and losing track of your own files.

Great! So now not only don't we know where our data is but when it is. Perhaps in a week or two's time the data will materialize in the folder it was supposed to be moved to with an accompanying "whorping" sound coming from the speakers?

Re:That's silly.

So what happens if you're moving a 120GB folder one directory level on a 150GB disk?

Typically if you are moving within the same logical device the file pointer is moved and no copying need take place.

When moving to another device your code reads and writes, within a loop and traps exceptions (such as the device suddenly vanished, where the OS should raise an exception and your application traps it.) A wide variety of errors could occur while moving and in the event any of them happen the user should be notified in an appropriate manner and the original data not deleted.

I've written a number of applications which moved files or data between databases and it's fundamental your application is on the watch for any problems. Not to have an exception raised or to trap any and all, well, that's simply an inexcusable lapse.

This sort of thing is extremely critical if you happend to be defragmenting a disk drive. Long before Macs and PCs we had to defrag our mainframe drives and the applications which did the work were quite careful. Often the best practice, if you had the resource of a second drive, was to simply defrag to a new drive then re-assign the new dist as the original.

Re:That's silly.

Why don't you just use rsync [anu.edu.au]?

Re:That's silly.

Oh yeah, rsync. Is that one still broken on 10.5? Apple's build of rsync on 10.4 consistently choked on a few files when my dad started using it on his Mac Pro.

Re:That's silly.

resource fork... from Apple's rsync manpage: -E --extended-attributes copy extended attributes, resource forks

With cryptic commands just to copy files Apple will ALWAYS have ~.001% market share. When will Apple get it into their head that programmers shouldn't design user interfaces?

Why doesn't Apple just copy the way Ubuntu does it and get with the 2000's?

Pride is what it is. That damned pride!

Re:That's silly.

Isn't this a Mac? I thought they were supposed to be easy to use.

Re:That's silly.

There have been resource fork patches for some time, but somewhat unreliable. Version 3.0 is going to support resource forks (and other types of extended attributes) out of the box. The setup we have been using extracts resource forks to a separate file on the mac and then backs them up alongside the original files. The restore process just performs the reverse. It will be nice once we can switch to rsync 3.0 and get rid of that step.

Re:That's silly.

this is not what current generation of typical user would do, but I believe they should be educated on this anyway

Reeducate the user, you say. Surely you must be joking, right?

Let's ignore for a moment that Leopard may have a few bugs that will have to be ironed out. That's only to be expected with *_any_* newly released OS and the reason why no sane person would ever dare to update the OS on a mission critical machine within the first few months of the release.

However, if you can't rely on your OS to perform a simple file move without risking data corruption, then the right solution is definitely not to verify every single operation by hand. Automating tedious tasks is exactly what computers do best, and that the OS ensures the integrity of the copy before throwing away the original is definitely something you should expect.

Re:That's silly.

I think you misunderstand something.

No, I understand perfectly.

I know for a fact that I've successfully canceled FTP-moves on Windows (brand); I also know for a fact that I've successfully canceled FTP-moves on all modern instances of the Windows OS with various FTP clients (programs).

I'm not talking about other clients. I'm talking about using Windows XP as the client.

What program were you using that fragged your FTP server directory exactly?

Windows XP, service pack 1.

I suspect you're a fud-fucking idiot

You need to work on your social skills.

although at the moment, I'm interested to hear if there's more than fabrication here. Pray tell: how can I reproduce this bug?

You might need to use WinXP service pack 1. I'm unable to reproduce it now on my home machine which has SP2 installed. At work we've just upgraded to service pack 2 within the last six months. Anyway, about a year ago I was trying to move nearly 40 gigs of data from a Solaris server at another site to my local machine. The Solaris server was an FTP server and that's the only access I had to it. I opened Internet Explorer and typed in ftp://servername [servername] and was prompted for a username and password which I entered. Then I saw the files in the explorer view just like I see files on my local machine. Since I was moving so much data I decided to do it one top-level directory at a time. To move the files I selected the folder, dragged it to a folder on my local machine, and held down shift when I let go of the mouse pointer so it would move instead of copy. I noticed that Windows went through three steps when it moved from the FTP site:

1. A dialog appears with the title Moving... and it says it's calculating how much time it'll take. This took a while itself.
2. Then the dialog says it's moving the files and the progress meter shows the progress
3. After the progress meter fills up, it goes to empty again and then says it's deleting the files it moved

I did this with most of the folders until I reached one that had about 18GB of data in it. I performed the same procedure as above to start the move and after it was copying files for about a minute or two I realized that I was moving them to the wrong location (my desktop instead of a directory on my C drive). At work our desktops are retargeted to a file server and not our local hard drives. I knew that by copying to the desktop I would hit my quota limit on the server whereas if I had copied to the C drive like I did with the other folders I had no limit aside from the free hard drive space. Not copying there in the first place was just a mistake. I clicked cancel to stop the transfer. When I did that, the dialog that had the progress bar changed to the empty progress bar and said it was now deleting. By the time I realized what was going on, it had deleted that folder and all of its content from the FTP site.

Now this could have been some weird bug or interaction between the fact that I was using a machine with SP1 instead of SP2, that my desktop and profile were retargeted to another machine, or that I was moving so much data. It wasn't a lot of files as these were data files for desktop publish programs for some brochures and catalogs, along with large print-ready images. I don't know what items worked together to cause the problem. I do know that it did happen and that I had to deal with the IT group at our other site (that had the Solaris server), open the helpdesk ticket to get them to restore the files from backup, wait for them to get around to it, etc. It was a huge pain and delayed what I was working on, causing grief for myself and my internal customer. One thing's for sure; I'll never use the built-in Windows FTP client again.

Tiger has this problem as well!!!

I lost a huge amount of data being MOVED (not copied) from one volume into a virtual volume DMG file. Lost and gone forever, lots of important files. What happened? I simply bumped the laptop Mac Book Pro during the move... zap... gone forever. The DMG file was blank! Yes, complely zero bytes except for a bit of header non-file data. It sucks bad.

Re:Tiger has this problem as well!!!

Just wanted to also confirm that the bug was in Tiger. I was backing up music files to do a clean format for Leopard and lost everything when the hard drive got disconnected by mistake.

Re:Tiger has this problem as well!!!

I've had that also (on Jaguar, IIRC) when an external drive lost power. Lost both the data and the old backup, simultaneously.

File The Bug!!

Go Here: http://developer.apple.com/bugreporter/ [apple.com] And File a bug. And They'll fix it.

Re:Or [possibly], go fix it.

Apple charges good money for Leopard, so let them fix it.

Re:Or [possibly], go fix it.

Finder, and any GUI parts of OS X in general, are not part of Darwin.

Re:Tiger has this problem as well!!!

Dude, Mac users are just getting this kind of functionality? Come on, we've had that feature in Windows for years...

Re:Tiger has this problem as well!!!

Can't agree with you. It is the exact type of that is usually caught by automated testing. The issue isn't that a hard drive was bumped. The issue is that the write operation failed. In this case due to a drive no longer being accessible. The failure is easily automated, and the result of that failure is easy to catch.

And I wouldn't exactly call this regression testing, as such functions as file movement aren't usually impacted by later changes. It should be pretty basic on the design chart. Sounds to me more like "working as intended...use move at your own risk". Which I think it stupid, but I don't see how this really was *missed*, especially since some are claiming it's been this way since at least Tiger.

Re:Tiger has this problem as well!!!

So your irreplaceable videos were made the previous year and you had no backup.

I suppose the PC makers should like you. You'll shoot up a PC and have to buy a new one every couple years. And each maker will get a piece of the action, since you'll blame the previous maker for routine data loss.

Or you could just do what everyone in the computer business has been telling you to do for at least 30 years and... make a backup.

Wait...

It's kind of strange that this didn't come up while people were beta testing OS X 10.5. Samba is used in many places. I hope they get it fixed soon.

Exactly which decade did you fall into your recently awkened from coma in? Testing? Testing? Nobody tests anything anymore, they just go play with all the new toys and stare at the eye-candy. Actual mundane, humdrum testing? That's an SEP if ever I didn't see one.

Re:Wierd

All fanboys were just happy with how blazing fast file copy was compared to Vista. The non-fanboys tried to check the real size of the dir by copying it to a Vista machine afterwards, but the progress bar got stuck on 413 hours left and counting, so they couldn't file the bug in time.

Re:No it isn't

You have to pay to be able to participate in the program

Wrong-o!! You absolutely do not have to pay anything to involved in the Appleseed program.

Re:No it isn't

Someone already mentioned that the bug is not in copy or delete functions of the kernel - it is in the Finder GUI that is not open source.

A great disturbance in the Apple

...As if millions of fanboys suddenly cried out in terror and were suddenly silenced.

Re:A great disturbance in the Apple

Yes, but if their data is stricken down, it will become more powerful than we can possibly imagine.

Re:A great disturbance in the Apple

In memory of them, I'm officially nicknaming Leopard: Vistapard.

"Vistard"

Terrible bug

But entirely fixable.

Knowing it exists means you can also work around it by doing copies every time and only manually deleting the source entry when you're sure the copy has completed properly. Now all you need to do is be mindful every time you want to move a file anywhere.

Sucks to be a Mac user (I am one, I know) but I'm sure Apple will have this fixed pretty quick.

Re:Terrible bug

Yes, that's easily avoidable. If you know it exists. And hopefully the first time you get bitten by this it isn't something critical. Would be garbage to have to work around that one.

TFA looked decent in it's details. Even step by step recreation. But it's a pretty serious bug, that as you mention, *needs* to be fixed quick and I didn't see any other sources confirming it.

Re:Terrible bug

You're asking if a bug wherein entire folder hierarchies can go *poof* in the event a network share drops should be considered critical? Are you serious?

Re:Terrible bug

is this even that critical? Doesn't leopard have the time machine in it? Can't you just go back and get your files out of the time machine if they were that important?


Only if a backup of the files was run, which is a requirement of Time Machine.

If this was Vista, then there would be a good chance to use 'previous versions' to recover the folder data, as it does not 'solely' rely on external backup for timeline file recovery. Vistas use volume level file version snapshots(a feature of NTFS that HPS+ doesn't support), so there are backups even on the drive if it hasn't ever been backed up.

(Remember Time Machine usually only runs once an hour, and all versioning or changes made in that hour are never kept or tracked.)

-PS Not trying to troll, but this is a perfect example of the difference between Vista's previous versions and Apple's Time Machine I have tried to point out in the past.

Vista does both volume level backups and external backups, unlike just external backups like Time Machine does.

See why IT people prefer Vista's method, even if a backup hasn't run that hour, even users themseleves can access files and folders rather easily that got deleted or changed. And since this has been around Since Windows 2003 Server, in corporate environments, even XP users accessing 2003 servers have had this feature for over 4 years now.

Vista's claim to fame is that it enables these features on the local hard drive, and also integrates with the Vista backup system, so file versions appear from the backups in addition to the 'snapshot versions' on the main hard drive.

That's missing the point...

Speaking as one of those IT people, NTFS is probably one of the coolest pieces of software ever to come out of Redmond. ACLs, alternate data streams, directory junctions, single-instance stores, shadow copy, the list of useful features is huge. Even more surprisingly, it works pretty much as advertised. Frickin' cool.

There's another angle, though. On paper, Vista's NTFS-based backup technology walks all over Time Machine. However, the USABILITY of Vista's technology is crap. This morning, I enabled Time Machine by plugging in a USB drive and clicking "Use as Backup Disk" when prompted. To do restores, I launch the cleverly named "Time Machine" application. I've already used it twice today just because it's fun to watch the spacey animations.

Compare that to Vista's clunky "Backup and Restore Center", which you have to use if you want to backup your files on an alternate volume. I guarantee you that using "Backup and Restore Center" is beyond most average users. Sure, it might be "better", but what good is it if it never gets used?

Re:Terrible bug

Actually, Time Machine runs every two minutes. You can see a countdown timer in the system preferences. Also, Time Machine is dodgy by itself. Apple confirms that after backing up about 10 GB of data, Time Machine may stop working. See here: http://docs.info.apple.com/article.html?artnum=306932 [apple.com]

Re:Terrible bug

but I'm sure Apple will have this fixed pretty quick.

Well, the problem is, this bug exists even in TIGER and has been repported many times! And no, not fixed yet. Abble is a coorporation like any other, and not the superpower that some users seem to think they are.

Re:Terrible bug

Error code 51 doesnt neccesarily mean the transfer failed. It can also mean lost connection to the share - which could have happened (in OSX's "mind") after it thought/was told the transfer was complete.

For instance, what if in stopping the share on the Windows session (incorrectly listed in the /. article as a FreeBSD Share), the SMB crap in Windows is "cleaning up" while shutting down and generating a "complete" code that it sends back to the Apple machine, which then in trying to communicate with the share, realizes it no longer can, and generates the correct error code as noted (51).

OSX did NOT generate a disk write error or any other error that would have been more applicable (like 43 or a bunch of other possibilities). It generated an error stating it lost communication with the other machine.

Again, still not enough info without knowing more about what the WinShare does when closed.

You just don't get it...

This is the new Leopard "iLostMyFrigginFiles" feature, next version they will add a badass black hole effect when it does that!

Re:You just don't get it...

Oh great! Soon, Windows users will experience "white holes" where Mac files magically appear in the My Documents folder.