Vista Runs Out of Memory While Copying Files

ta bu shi da yu writes "It appears that, incredibly, Vista can run out of memory while copying files. ZDNet is reporting that not only does it run out of memory after copying 16,400+ files, but that 'often there is little indication that file copy operations haven't completed correctly.' Apparently a fix was scheduled for SP1 but didn't make it; there is a hotfix that you must request."

That's OK then

the box I "make use of" has just 15,000 mp3s...

Billy G says

16k files should be enough for everybody.

Re:Billy G says

Found the quote on wikiquote [wikiquote.org]:

I laid out memory so the bottom 640K was general purpose RAM and the upper 384 I reserved for video and ROM, and things like that. That is why they talk about the 640K limit. It is actually a limit, not of the software, in any way, shape, or form, it is the limit of the microprocessor. That thing generates addresses, 20-bits addresses, that only can address a megabyte of memory. And, therefore, all the applications are tied to that limit. It was ten times what we had before. But to my surprise, we ran out of that address base for applications within--oh five or six years people were complaining.

Re:Billy G says

Funny, I thought he just bought the OS from someone else - didn't lay it out at all.

Re:Billy G says

The problem was that the operating system's facilities for addressing the hardware - in particular the screen - were so piss-poor that programmer's had to address the hardware directly in order to achieve reasonable results. There weren't even system calls to allow the software to discover dynamically where the hardware was - the locations had to be hard coded. As a result it then became impossible to move on to a new generation of hardware (with a different limit) because the old software then wouldn't run any more.

No. That was not the problem. The problem was that DOS programs were 16-bit real mode programs. This means that they used 16-bit pointers to refer to memory locations. This is what limits a DOS program to 1 megabyte of memory, not any deficiency in MS-DOS (which it had many of, admittedly). The segmented perversion of 8086 made things even worse by making memory divided into 64kB chunks rather than contiguous.

In any case, as time went on, most DOS programs did move to next-gen hardware, first by using EMS and XMS memory, and later by using DOS extenders to run in 32-bit protected mode. Having fixed screen memory location was never the problem, quite on contrary: it made it possible to access the video card memory directly from protected mode without having to convert a 16-bit pointer from DOS into 32-bit one.

1) Methods to drive the screen purely by system calls, without having to be aware of the hardware at all.

We are talking about unaccelerated graphics card here. The fastest way to use them was to write directly to memory. Going through a system call would not only have been slower, meaning no one would had used it, but required said operating system to contain some kind of graphics driver, which would had taken up precious memory space and therefore hindered every program.

It's not as if all this wasn't known at the time. Other earlier offerings provided all this and more, but MS-DOS really didn't deserve the title of "Operating System" at all. It was a filing system and process loader with a few extra bits cobbled onto it to avoid being prosecuted under trades description laws.

DOS is perfect for what it's designed for - a filing system for two 360 kB diskettes that takes up little memory and doesn't get in your way, and lets you get your program into the memory. Of course a system resulting from these design parameters doesn't work too well in a modern machine with 500 GB hard disk, gigabytes of memory and a dazzling array of extension cards.

And, frankly, I doubt anyone at either IBM nor Microsoft realized that the IBM PC would still be in use, extended beyond nearly all recognition, 26 years later.

Re:Billy G says

And arguably, they never have.

Zing!

Re:Billy G says

32 bit extensions and a graphical shell for a 16 bit patch to an 8 bit operating system originally coded for a 4 bit microprocessor, written by a 2 bit company, that can't stand 1 bit of competition perhaps?

Welcome to Windows Vista

At the end, there will be free therapy. And Cake!

Re:Welcome to Windows Vista

the cake is a lie
the cake is a lie
the cake is a lie

Figures...

And that is one of many reasons we are all still running XP

Re:Actual info...

actually, fta:

Although the problem occurs where users are running Kaspersky security products, it's a kernel leak that lies at the root of problem (the problem's not confined to systems running Kaspersky software, that just that this application seems to exacerbate the issue).

Re:Actual info...

Actually, the bug is in the shell, not the kernel and only files with altnerate data streams trigger the leak. The KB article that Adrian links to states that very clearly, but he's been on an anti-Windows rampage lately that's blinded him to the facts.

Very few files have data streams, so the vast majority of users won't ever see a problem. Kaspersky choses to pollute every single file with a stream, however, which is why systems with it installed exhibit the problem.

Re:Actual info...

Very few files have data streams, so the vast majority of users won't ever see a problem. Kaspersky choses to pollute every single file with a stream, however, which is why systems with it installed exhibit the problem.

So it's Kaspersky's fault that alternate data streams are apparently no longer supported by Vista, despite being a basic part of NTFS?

Re:Actual info...

First of all, the issue is how Explorer handles extended attributes (EAs), which are distinct from alternate data streams (ADSes). The kernel and NTFS have always provided full support for EAs and ADSes (since NT 3.1). Explorer (and for that matter Win32) has never had very good support for ADSes, and almost nonexistent support for EAs. EAs were implemented in support of the OS/2 subsystem. ADSes are the 'official' way to attach metadata to a file, and scale better than EAs. The only Win32 functions that have ever provided access to EAs are the BackupRead and BackupWrite functions which are designed to handle all metadata on a file transparently. Looking at the imports from shell32.dll to ntdll.dll on Vista, it looks like the shell bypasses Win32 when dealing with EAs, invoking the syscalls NtQueryEaFile and NtSetEaFile directly (bypassing API layers like this is something Microsoft tells ISVs is a big no-no).

This is just Yet Another Windows 95 shell bug (yes Vista uses the same shell architecture ported through each version from Win95). It is not the end of support for EAs or ADSes. If anything, it's a belated attempt at better support, done poorly. The shell has always been, IMO, one of the lower quality windows components, especially when it comes to properly interfacing with lower layers. This bug does not surprise me. I've been using robocopy for nontrivial file transfer for a while now.

Re:Actual info...

only files with altnerate data streams trigger the leak

Well that's what you get for crossing the streams. Egon warned us. Kaspersky's risking total protonic reversal. I guess they were fuzzy on the whole good/bad thing.

"Waiter, there's a fly in my soup."

"Very few files have data streams, so the vast majority of users won't ever see a problem. Kaspersky choses to pollute every single file with a stream, however, which is why systems with it installed exhibit the problem."

Yeah, that's the typical Windows world attitude.

The operating system is specified to do certain things. It doesn't do them. Well, if not many people use this feature, so what? One of the way we make the feature list long is by including lots of features that don't work, but we figure nobody will use them and nobody will find out...

"Waiter, there's a fly in my soup."

"What kind of soup?"

"The orange scented celery puree. [thesoupspoon.com]

"Oh, hardly anyone orders that. You should expect flies in it. It's your own fault for being foolish enough to order it."

Re:"Waiter, there's a fly in my soup."

If other OS's do it the Microsoft way, it it still the wrong way.

First you implement and fix basic OS functions, like file copy, etc, so that it works correctly.
Then you implement and fix fancy stuff that most people want and use so that it works correctly.
Then you implement and fix fancy a few people want and use so that it works correctly.
Then you go about implementing and and fixing stuff that almost no one use.

Implementing all at once in a way that doesn't work, then fixing fancy stuff most people use, then fixing basic OS functions, then fixing what a few people use, is the wrong way to go about it.

Re:Actual info...

The underlying problem is a Windows OLE component memory leak. Microsoft has a hotfix for the issue at http://support.microsoft.com/kb/942435/en-us [microsoft.com]

OLE mem leak; only affects 'extended attrib' files

According to the cited "hotfix" link, http://support.microsoft.com/kb/942435/en-us [microsoft.com] , the problem is due to an OLE memory link when dealing with files that have "extended attributes".

This problem occurs if the following conditions are true:
  * The files include extended attributes.
  * You copy lots of files in a single operation.

CAUSE
This problem occurs because of a memory leak in the Windows OLE component. This memory leak is triggered by the way that Windows Explorer deals with the extended attributes of the files.

Re:Vista

I have 13K+ music tracks on a backup disk. If I try to copy them with the Explorer UI, it does nothing - No error message or anything. I reverted to Robocopy, which works fine. You must be doing the same thing. Doesn't anyone at Microsoft have a big music collection to copy, or do they just use their Macs and iPods for that? ;-)

Re:Cumulative copies!

If you are using Vista as a server, you pretty much deserve what ever happens to you.

I'm a little suspicious

The reason being is I've setup a Vista system and copied about 100,000 files (totaling about 60GB) drive to drive in a single operation, without error. So while I'm not saying this isn't a Vista error, I'm wondering what else has to be done to trigger it. The persisting across reboots, even if you break it down smaller really makes it sound like another program is somehow interfering with the copy. I'll have to mess around with it at work, we have Vista test machines and Cadence installs north of 250,000 files when you install its libraries. I know it installs fine, though that isn't a copy strictly speaking as it is files being extracted from archives.

I'm just wondering if perhaps there isn't more to this than just "OMG Vista runs out of memory!" If it is a memory issue, why then haven't I encountered it, doing far larger amounts of files?

Re:I'm a little suspicious

KB 942435:

This problem occurs because of a memory leak in the Windows OLE component. This memory leak is triggered by the way that Windows Explorer deals with the extended attributes of the files.

Its only files with streams, and apparently kaspersky makes it wose by that fact that it tags every single file with a stream.

Refresh of an oldie...

I don't want to start a holy war here, but what is the deal with you Vista fanatics? I've been sitting here at my freelance gig in front of a Vista PC (an Intel Core 2 Duo w/4 gigs of RAM) for about 20 minutes now while it attempts to copy a 17 Meg file from one folder on the hard drive to another folder. 20 minutes. At home, on my ancient Mac running OS 9, which by all standards should be a lot slower than this Vista PC, the same operation would take about 2 minutes. If that.

In addition, during this file transfer, Firefox will not work. And everything else has ground to a halt. Even Notepad is straining to keep up as I type this.

I won't bore you with the laundry list of other problems that I've encountered while working on various Vista PCs, but suffice it to say there have been many, not the least of which is I've never seen a Vista PC that has run faster than its Mac OSX counterpart, despite the Vista PC's same chip architecture. My 286/12 with 2 megs of ram runs faster than this 2.4ghz mhz machine at times. From a productivity standpoint, I don't get how people can claim that Vista is a superior operating system.

Vista lovers, flame me if you'd like, but I'd rather hear some intelligent reasons why anyone would choose to use Vista over other faster, cheaper, more stable systems.

For those that aren't getting the joke...

The original rant may be found here [kottke.org].

It's because

they can only send 16,000 files to the RIAA and MPAA to check, at once.

Exxon Val-Vista

Oh, the fools! If only they'd built it to let you copy 16,401 files!

What about those of us who need high performance?

I want an OS that lets me re-organize my pr0n anytime I want. I *need* to be able to select 50K-100K files at a time and move them from place to place without slowdowns. Ever try, in Windows, to search your network for all the *.jpg files, select a few hundred thousand of them in the search window, and drag them to the new firewire disk you just plugged in? It's painful, lemme tell ya.

Anybody want to suggest an OS that would work for me? I'm serious.

Of course file management is secondary...

.. when your GUI is using 2 gigs of RAM.

Bad summery

Apparently the submitter skimmed the article, and decided to post up a Vista bash on Slashdot.

FTA:

The "Out of Memory" error (which is affectionately known at the PC Doc HQ as the "Out of Cheese" error ... don't ask why ...) is one of the biggest and most baffling of Vista's file handling problems has been occurs when a Vista user (running Kaspersky Anti Virus 6 or 7) tries to copy a large number of files (~16,400)

Apparently its just a problem with this antivirus program running in Vista. I move large amounts of files around in Vista quite often (granted, its Vista 64), sometimes well over 20,000 files at a time, and have never run into this issue.

Re:Bad summery

Lol the TFA is FUD. Read the HOTFIX notes which explains that the issue is with Windows OLE (NOT part of the kernel) and files that utilize extended attributes. Note that the crappy AV product adds extended attributes to all of your files (which i'm sure speeds up every file operation on your PC), thus with a kapersky infected computer - you are assured to have the problem. With "normal" files it's unlikely you will have this issue. Media files and the new office files are more likely to pose a problem than your standard files.

The article does not state clearly wether physical memory is a constraint.

Not Just Vista

I don't think there has ever been a version of Windows that could deal with large numbers of files. Particularly if you are using the GUI interface. The whole thing is a toy operating system, really.

A few years ago, while investigating a similar problem with a production server (a SERVER not a client machine) the machine would gradually grind to a halt doing the copy, while still responding (but slowly) to other operations.

I found that the "copy" command did much better than a drag and drop operation, but still would have a problem eventually. Finally, I found that this was a known problem, and that to solve it, a dedicated MS employee had written a utility called "robocopy" the "robo" not being for "robot", but for "robust" (really, it said that!).

Using that usually got the job done, much more slowly than it should have, but at least I didn't have to re-boot the machine daily to clear things up.

Now that Gates is too busy with other things to take tours of the data center, really, Microsoft should do itself a favor and ditch the VMS underpinnings of Windows (some of which they have probably forgotten how to maintain) and build your nice GUI on top of BSD or something similar. That way you won't break your budget (in manpower and electricity) trying to match the Google server farms.

Once that's done you will have the experience needed to do the same on the desktop. You will be doing the world, and yourselves a favor. Thanks in advance!

Oh quit whining

This is a minor problem, absolutely rare event, occurs with next to no regu...

**OUT OF MEMORY ERROR, SYSTEM HALT**

When will people learn to buy the cable?

"It appears that, incredibly, Vista can run out of memory while copying files."

That's not really the incredible part. What is incredible is that Vista can copy any files at all without buying a special cable [pricegrabber.com]! This is what you get when you hack Vista and just start copying files left and right without buying the cable. If you buy the cable, it doesn't run out of memory until after 16,500 files!

Possible workaround: Use ONE file for everything

I expected this very problem to happen and have taken corrective measures. Simply, I use one large file for everything.

Think about the innovation that is being created here. I can access all the spreadsheets I have ever created whilst updating my current webpage project, search for an e-mail archive and read the latest TPS Report Coversheet without changing files. It means I don't have to partition my hard drive. And I only have *one* file ever to backup.

This has simplified my daily work to the point I fired all my IT staff. Thanks Microsoft!

Who can copy 16,400 files anyway?

Honestly, how can [Cancel or Allow] anyone co[Cancel or Allow]py that m[Cancel or Allow]any files anyway?

Re:Maybe this stems from...

Isn't it a little odd that if you strip off the first and last digits of the number "16,400", it's 640, as in 'no one needs more than 640k"?

Re:Maybe this stems from...

Actually, it's more like 90 / 90000, but you're only a few orders of magnitude off.

Not necessarily his fault, maybe he used Excel to calculate it...

Re:Maybe this stems from...

Yes, but everyone knows one-in-a-million chances come up nine-times out of ten.

Re:Maybe this stems from...

and then you take the 640, divide by 2, get 320, remove the last 0, and get 32, the reverse of 23.

Then you multiply 23 by 30, get 690, take 42, reverse it to 24 and subtruct this from 690. You get 666.

What does that tell you? Ha? Ha?

Re:Maybe this stems from...

What does that tell you? Ha? Ha?

You've got too much time on your hands?

No more going back to XP?

M$ is scared that people will try to copy their documents to another computer before reverting back to XP. Smart, very smart Micro$oft! On a tech note, what kind of number is 14,600? I would have thought 16,384 would be better.

Re:No more going back to XP?

> I would have thought 16,384 would be better.

It probably is 16384.

"16400" is clueless technical writerspeak for "The developer said '16,384', and the style guide says to use three significant digits".

(Alternate explanation: "The developer said '0x4000', and the style guide says 'convert to decimal' as well as 'if it's not a round number, use three significant digits'")

There are enclued technical writers, but 16400 is so close to 16384 that it makes me suspect that the author of the MSKB article isn't one of them.

Re:Just wondering...

Maybe you're backing up to an external hard drive?

Re:Maybe this stems from...

How the F%$^ can this be a problem? A file copy is a simple operation. There's simply no excuse for this... This should have never been a problem in the first place. What pisses me off is that I need to buy a new laptop, Vista is now forced down my throat, and I have no option to get XP pre-installed.

Re:Maybe this stems from...

I can make XP croak as well copying huge files from a CD or floppy which is handles very very badly (see 98). And my biggest drive is 20G (albeit a damn fast one). It seems to do ok copying big files from hard disk to hard disk but even with SCSI RAID with huge caches and the correct drivers you can't expect much left of your CPU when its doing this. Do two at once and you may as well go rebuild your transmission while you're waiting. Apparantly DMA and interrupts are unknown concepts at Redmond; PDP-11's did this just fine (unless you turned off DMA and interrupts in which case it was no faster than a 4Mhz Z-80 CP/M system)

Actually, a lot of the problems I've noticed with XP is related to the stupid fucking way that Windows handles it's file cache. It will literally swap out PROGRAMS YOU ARE ACTIVELY USING to expand the file cache during a large copy/read operation.

Anybody that has ever tried to alt-tab while copying huge files knows about this.... then you sit and wait for the pages to be swapped back into memory. And you might as well get some coffee, cuz with the hard drive already being pegged for the copy operation, it's gonna take awhile. Oh, and once it's finally done and you need to alt-tab back to the original program.... well, hope you need more coffee.

Lately I've been playing with a program called CachemanXP. Google it. It seems to give you more control over the memory and process management functions of Windows. It also lets you do a 'kill -9' equiv, which (as far as I'm aware) even Task Manager won't do, as it insists on trying to do a graceful shutdown first.

Re:Maybe this stems from...

For those situations...

Run -> "cmd" -> del %dir\*.* /s

It will clear most stuff and you will see error messages fly by... redirect output to a file for later examination if desired.

I use the good old 'del' whenever I know I will be deleting something like 20k files and do not wish to waste time waiting for windows to prepare for that operation... why the heck does Windows need to scan directories to be deleted before deleting them is beyond me, just delete them and be done with it. Same thing for copying, Windows wastes time scanning the source directory for no apparent reason since it won't tell you you have insufficient disk space to complete the operation until the target drive runs out of disk space... or any other errors for that matter, until it runs into them while carrying out the actual operation.

Linux has quirks, so does Windows. Linux has the excuse of being an relatively immature desktop OS but on the Windows side, it can only be written off as the result of half-ass design decisions.

Re:Maybe this stems from...

why the heck does Windows need to scan directories to be deleted before deleting them is beyond me (...) Same thing for copying, Windows wastes time scanning the source directory for no apparent reason since it won't tell you you have insufficient disk space to complete the operation

The scan is to know beforehand the size and number of files before copy/delete operation, so windows can make a <sarcasm>VERY RELIABLE</sarcasm> progress bar, with a <sarcasm>VERY RELIABLE</sarcasm> estimated time that <sarcasm>ALWAYS</sarcasm> measures correctly the time left for the operation to complete.

Re:Maybe this stems from...

Umm...no offense, but this isn't exactly a problem the average user is going to encounter.

That sort of depends if you are talking average home user or average work user. The average home user may actually have this kind of problem - since downloads to the tmp directory are then copied to the correct folder once downloads are complete. Update EQII, WOW & FFXI & you've gone a long ways towards 16K files. Add in patch Tuesday, and your average user is probably going to hit real close to 16K files if they try to keep the PC up for a month.

I probably come reasonably close to 16K files copied in a week on my work PC, so a crash like that would hit me every other week or so - not something I would consider 'Enterprise Ready'.

MS has a habit of programming for the home environment & pushing it into the Business environment.

Re:Maybe this stems from...

It's forced down your throat because you couldn't be bothered to source a laptop from anywhere but CC?

Give me a break.

Not really that simple in Vista...

It's not a simple operation. Think about how a file copy works within Vista:

1. The file is opened.
2. The file is scanned for viruses.
3. The file is scanned for adware.
4. The file is scanned for DRM violations.
5. The user is asked if they're really sure they want to copy the file.
6. The user is asked again if they're sure they want to copy it.
7. The OS makes a judgement on how long it will take to copy so it can update the pretty stats in the gui.
8. Lots of flashy graphics and widgets are loaded to show you a pretty animation while you wait.
9. The file is copied.
10. The destination file is verified that it is intact.
11. The destination file is scanned for viruses.
12. The destination file is scanned for adware.
13. The destination file is scanned for DRM violations.
14. The file is successfully copied.

Hell - I'm surprised their OS can even handle copying 1,600 files, let alone 16,000.

Re:Maybe this stems from...

"Vista is now forced down my throat"

I got news for you, that is Vista, but it isn't your throat it being forced into.

Re:We *ALL* need to give Microsoft a dope slap

mod parent as stupid.

Likely, they're allocating memory to store file attributes or some such that are not being free'd when done with. Hence running out of memory. If you had coded a day in your life you'd see that.

Tom