Vista Runs Out of Memory While Copying Files

ta bu shi da yu writes "It appears that, incredibly, Vista can run out of memory while copying files. ZDNet is reporting that not only does it run out of memory after copying 16,400+ files, but that 'often there is little indication that file copy operations haven't completed correctly.' Apparently a fix was scheduled for SP1 but didn't make it; there is a hotfix that you must request."

Actual info...

[EveryNickIsTaken]

From TFA:

...occurs when a Vista user (running Kaspersky Anti Virus 6 or 7) tries to copy a large number of files (~16,400).

So if you're like most people in the world, and have never touched Kaspersky AV (or Vista, for that matter), then this is a non-issue.

Vista

[MyLongNickName]

When I copy a bunch of files from one directory to another, I get 'Explorer has stopped working and must restart'. I've resorted to using DOS to copy the files. I wish I had stuck with 2000 Server :)

Re:Actual info...

[Phil246]

actually, fta:

Although the problem occurs where users are running Kaspersky security products, it's a kernel leak that lies at the root of problem (the problem's not confined to systems running Kaspersky software, that just that this application seems to exacerbate the issue).

Re:Just wondering...

[808140]

Maybe you're backing up to an external hard drive?

Re:Billy G says

[Seismologist]

Found the quote on wikiquote [wikiquote.org]:

I laid out memory so the bottom 640K was general purpose RAM and the upper 384 I reserved for video and ROM, and things like that. That is why they talk about the 640K limit. It is actually a limit, not of the software, in any way, shape, or form, it is the limit of the microprocessor. That thing generates addresses, 20-bits addresses, that only can address a megabyte of memory. And, therefore, all the applications are tied to that limit. It was ten times what we had before. But to my surprise, we ran out of that address base for applications within--oh five or six years people were complaining.

Re:Actual info...

[philg8]

The underlying problem is a Windows OLE component memory leak. Microsoft has a hotfix for the issue at http://support.microsoft.com/kb/942435/en-us [microsoft.com]

OLE mem leak; only affects 'extended attrib' files

[I'm Don Giovanni]

According to the cited "hotfix" link, http://support.microsoft.com/kb/942435/en-us [microsoft.com] , the problem is due to an OLE memory link when dealing with files that have "extended attributes".

This problem occurs if the following conditions are true:
  * The files include extended attributes.
  * You copy lots of files in a single operation.

CAUSE
This problem occurs because of a memory leak in the Windows OLE component. This memory leak is triggered by the way that Windows Explorer deals with the extended attributes of the files.

Re:Actual info...

[EveryNickIsTaken]

Even spending 5 seconds googling this issue would tell you that this only occurs when the user is running Kaspersky. The blogger that posted about this added his two cents in, which is factually incorrect.

Re:Actual info...

[Anonymous Coward]

Actually, the bug is in the shell, not the kernel and only files with altnerate data streams trigger the leak. The KB article that Adrian links to states that very clearly, but he's been on an anti-Windows rampage lately that's blinded him to the facts.

Very few files have data streams, so the vast majority of users won't ever see a problem. Kaspersky choses to pollute every single file with a stream, however, which is why systems with it installed exhibit the problem.

Bad summery

[gravis777]

Apparently the submitter skimmed the article, and decided to post up a Vista bash on Slashdot.

FTA:

The "Out of Memory" error (which is affectionately known at the PC Doc HQ as the "Out of Cheese" error ... don't ask why ...) is one of the biggest and most baffling of Vista's file handling problems has been occurs when a Vista user (running Kaspersky Anti Virus 6 or 7) tries to copy a large number of files (~16,400)

Apparently its just a problem with this antivirus program running in Vista. I move large amounts of files around in Vista quite often (granted, its Vista 64), sometimes well over 20,000 files at a time, and have never run into this issue.

Re:I'm a little suspicious

[coolnicks]

KB 942435:

This problem occurs because of a memory leak in the Windows OLE component. This memory leak is triggered by the way that Windows Explorer deals with the extended attributes of the files.

Its only files with streams, and apparently kaspersky makes it wose by that fact that it tags every single file with a stream.

Re:We *ALL* need to give Microsoft a dope slap

[tomstdenis]

mod parent as stupid.

Likely, they're allocating memory to store file attributes or some such that are not being free'd when done with. Hence running out of memory. If you had coded a day in your life you'd see that.

Tom

Re:Bad summery

[mysticgoat]

Yes, for Vista it has certainly been a bad summery. The forecast for the upcoming wintery doesn't look so good, either.

Pun-ishments aside, those who RATFA know that the fault is in the Vista kernel, it is consistently triggered by Kapersky, but is also triggered by other software, and by implication it is not consistently repeatable and therefore cannot be easily worked around.

On my WinXP home machine, I routinely copy more than 16,400 files when doing a full data backup to an external drive, which I do two or three nights a week. Even if Vista was perfect in every other way, this would be a show-stopper for me.

Re:Bad summery

[Zebra_X]

Lol the TFA is FUD. Read the HOTFIX notes which explains that the issue is with Windows OLE (NOT part of the kernel) and files that utilize extended attributes. Note that the crappy AV product adds extended attributes to all of your files (which i'm sure speeds up every file operation on your PC), thus with a kapersky infected computer - you are assured to have the problem. With "normal" files it's unlikely you will have this issue. Media files and the new office files are more likely to pose a problem than your standard files.

The article does not state clearly wether physical memory is a constraint.

Re:Actual info...

[Foolhardy]

First of all, the issue is how Explorer handles extended attributes (EAs), which are distinct from alternate data streams (ADSes). The kernel and NTFS have always provided full support for EAs and ADSes (since NT 3.1). Explorer (and for that matter Win32) has never had very good support for ADSes, and almost nonexistent support for EAs. EAs were implemented in support of the OS/2 subsystem. ADSes are the 'official' way to attach metadata to a file, and scale better than EAs. The only Win32 functions that have ever provided access to EAs are the BackupRead and BackupWrite functions which are designed to handle all metadata on a file transparently. Looking at the imports from shell32.dll to ntdll.dll on Vista, it looks like the shell bypasses Win32 when dealing with EAs, invoking the syscalls NtQueryEaFile and NtSetEaFile directly (bypassing API layers like this is something Microsoft tells ISVs is a big no-no).

This is just Yet Another Windows 95 shell bug (yes Vista uses the same shell architecture ported through each version from Win95). It is not the end of support for EAs or ADSes. If anything, it's a belated attempt at better support, done poorly. The shell has always been, IMO, one of the lower quality windows components, especially when it comes to properly interfacing with lower layers. This bug does not surprise me. I've been using robocopy for nontrivial file transfer for a while now.

Re:Maybe this stems from...

[InvalidError]

For those situations...

Run -> "cmd" -> del %dir\*.* /s

It will clear most stuff and you will see error messages fly by... redirect output to a file for later examination if desired.

I use the good old 'del' whenever I know I will be deleting something like 20k files and do not wish to waste time waiting for windows to prepare for that operation... why the heck does Windows need to scan directories to be deleted before deleting them is beyond me, just delete them and be done with it. Same thing for copying, Windows wastes time scanning the source directory for no apparent reason since it won't tell you you have insufficient disk space to complete the operation until the target drive runs out of disk space... or any other errors for that matter, until it runs into them while carrying out the actual operation.

Linux has quirks, so does Windows. Linux has the excuse of being an relatively immature desktop OS but on the Windows side, it can only be written off as the result of half-ass design decisions.

Re:Maybe this stems from...

[Master of Transhuman]

Yup. I've just about given up trying to copy gigabytes of files from large drives to backup drives because of this. I mean, once you've done this ONE TIME, it should be OBVIOUS to ANY designer that it needs to be fixed to allow the copy to continue - or resume - or SOMETHING other than just DYING.

Also, if Windows sees a zero-byte file, it can't handle it. I have to boot Linux and use it to delete the file.

Daily while working with clients I ask myself how anybody could use this garbage on a daily basis. When I reboot into Windows on my dual-boot openSuse/XP machine, I always dread it because I KNOW something is not going to work properly, or something extraneous will have to be done BEFORE I can do what I need to do.

Pathetic OS. Just pathetic.

Re:Billy G says

[ultranova]

The problem was that the operating system's facilities for addressing the hardware - in particular the screen - were so piss-poor that programmer's had to address the hardware directly in order to achieve reasonable results. There weren't even system calls to allow the software to discover dynamically where the hardware was - the locations had to be hard coded. As a result it then became impossible to move on to a new generation of hardware (with a different limit) because the old software then wouldn't run any more.

No. That was not the problem. The problem was that DOS programs were 16-bit real mode programs. This means that they used 16-bit pointers to refer to memory locations. This is what limits a DOS program to 1 megabyte of memory, not any deficiency in MS-DOS (which it had many of, admittedly). The segmented perversion of 8086 made things even worse by making memory divided into 64kB chunks rather than contiguous.

In any case, as time went on, most DOS programs did move to next-gen hardware, first by using EMS and XMS memory, and later by using DOS extenders to run in 32-bit protected mode. Having fixed screen memory location was never the problem, quite on contrary: it made it possible to access the video card memory directly from protected mode without having to convert a 16-bit pointer from DOS into 32-bit one.

1) Methods to drive the screen purely by system calls, without having to be aware of the hardware at all.

We are talking about unaccelerated graphics card here. The fastest way to use them was to write directly to memory. Going through a system call would not only have been slower, meaning no one would had used it, but required said operating system to contain some kind of graphics driver, which would had taken up precious memory space and therefore hindered every program.

It's not as if all this wasn't known at the time. Other earlier offerings provided all this and more, but MS-DOS really didn't deserve the title of "Operating System" at all. It was a filing system and process loader with a few extra bits cobbled onto it to avoid being prosecuted under trades description laws.

DOS is perfect for what it's designed for - a filing system for two 360 kB diskettes that takes up little memory and doesn't get in your way, and lets you get your program into the memory. Of course a system resulting from these design parameters doesn't work too well in a modern machine with 500 GB hard disk, gigabytes of memory and a dazzling array of extension cards.

And, frankly, I doubt anyone at either IBM nor Microsoft realized that the IBM PC would still be in use, extended beyond nearly all recognition, 26 years later.

Re:Maybe this stems from...

[rs79]

Yes but it can be fixed, or rather, worked around. It's been this way for literally decades.

Re:Maybe this stems from...

[justthinkit]

Someone else in this thread mentioned CachemanXP [outertech.com]. This non-crippled shareware allows you to limit the amount of free RAM XP uses for file caching (and has a nifty "kill" option, for when you absolutely positively wanna toast the sucka -- the reason I checked it out).

Re:Maybe this stems from...

[Allador]

TFA (where A = Author) is an idiot, and didnt bother to read the KB article which he linked to on his post.

From the KB article describing the problem, ie TRFA (where R = Real):

http://support.microsoft.com/kb/942435/en-us [microsoft.com]

This problem occurs because of a memory leak in the Windows OLE component. This memory leak is triggered by the way that Windows Explorer deals with the extended attributes of the files.

This is 100% in the shell and UI layers, not in the kernel.