big dirs slowssssss waaaaay dowwwwwn

[dbarton]

We have an app that creates 40,000 files in one directory.

If I browse them in Windows Explorer, or try and use the "open" dialog in a program, it takes about a minute before the machine allows me to scroll down at all.. It seem to be loading all the names into a cache or whatever, and then creating the tiny icons next to each name.

Any way to improve this speed?

 

[JeffreyLebowski]

faster HDD. My pr0n pics folder is like that.
Make sure thumbnail view is off and details is off. just use "list"

 

[Shaftatplanetquake]

Originally posted by: JeffreyLebowski
faster HDD. My pr0n pics folder is like that.
Make sure thumbnail view is off and details is off. just use "list"

What he said.

 

[Nothinman]

Explorer is just plain slow when it comes to lots of files. Type the filename into the Open dialog to avoid browsing or use a non-graphical tool on them.

 

[sciencewhiz]

What OS? 2000 and XP handle it better then 9x, no matter the filesystem.

 

[Nothinman]

What OS? 2000 and XP handle it better then 9x, no matter the filesystem.

If the filesystem is FAT it won't matter what the OS is, FAT sucks. But I also believe there's an upper limit of 32767 files per directory on FAT so he's probably using NTFS.

 

[dbarton]

Yes, XP, NTFS, details only shown.

Sound like there is no solution, as that's as fast as the OS is,

 

[BigJ]

Why not write a simple program to output all the files in the directory to a .txt file or a spreadsheet and browse from there?

 

[Sunner]

We had an NT4 box with shitloads of files in some directories, same problem.
Ultimately the solution was to use the command line and never do a directory listing.

 

[Nothinman]

Sound like there is no solution, as that's as fast as the OS is,

It's only the UI that's slow.

 

[VirtualLarry]

Originally posted by: dbarton
We have an app that creates 40,000 files in one directory.

LOL. Don't use Windows. Period. You're asking for trouble.

Alternatively, you could try a 3rd-party RAM-based filesystem driver, and see if that might suffice, or perhaps someone can help with some registry tweaks for Windows' internal filesystem caching settings, although I don't know if you can set them high enough to accomodate for 40000 dirents.

 

[VirtualLarry]

Originally posted by: Nothinman

Sound like there is no solution, as that's as fast as the OS is,

It's only the UI that's slow.

Have you tried using Progman.exe to access instead? (I think it's still included on the distribution media, or at least it used to be... hmm.)

I think that you might find that, while Explorer has it's own slowdown issues, the NT filesystem drivers themselves will also have severe slowdown issues, once you reach the point of having more files in any one directory than can be cached all at once into RAM in the filesystem cache. In that case, I think that it reverts to a worst-case uncached behavior for directory lookups, although I don't know what happens in terms of NTFS directory-file modifications, since by default, NTFS rewrites the *entire* directory-file *every* time it is written to, in order to always maintain a sorted copy, so that binary-search lookups can be performed. Perhaps a registry tweak to disable that behavior would help here? (Is there one?)

That's one reason why reading/writing many small files in the same directory on NTFS is so slow, btw.

 

[htne]

I remember working on a similar problem on a Sparc box running Solaris a good 10 years ago. To maintain maximum performance, the number of files per directory needed to be limited to 511. The application creating the many, many small files was modified to create a new subdirectory every 512th file.

 

[Bozo]

Have you disabled Indexing service?

In Explorer, right click each partition, click properties and uncheck the Indexing Service box.


Bozo

 

[Nothinman]

Have you tried using Progman.exe to access instead?

No, but I've used the CLI.

That's one reason why reading/writing many small files in the same directory on NTFS is so slow, btw.

Have you ever compared it to FAT though? It's at least 10x as fast.

I remember working on a similar problem on a Sparc box running Solaris a good 10 years ago.

Luckily things have progressed since then, I'm sure even Solaris has introduced some directory hashing/indexing to avoid the kind of hacks that you had to use.

 

[VirtualLarry]

Originally posted by: Nothinman

That's one reason why reading/writing many small files in the same directory on NTFS is so slow, btw.

Have you ever compared it to FAT though? It's at least 10x as fast.

Actually, the workaround for that situation, that I've always heard mentioned, was switching to FAT32. Do you have any performance numbers to show that it is somehow still slower? It shouldn't be - file deletes out of a FAT directory file only require marking the file as deleted, and then zapping the FAT chain. No need to constantly re-write the entire directory file.

Originally posted by: Nothinman

I remember working on a similar problem on a Sparc box running Solaris a good 10 years ago.

Luckily things have progressed since then, I'm sure even Solaris has introduced some directory hashing/indexing to avoid the kind of hacks that you had to use.

Well, at least for this issue, that solution wouldn't have helped.

 

[Nothinman]

Actually, the workaround for that situation, that I've always heard mentioned, was switching to FAT32. Do you have any performance numbers to show that it is somehow still slower? It shouldn't be - file deletes out of a FAT directory file only require marking the file as deleted, and then zapping the FAT chain. No need to constantly re-write the entire directory file

A while back I used a simple for loop to touch as man files as possible in a directory and not only was FAT magnitudes slower I believe it only allows 32768 entries per directory. Of course I don't remember the specifics, but it wouldn't be difficult to run again except for the fact that I don't have any FAT filesystems.

Well, at least for this issue, that solution wouldn't have helped.

You never know, I just created 40000 files on my XFS /home partition and it took ~53s and FF's open dialog shows that directory instantaneously.