• We’re currently investigating an issue related to the forum theme and styling that is impacting page layout and visual formatting. The problem has been identified, and we are actively working on a resolution. There is no impact to user data or functionality, this is strictly a front-end display issue. We’ll post an update once the fix has been deployed. Thanks for your patience while we get this sorted.

big dirs slowssssss waaaaay dowwwwwn

D

dbarton

Senior member

We have an app that creates 40,000 files in one directory.

If I browse them in Windows Explorer, or try and use the "open" dialog in a program, it takes about a minute before the machine allows me to scroll down at all.. It seem to be loading all the names into a cache or whatever, and then creating the tiny icons next to each name.

Any way to improve this speed?
 
Explorer is just plain slow when it comes to lots of files. Type the filename into the Open dialog to avoid browsing or use a non-graphical tool on them.
 
What OS? 2000 and XP handle it better then 9x, no matter the filesystem.
Click to expand...

If the filesystem is FAT it won't matter what the OS is, FAT sucks. But I also believe there's an upper limit of 32767 files per directory on FAT so he's probably using NTFS.
 
Why not write a simple program to output all the files in the directory to a .txt file or a spreadsheet and browse from there?
 
We had an NT4 box with shitloads of files in some directories, same problem.
Ultimately the solution was to use the command line and never do a directory listing.
 
Originally posted by: dbarton
We have an app that creates 40,000 files in one directory.
Click to expand...
LOL. Don't use Windows. Period. 😛 You're asking for trouble.

Alternatively, you could try a 3rd-party RAM-based filesystem driver, and see if that might suffice, or perhaps someone can help with some registry tweaks for Windows' internal filesystem caching settings, although I don't know if you can set them high enough to accomodate for 40000 dirents.

 
Originally posted by: Nothinman
Sound like there is no solution, as that's as fast as the OS is,
Click to expand...

It's only the UI that's slow.
Click to expand...

Have you tried using Progman.exe to access instead? (I think it's still included on the distribution media, or at least it used to be... hmm.)

I think that you might find that, while Explorer has it's own slowdown issues, the NT filesystem drivers themselves will also have severe slowdown issues, once you reach the point of having more files in any one directory than can be cached all at once into RAM in the filesystem cache. In that case, I think that it reverts to a worst-case uncached behavior for directory lookups, although I don't know what happens in terms of NTFS directory-file modifications, since by default, NTFS rewrites the *entire* directory-file *every* time it is written to, in order to always maintain a sorted copy, so that binary-search lookups can be performed. Perhaps a registry tweak to disable that behavior would help here? (Is there one?)

That's one reason why reading/writing many small files in the same directory on NTFS is so slow, btw.
 
I remember working on a similar problem on a Sparc box running Solaris a good 10 years ago. To maintain maximum performance, the number of files per directory needed to be limited to 511. The application creating the many, many small files was modified to create a new subdirectory every 512th file.
 
Have you disabled Indexing service?

In Explorer, right click each partition, click properties and uncheck the Indexing Service box.


Bozo
 
Have you tried using Progman.exe to access instead?
Click to expand...

No, but I've used the CLI.

That's one reason why reading/writing many small files in the same directory on NTFS is so slow, btw.
Click to expand...

Have you ever compared it to FAT though? It's at least 10x as fast.

I remember working on a similar problem on a Sparc box running Solaris a good 10 years ago.
Click to expand...

Luckily things have progressed since then, I'm sure even Solaris has introduced some directory hashing/indexing to avoid the kind of hacks that you had to use.
 
Originally posted by: Nothinman
That's one reason why reading/writing many small files in the same directory on NTFS is so slow, btw.
Click to expand...
Have you ever compared it to FAT though? It's at least 10x as fast.
Click to expand...
Actually, the workaround for that situation, that I've always heard mentioned, was switching to FAT32. Do you have any performance numbers to show that it is somehow still slower? It shouldn't be - file deletes out of a FAT directory file only require marking the file as deleted, and then zapping the FAT chain. No need to constantly re-write the entire directory file.
Originally posted by: Nothinman
I remember working on a similar problem on a Sparc box running Solaris a good 10 years ago.
Click to expand...
Luckily things have progressed since then, I'm sure even Solaris has introduced some directory hashing/indexing to avoid the kind of hacks that you had to use.
Click to expand...
Well, at least for this issue, that solution wouldn't have helped.
 
Actually, the workaround for that situation, that I've always heard mentioned, was switching to FAT32. Do you have any performance numbers to show that it is somehow still slower? It shouldn't be - file deletes out of a FAT directory file only require marking the file as deleted, and then zapping the FAT chain. No need to constantly re-write the entire directory file
Click to expand...

A while back I used a simple for loop to touch as man files as possible in a directory and not only was FAT magnitudes slower I believe it only allows 32768 entries per directory. Of course I don't remember the specifics, but it wouldn't be difficult to run again except for the fact that I don't have any FAT filesystems.

Well, at least for this issue, that solution wouldn't have helped.
Click to expand...

You never know, I just created 40000 files on my XFS /home partition and it took ~53s and FF's open dialog shows that directory instantaneously.
 
You must log in or register to reply here.

TRENDING THREADS

Back
Top