best way to store & quickly access *lots* of tiny files?

[Fox5]

Compressing the folders using windows' drive compression might help.

If you could transition to a linux machine running eclipse, the reiser filesystem was made for this type of file storage. XFS and ext4 would also perform well in this situation. Both Reiserfs and XFS are able to pack files together to eliminate free space waste, reiserfs achieves almost zip levels of space savings in certain situations. Packing files like that also makes data recovery nearly useless if the hard drive is formatted or the partition table destroyed.
Reiser4 is even better, it has gzip compression built in as an optional function of the file system.

 

[DominionSeraph]

Originally posted by: taltamir

2GB is not enough today, common programs in 2009 take so much ram that 2GB is just too little.
Anyways, even the most mild usage patterns in 2009 using the latest versions of common programs for non intensive things requires more than 2GB

Hmmm... currently torrenting, browsing, and running warcraft III.
454MB.

Your bloatware does not constitute a memory crisis for anyone else.

 

[chrisf6969]

The size of your data is small. I doubt you need more ram. hit CTRL+ALT+DEL and see what your memory / pagefile / VM usage is. Sounds like you just have a crap load of small I/O's / files.

IE: get a small (20-80Gb) quality (indillix or Intel) SSD.

 

[taltamir]

Originally posted by: NTB

Originally posted by: Denithor
USB flash drives are not fast at all - the fastest are still much slower than even slow HDD at transfer rates.

Just use one of those RAMdisk programs to create a 512MB drive and copy all the data files onto it. If you need the whole 2GB for system memory add another 1GB stick to the computer and use the whole stick for your RAMdisk.

Remember to back up when through working so you don't lose everything in case of power failure/reboot/etc.

But the access time is much lower, which should help - I'm dealing with *tons* of tiny files, as opposed to a few larger ones. For conditions like this, random read / write performance is more important than linear read / write, correct?

Nathan

no, it is unbelievably slow. especially on writing small files.

 

[taltamir]

Originally posted by: DominionSeraph

Originally posted by: taltamir

2GB is not enough today, common programs in 2009 take so much ram that 2GB is just too little.
Anyways, even the most mild usage patterns in 2009 using the latest versions of common programs for non intensive things requires more than 2GB

Hmmm... currently torrenting, browsing, and running warcraft III.
454MB.

Your bloatware does not constitute a memory crisis for anyone else.

you are the exception to the rule than (assuming you are measuring it correctly).

As far as "MY bloatware" is concerned. I have none... The average consumer does not have a lean clean mean machine... they have a ton of crap. Even for non average consumers who are more tech savvy, it is still the norm to need more than 2GB.

PS. your OS manages your ram, so if you have too little ram it will "use" less but cause thrashing. To know how much you need, put more ram in it without changing ANYTHING. You will notice the ram requirements increase as thrashing and pagefile use decrease; thus improving your performance.
"runnable" and "good experience" are different things.
And my laptop is also limited at 2GB.

 

[Fox5]

If you're adventurous, try flashfire. It's made for flash drives, but will work for normal hard drives too. It caches small writes using system ram, so it speeds up small writes to the speed of large writes.