How does the cluster size affect disk size and disk performance? I currently have a 40gig drive with '512 bytes in each allocation unit' - NTFS. I would think this means that the drive is configured at a 512 byte cluster size atm. My other 30 gig drive - FAT32 is '16384 bytes in each allocation unit'. I'm planning on doing a reformat soon and would like to know which sizes to use for the drives. Thanks.
Can Someone Explain the Logic Behind Cluster Sizes?
- Thread starter DJediMaster
- Start date
You can think of your entire hard drive being broken down into very small blocks (clusters), and each cluster can only hold a maximum of one file. So, let's say that you have your cluster size at 512 bytes, and you have a file that is 800 bytes in size. 512 bytes of that file would go into one cluster and the remaining 288 bytes would go into the next cluster. However, the remaining 224 bytes of the second cluster cannot be used and, therefore, is wasted space. This is why when you look at the properties of a file you see two numbers.....the size (file size) and the size on disk (the sum of the size of all the clusters that contain the file). So, if you set your cluster size too high, you're going to get a lot of wasted space since a lot of your files won't occupy the entire cluster. On the flip side, if you set your cluster size too low, then you'll probably get a lot of fragmentation. I've been told that the optimum cluster size for NTFS is 4.00 KB (4096 bytes).
Heres the logic. Each cluster is of equal size, and if a cluster isnt compeletly filled it will still register as using that much space. So large clusters (16K for instance) when holding small files (8K for instance) will be inconcievably wasteful. A 16K cluster holding an 8K file, will waste aproximatley 8K, so its best to haf 16K clusters. But theres another side to that coin.
Each cluster needs an address. Thus, a harddrive formated for 8K clusters will have 2X the addresses used as a 16K cluster harddrive. And in FAT16 the address size is 16 bytes. In Fat32 its 32 bytes. I believe NTFS uses 32 bytes. Anyways, 2 gig was the limit in FAT 16 because even using the largest cluster size it could only fit so many clusters into the 16 bit addressing scheme. You only have 65536 addresses in the fat scheme. Multiply that many addresses by 16K each and you have 1GB. Even when using 32 K clusters, (EXTREMLEY inefficent) you only get 2 gig. Ouch! Thus, FAT32 was born. NTFS was always a server/workstation based addressing system, and was quite advanced. FAT16 is nearly 20 years old. FAT32 isnt old at all, and was just an extension to the FAT table. Limit is 2000GB. Thats the logic. Advice, well, you other people have any advice?
Each cluster needs an address. Thus, a harddrive formated for 8K clusters will have 2X the addresses used as a 16K cluster harddrive. And in FAT16 the address size is 16 bytes. In Fat32 its 32 bytes. I believe NTFS uses 32 bytes. Anyways, 2 gig was the limit in FAT 16 because even using the largest cluster size it could only fit so many clusters into the 16 bit addressing scheme. You only have 65536 addresses in the fat scheme. Multiply that many addresses by 16K each and you have 1GB. Even when using 32 K clusters, (EXTREMLEY inefficent) you only get 2 gig. Ouch! Thus, FAT32 was born. NTFS was always a server/workstation based addressing system, and was quite advanced. FAT16 is nearly 20 years old. FAT32 isnt old at all, and was just an extension to the FAT table. Limit is 2000GB. Thats the logic. Advice, well, you other people have any advice?
<< Is the optimum cluster size for FAT32 also 4k? >>
it's the least wasteful if that's what you mean but can only be used for partition sizes up to 8,191 MB or else you'll find windows will have problems like running scandisk & defrag on any bigger partition sizes!
The "optimum" cluster size varies for everyone, depending on what you're doing. The need for very small cluster sizes has essentially gone away in the past several years. Originally there were two related reasons: slack space, and lack of space. Lack of space was exacerbated by slack space.
Slack space is described above, empty space caused by clusters being larger than the files they contain. With small hard drives and lots of small files, this could be a problem with a large partition, since FAT16 used 32K clusters for a 2GB partition. FAT32 alleviated that problem by allowing 4K clusters up to 8GB, though you can also specify the cluster size to use rather than the default.
With larger and larger hard drives, the problem of wasted space becomes lessened. Who really cares if we lose a few hundred megs out of 20GB or more? In addition, how many files are less than 16KB in size these days? (16K is the cluster size for up to 32GB in FAT32.) While there are a lot of files under that size, a good huge percentage of files on a system are now WAY over that, so that you're not wasting as much as you used to with a 16K cluster size. So you might waste 12KB with a 16MB file that doesn't quite fill that last cluster. Who cares?
In addition, the smaller your clusters, the MORE clusters you need to fill a particular partition size. Windows has to keep track of all those clusters all the time (the addressing of them) and it has to keep them in memory. When you do a disk defrag or scandisk, the more clusters it's having to track, the slower it goes.
I keep my clusters at 16KB now. With smaller drives (and smaller partitions, like 2GB) I used 8KB. This seems to be the optimum for me.
Oh yes, NTFS uses 64bit addressing, which is why its maximum size is still larger than FAT32.
Slack space is described above, empty space caused by clusters being larger than the files they contain. With small hard drives and lots of small files, this could be a problem with a large partition, since FAT16 used 32K clusters for a 2GB partition. FAT32 alleviated that problem by allowing 4K clusters up to 8GB, though you can also specify the cluster size to use rather than the default.
With larger and larger hard drives, the problem of wasted space becomes lessened. Who really cares if we lose a few hundred megs out of 20GB or more? In addition, how many files are less than 16KB in size these days? (16K is the cluster size for up to 32GB in FAT32.) While there are a lot of files under that size, a good huge percentage of files on a system are now WAY over that, so that you're not wasting as much as you used to with a 16K cluster size. So you might waste 12KB with a 16MB file that doesn't quite fill that last cluster. Who cares?
In addition, the smaller your clusters, the MORE clusters you need to fill a particular partition size. Windows has to keep track of all those clusters all the time (the addressing of them) and it has to keep them in memory. When you do a disk defrag or scandisk, the more clusters it's having to track, the slower it goes.
I keep my clusters at 16KB now. With smaller drives (and smaller partitions, like 2GB) I used 8KB. This seems to be the optimum for me.
Oh yes, NTFS uses 64bit addressing, which is why its maximum size is still larger than FAT32.
<< How does the cluster size affect disk size and disk performance? I currently have a 40gig drive with '512 bytes in each allocation unit' - NTFS. I would think this means that the drive is configured at a 512 byte cluster size atm. My other 30 gig drive - FAT32 is '16384 bytes in each allocation unit'. I'm planning on doing a reformat soon and would like to know which sizes to use for the drives. Thanks. >>
Did you happen to see this thread at the HardForum?
So what are the rules here:
-The max size for FAT32 cluster size is 16KB, which is somehow also related to the size of the formatted drive/partition?
-The max size for an NTFS partition/drive is? Is this somehow related and dependent on the actual size of this partition drive?
-For the average user/workstation, the optimum cluster size is 16KB?
-If so, how does one go about doing this with a brand new drive with plans to install Win2k or WinXP using NTFS?
-How does one go about doing this with Win2K or WinXP (NTFS for both) already installed and somehow it has defaulted to 4KB cluster size in Win2k and 512bytes in WinXP?
-What if one had the following setup:
C= current cluster size
C:\ FAT32 1GB 4KB C
D:\ FAT32 5GB 4KB C w/WIN98SE
E:\ NTFS 6GB 4KB C W/ WIN2K
F:\ NTFS 6GB 512b C W/ WinXP
G:\ FAT32 10GB 4KB C W/DATA
Etc...
Thanks!
-The max size for FAT32 cluster size is 16KB, which is somehow also related to the size of the formatted drive/partition?
-The max size for an NTFS partition/drive is? Is this somehow related and dependent on the actual size of this partition drive?
-For the average user/workstation, the optimum cluster size is 16KB?
-If so, how does one go about doing this with a brand new drive with plans to install Win2k or WinXP using NTFS?
-How does one go about doing this with Win2K or WinXP (NTFS for both) already installed and somehow it has defaulted to 4KB cluster size in Win2k and 512bytes in WinXP?
-What if one had the following setup:
C= current cluster size
C:\ FAT32 1GB 4KB C
D:\ FAT32 5GB 4KB C w/WIN98SE
E:\ NTFS 6GB 4KB C W/ WIN2K
F:\ NTFS 6GB 512b C W/ WinXP
G:\ FAT32 10GB 4KB C W/DATA
Etc...
Thanks!
<< The max size for FAT32 cluster size is 16KB >>
I think its 32K
4k clusters up to 8 Gig partition
8k clusters uo to 16 Gig
16k clusters up to 32 Gig
32k clusters above 32 Gig
I use the default 4K NTFS cluster size. If you convert a Fat32 partition to NTFS, it uses the 512 byte cluster size.
An example of how cluster size can kill you 
I had a user who wrote a data-mining script that stored temp data in files. His script produced 1,027,000 files (I"m not kidding, one million!) on an NTFS partition of our one NT server. The files were between 9 bytes and 2k. There was like 237 megs worth of actual data, but it was taking more than 2 gigs (I forget the actual figure) of hard drive space because these little 9 byte files were taking an entire cluster.
If your machine is dedicated to a purpose, and you know you will have mainly tons of small files, or mainly large files, then you might want to go with something other than the standard or default cluster size. But this is pretty rare. The suggestions I've seen here look good.
I had a user who wrote a data-mining script that stored temp data in files. His script produced 1,027,000 files (I"m not kidding, one million!) on an NTFS partition of our one NT server. The files were between 9 bytes and 2k. There was like 237 megs worth of actual data, but it was taking more than 2 gigs (I forget the actual figure) of hard drive space because these little 9 byte files were taking an entire cluster.
If your machine is dedicated to a purpose, and you know you will have mainly tons of small files, or mainly large files, then you might want to go with something other than the standard or default cluster size. But this is pretty rare. The suggestions I've seen here look good.
TRENDING THREADS
-
Discussion Zen 5 Speculation (EPYC Turin and Strix Point/Granite Ridge - Ryzen 9000)- Started by DisEnchantment
- Replies: 24K
-
Discussion Intel Meteor, Arrow, Lunar & Panther Lakes Discussion Threads
- Started by Tigerick
- Replies: 19K
-
-
-
Facebook
Twitter