What allocation size to choose for 3TB drives?

[SadPanda]

I want to have 3 x 3TB drives for storage later on, but I don't know which allocation size to format them with?

1. A drive for only pics ranging anywhere from 50kb-3mb each.

2. A drive for music ranging anywhere from 3mb-20mb each.

3. A drive for movies ranging anywhere from 100mb-10gb each.

Does it even matter which allocation size I choose?

 

[bononos]

Do you mean the cluster size? Generally the larger size files should work better with larger cluster size. Anything with the default 4k or larger should be fine.

 

[C1]

My understanding is that it might. Smaller clusters (allocations) involve a larger master file allocation table and the look up is slower and at some point it even exceeds a system's and/or OS's capability to read it.

The 4K cluster default is required if one plans to encrypt.

Otherwise, for a drive that size, I might try at least 64Kb clustering.

There's lots more to all this (eg, wasted space), but that is for HDD affectionadoes.

 

[G73S]

for movies, choose the largest possible cluster size, for other, stick to the default

 

[VirtualLarry]

What about 4x3TB drives in a RAID5 config? Then you have to worry about stripe sizes too...

 

[G73S]

What about 4x3TB drives in a RAID5 config? Then you have to worry about stripe sizes too...

For a RAID stripe I would choose 64K as it is a good medium

That's also the stripe size that my Alienware motherboard recommends (gives you a message in the BIOs when creating the RAID array that 64K is the recommended size)

 

[SadPanda]

What if you're constantly overwriting the drives with new data? Say I always delete movies and replace them with new ones. Would the cluster size matter then?

Also, do people defrag storage drives at all?

 

[Atreidin]

I doubt you will notice a significant difference if you change it from default.

 

[Cerb]

http://forums.anandtech.com/showthread.php?t=2389041

A recent thread on the subject, with links and testing.

Short version: 8.3 names and atime can increase HDD IO enough to worry about, in some cases, but cluster size is fine at default, and hasn't been worth changing in a long time. It was something to do back when you had MailDir inboxes on 2GB HDDs, and such situations as that.

 

[BonzaiDuck]

I limited my choice of new Seagate NAS drives to 2TB. That way, I wouldn't have to format as GPT. While WHS-2011 "handles" drives of a size larger than 2TB (GPT format), my "research" concluded that it would require a 2TB + 1TB logical-drive structure on a 3TB GPT drive. I didn't want to introduce that sort of complexity into my drive-pool.

Now . . .the plot thickens a bit . . . Once I'd installed Acronis TI 2014 Premium on the server, I discovered that it has a feature allowing extension of the 2TB logical disk size to the full capacity of the drive and partition.

But I'm not eager to "insert" some 3rd-party "fix" into the equation.

Of course, the other factor was price, and I could get the 2TB units for about $100 while the larger units were $40 to $60 more. That was in February or March, so I have no idea of the current status-quo. I can only say that 8TB (4x2TB) is plenty for my server here.

. . . At least -- for now . . .

 

[SadPanda]

BonzaiDuck...I have no idea what you just posted lol.

Ok, so 64kb cluster size for movies and default for music and pics.

Is it a bad idea to defrag storage drives?

 

[BonzaiDuck]

BonzaiDuck...I have no idea what you just posted lol.

Ok, so 64kb cluster size for movies and default for music and pics.

Is it a bad idea to defrag storage drives?

Don't want to belabor my sidebar comments, but from your initial post, you have 3TB drives with GPT partitions. I understand your question about allocation size and formatting. My fault -- for digressing about issues using GPT (>2TB) drives under various OS's.

What OS are you using? If Win 7 or Win 8, you should be able to create a GPT partition that allows for a single 3TB logical volume. My point: I can't do that with WHS-2011 -- which nevertheless handles GPT formats and AF. It apparently limits logical volumes to 2TB.

Sorry I made this digression -- which would be confusing.

Oh. Also. I don't think there's a problem defragging storage volumes. We defrag RAID0 thru RAID5 configurations all the time. The only prohibition on defragging I know of: you don't do it with SSDs.

My guess is that you haven't got these drives set up under RAID, or you would be concerned exclusively with STRIPE size -- not disk-by-disk allocation size in the formatting process.

I think the advice already given -- "that it shouldn't matter much" -- is ballpark. I also wonder if you've thought through the details of this.

Take for instance my own server. As if it were configured as "JBOD," all the drives are running as "AHCI mode" and combined into a drivepool. So I have drives F:, G:, H:, I: which are used to create a "virtual drive volume" J:. J: is the only drive I write to from client systems, and the only way I bother to read or copy files even when doing maintenance on the server console. Like you, I want to organize "Pictures," "Movies," "Music," etc. -- but those are shared folders on the virtual "J:" drive.

As much as you seem wanting to fine-tune this three-drive storage system, I personally wouldn't trouble myself to use a physical drive exclusive to each file category: one drive might fill up faster than others, and the "others" would have a lot of unused space -- maybe unused space that never gets used.

I'd simply pick the default allocation size when formatting them -- in my own current configuration. If I were to make a RAID0 or RAID5, the result would be a single logical volume -- OR -- I could create multiple logical volumes across the array. At that point, the chosen stripe size might be more beneficial to one file type versus another, but this is more a question of storage efficiency than anything. And in the RAID configuration, I couldn't very well pick multiple stripe sizes even with multiple logical volumes.

So it would seem that either the default allocation size would be fine on each drive -- OR -- the stripe size for a RAID array with either single or multiple logical volumes would be a "happy medium."

Were you planning to put these three 3TB drives in a workstation/standalone-desktop? Or did you want to share them on a household LAN in a server system? Or -- a NAS?

 

[stlc8tr]

I've always just used the default cluster size but I believe Partition Magic will allow you to switch sizes without losing data if you want to compare and benchmark.

 

[SadPanda]

What OS are you using? If Win 7 or Win 8, you should be able to create a GPT partition that allows for a single 3TB logical volume. My point: I can't do that with WHS-2011 -- which nevertheless handles GPT formats and AF. It apparently limits logical volumes to 2TB.

I will be building a new desktop with Win 8.1. What is WHS-2011 and AF?


Sorry I made this digression -- which would be confusing.

No probs, we're all here to learn...was a lil confused is all


Oh. Also. I don't think there's a problem defragging storage volumes. The only prohibition on defragging I know of: you don't do it with SSDs.

How much free space is required to efficiently defrag a 3TB drive? Why is it bad to defrag a SSD?


My guess is that you haven't got these drives set up under RAID, or you would be concerned exclusively with STRIPE size -- not disk-by-disk allocation size in the formatting process.

Nope...still learning about this stuff as I go along. Although I want to mirror my drives in case one fails.


Take for instance my own server. As if it were configured as "JBOD," all the drives are running as "AHCI mode" and combined into a drivepool. So I have drives F:, G:, H:, I: which are used to create a "virtual drive volume" J:. J: is the only drive I write to from client systems, and the only way I bother to read or copy files even when doing maintenance on the server console. Like you, I want to organize "Pictures," "Movies," "Music," etc. -- but those are shared folders on the virtual "J:" drive.

I'm not computer savvy enough to know what all this means. Please excuse my ignorance.


As much as you seem wanting to fine-tune this three-drive storage system, I personally wouldn't trouble myself to use a physical drive exclusive to each file category: one drive might fill up faster than others, and the "others" would have a lot of unused space -- maybe unused space that never gets used.

The thing is, if I put small files with large files...wouldn't choosing the wrong cluster size waste space because of the smaller files? I just figured maybe it would be best to put the smaller files separate from the large files.


So it would seem that either the default allocation size would be fine on each drive -- OR -- the stripe size for a RAID array with either single or multiple logical volumes would be a "happy medium."

I don't know what stripe size for Raid array means lol.


Were you planning to put these three 3TB drives in a workstation/standalone-desktop?

I was planning to just have a standalone desktop for everything such a streaming/browsing/downloading/music/tv shows/storage/photoshop/video editing. A 250GB SSD for the OS and 3x3TB drives for storage.

I should probably read up on RAID configurations and NAS setups, but I'm gonna learn how to OC my new computer soon so that might have to wait for now ^_^

 

[BonzaiDuck]

I should probably read up on RAID configurations and NAS setups, but I'm gonna learn how to OC my new computer soon so that might have to wait for now ^_^

WHS-2011 is the home-server Windows-version based on Win Server 2008 R2 -- loosely compatible with Win 7 in terms of drivers and essential features.

"AF" means "Advanced Format," which requires Win 7 and WHS-11 to "emulate" pre-AF 512-byte sector-size from a new AF sector-size of 4096. With Win 7 (AFIK) this was implemented in SP-1, as with WHS in updates, roll-ups etc. Win 8 should not have the "transitional" fixes precipitated by AF.

"Free space minimum to effectively defragment . . . " I'd say it would become more difficult with less and less free space, but I'd begin to worry with only 20% free space left -- thinking to move files to another location. Frankly, I try to keep at least 30 to 40% "free," or begin to consider migrating files with less than that. Someone else might be able to tell you a more robust "rule-of-thumb."

My understanding or explication may be imprecise, but there's no advantage to defragging SSDs, since there's no delay in reading from one fragment after another. Further, it degrades SSD by adding unnecessary writes to usage. Again, someone else can offer keener insight to this.

As for "striping" and "stripe-size:" You mentioned a desire to "mirror" disks, which would imply "RAID1". So "mirroring" is one approach to how two drives might be configured. "Striping" is the basis for RAID0. Files and data are broken up into chunks or stripes allocated equally between two physical disks. Thus, if you lose one drive in a RAID0 -- you lose it all (unless it had been backed up).

These new high-capacity HDDs are certainly not expensive, even after the "Tsunami-effect" that took place in the market a few years ago. So I can't try and tell someone that 9TB is a lot more disk capacity than you'd want in a workstation -- and that may reflect what "I think is necessary" anyway.

But you might begin to think how you'll back up your material to stave off disaster if a drive fails and becomes unrecoverable.

The other thing to think about is the power draw. I'm replacing all the boot-drives in our household workstations with SSDs (more expensive) with hopes of reducing our power consumption. I used to have a workstation with five HDDs running 24/7; other workstations with one and maybe two (RAID0) boot-volumes; and at least three disks running in my server. Now I have two SDDs and four HDDs in the server. I have a single HDD in my main workstation for DVR recordings -- two SSDs for boot volume and program -- and/or other data files.

A lot of my working files are stored for direct-access on the server -- which runs 24/7, and the server takes backup images of all the house workstations every night -- allowing for a bare-metal restore if a boot SSD or HDD fails on a Workstation.

You have a lot of options as to how you might configure those three drives. You could, for instance, make a 3-drive RAID0 (with the understandable risks) or a 3-drive RAID5 (incorporating the fail-safe features of RAID1 and the speed/capacity of RAID0).

Or -- you could configure your system BIOS for SATA "AHCI-mode" and partition/format the drives separately with separate labels. Or -- you could apply all the details in the first sentence here, and then create a "virtual drive" and drive-pool from the three disks -- with inexpensive drive-pooling software. Or -- the hardware alternative to the drive-pooling: a RAID BIOS option that simply makes the disks a "JBOD" or just-a-bunch-of-disks offering a single drive label.

I and others use StableBit Drive-Pool on our server systems, but the software developers have now assured that StableBit works for both Win 7 and Win 8 (as well as for most current/more-expensive server OS flavors. But StableBit is by no means the only drive-pooling software option.

One of the features of the drive-pooling software options: You can duplicate folders and files across more than one HDD so that nothing is lost (of those files and folders) if a single HDD goes south.

Take your time; do some reading; try and formulate a strategy that also includes some kind of workstation backup for this huge disk capacity.