New Windows Server 2012 file server - NTFS or ReFS for data volume?

[smakme7757]

I've just setup and got my new Server 2012 file server up and running. I've done a test restore from the backup to make sure everything is running smoothly.

At the moment I have the data volume formatted as NTFS. If i decide to stick with NTFS i'll just leave it as is, but I'd like some input on wether or not ReFS would be a better bet.

The data is user files, ISO images and video as well as visual studio projects. Lets just call it a big mix of digital "stuff".

For a new file server would there be any disadvange to storing the data on a ReFS volume?

Self repairing does seem like a very nice feature, but I'd hate to jump ship from NTFS and then have to reformat the data and restore again due to a poor choice of file system.

Any thoughts?

NTFS or ReFS?

Thanks people .

 

[metril]

ReFS

 

[hennessy1]

Recently I set up a 28TB x 2 server for computer images and other misc. storage of data. I looked into the zfs type of system but it would not work with the other plans I had for the server. I read up on ReFS and went that route. It has not been up long enough to give any tangible results to you to help but I also have not had any issues either. If it does what it is suppose to do then I should get some of the key benefits that zfs offers. I just couldn't see myself putting NTFS on there for that amount of storage I knew something would mess up eventually.

 

[smakme7757]

ReFS

Recently I set up a 28TB x 2 server for computer images and other misc. storage of data. I looked into the zfs type of system but it would not work with the other plans I had for the server. I read up on ReFS and went that route. It has not been up long enough to give any tangible results to you to help but I also have not had any issues either. If it does what it is suppose to do then I should get some of the key benefits that zfs offers. I just couldn't see myself putting NTFS on there for that amount of storage I knew something would mess up eventually.

28TB is a lot of data, I don't have quite that much, around 7TB. It's good to know you haven't had any troubles which is the most imporatnt thing at the stage .

I will reformat the partition tomorrow as ReFS after work and re populate from backups.

I appreciate the input.

 

[milee]

Use ReFS, just make sure you enable integrity streams to benefit from self healing (on mirrored storage spaces it gets automatically enabled) and don't use the volume for sql dbs. I also run 2 file servers using ReFS data volumes for about 1 year with no problems.

 

[hennessy1]

@milee-Any specific reason you advise not to use sql dbs on an ReFS volume?

 

[Mushkins]

@milee-Any specific reason you advise not to use sql dbs on an ReFS volume?

Im no expert, but i'd imagine the self healing and protective features of ReFS in and of themselves could wind up corrupting your live databases while thinking it's "helping." You could turn off a lot of the protections specifically for the database files, but it still strikes me as playing with fire at first glance. Maybe leave some space unallocated so you can create a separate NTFS partition if you ever need one for compatibility reasons?

 

[smakme7757]

Use ReFS, just make sure you enable integrity streams to benefit from self healing (on mirrored storage spaces it gets automatically enabled) and don't use the volume for sql dbs. I also run 2 file servers using ReFS data volumes for about 1 year with no problems.

I didn't know about Integrity Streams.

I looked it up in my massive Server 2012 Inside out book and they mention that it can either be enabled during formatting of the volume or set on a by folder basis.

It's a pitty i didn't see your comment before now because I'm already restoring from backup again .

This also got me thinking if I should use Storage Spaces instead of the built in motherboard RAID controller (Intel on a consumer board).

The more i read the more i feel like just dumping everything I've done and rebuilding from the ground up with all these new technologies.

 

[milee]

@hennessy1 ReFS left out some of the NTFS capabilities such as quotas, hard links, sparse files, object ids, named streams and a few more. SQL Server uses for example named streams, so ReFS is not supported for SQL Server.

But why would you put live sql db files on a general purpose storage array anyway? You can use the array for sql backups no problem, but should use a separate array for db files (like smaller disks, maybe faster, with NTFS formatted volumes).

@jack Yeah, I think you would benefit the most if you switch to Storage Spaces (mirrored not parity!) and ReFS with integrity streams ON. Don't dump everything, just set the controller to AHCI, create a pool and a mirrored space (I advise against thin provisioning) and ReFS format the VHD(s) that's it.

 

[smakme7757]

@jack Yeah, I think you would benefit the most if you switch to Storage Spaces (mirrored not parity!) and ReFS with integrity streams ON. Don't dump everything, just set the controller to AHCI, create a pool and a mirrored space (I advise against thin provisioning) and ReFS format the VHD(s) that's it.

I was not planning to use Hyper-V for the file server (virtualize it), so I'm curious as to what you mean by "and ReFS format the VHD(s)"?

 

[imagoon]

If you plan to use things like exchange / deduplication / sql / anything else that uses named streams or reparse points, NTFS. ReFS for a pure datastore is what it was designed for so it should work. ReFS also tends to not be supported [well] in with VMWare converter and won't let you sync disks prior to P2V.

I personally consider the file system "too young" for enterprise use but that may change with 2012 R2 adding some ReFS changes,

 

[milee]

Sorry for the confusion. Given a storage pool of physical disks you can create one or more virtual disks (just like you create an iSCSI virtual disk on a volume). That's why I said VHD(s), I wasn't referring to Hyper-V VHDs.

@imagoon It is young compared to many others. I think it is ready for enterprise use as it is, but for simple scenarios. I too got 2012 R2 prev of MSDN a few days ago but haven't got the time to look at it. Any news on ReFS on R2 that you know already?

 

[smakme7757]

Sorry for the confusion. Given a storage pool of physical disks you can create one or more virtual disks (just like you create an iSCSI virtual disk on a volume). That's why I said VHD(s), I wasn't referring to Hyper-V VHDs.

Not a problem thank you for the clarification

 

[imagoon]

Sorry for the confusion. Given a storage pool of physical disks you can create one or more virtual disks (just like you create an iSCSI virtual disk on a volume). That's why I said VHD(s), I wasn't referring to Hyper-V VHDs.

@imagoon It is young compared to many others. I think it is ready for enterprise use as it is, but for simple scenarios. I too got 2012 R2 prev of MSDN a few days ago but haven't got the time to look at it. Any news on ReFS on R2 that you know already?

I think ReFS itself is the same but the storage pool potions of it have been updated.

 

[milee]

I doubt ReFS makes it to R2 untouched. Even NTFS got updated more frequently back in the day. Storage Spaces is already included in Windows 8, but ReFS will be included in 2012 R2 and Windows 8.1 Client also.

@Jack What is the physical layout of that 7TB array?

 

[smakme7757]

Fresh 2012 Server install on a 160gb Intel SSD.



With

4x 1TB RE4

2x 640gb WD

1x 1.5TB wd



I've got all the disks in a single storage pool at the moment without any disks creates yet.



*mobile post

 

[milee]

So it's not 7TB net space, more like ~3TB if mirrored. Be aware that, if that's a simple storage pool (not mirrored), even if you do enable integrity streams, you won't benefit from data self healing; the file system will use copy on write for data (and metadata) but in case of a physical disk failure there's no data copy to recover it from.

Like I said, you'll benefit the most if you mirror the drives in the pool and use ReFS. For a 2-way mirror you'll get almost hardware RAID 10 performance out of your storage, with the added benefit of data protection, striped along the drives, checksummed and automatically scrubbed (with low memory consumption compared to ZFS for example).

Also, is that 1.5TB a low rpm drive? If yes, maybe you should let it out of the pool. For a mirrored pool the best practice is to use an even number of physical drives for an evenly column distribution.

 

[smakme7757]

When you say mirror the drives in the pool, do you mean creating a single virtual disk using the mirror configuration? I presume id have to use space from a pool consisting of an even number of disks.

All the drives are 7200rpm and have 64mb cache except the 2x640 drives which have 32mb.

I haven't had a chance to check but would it be a bad idea to mix different sized drives? Wouldn't a new disk be constrained by the disk with the smallest capacity?

 

[imagoon]

I doubt ReFS makes it to R2 untouched. Even NTFS got updated more frequently back in the day. Storage Spaces is already included in Windows 8, but ReFS will be included in 2012 R2 and Windows 8.1 Client also.

@Jack What is the physical layout of that 7TB array?

There still have been zero announcements about ReFS.sys itself. Everything has revolved around storage spaces which doesn't directly affect ReFS.sys.

now if they would add back linking / reparses etc it would greatly enhance its usefulness. Not being able to dedup, use it for VHDs, SQL, DFS-R really limits it.

 

[milee]

There's no restriction to use an even number of physical drives in a 2-way mirrored space like there's no restriction to use a multiple of 3 physical drives in a 3-way mirrored space.

The problem is you'll put pressure on one of the physical drives (in a 2-way mirror) which will have to read/write columns for 2 data sets in a "stripe". Maybe with a future update they will at least rotate the disk pairs to distribute the load evenly.

No, it's not a bad idea to mix different sized drives as long as you use them in pairs (for 2-way) or in multiples of 3 (for 3-way). In your case you got that covered (4x 1TB and 2x 640GB). The problem (or not) is that you'll benefit of sequential speed of a 6 drive RAID 10 array only as long as you use less than 640GB of data. Once you get past 640GB the speed will slow down to a 4 drive RAID 10 for writing AND reading the data from now on (for reading the data below 640GB you'll still have the ~3x speed 6 drive RAID 10 like).

So if you want speed consistency (unlikely for bulk storage) you should use similar drives for the entire storage space. Otherwise you're good.

 

[milee]

@imagoon Any enhancements of Storage Spaces that you know of coming in R2?

 

[smakme7757]

There's no restriction to use an even number of physical drives in a 2-way mirrored space like there's no restriction to use a multiple of 3 physical drives in a 3-way mirrored space.

The problem is you'll put pressure on one of the physical drives (in a 2-way mirror) which will have to read/write columns for 2 data sets in a "stripe". Maybe with a future update they will at least rotate the disk pairs to distribute the load evenly.

No, it's not a bad idea to mix different sized drives as long as you use them in pairs (for 2-way) or in multiples of 3 (for 3-way). In your case you got that covered (4x 1TB and 2x 640GB). The problem (or not) is that you'll benefit of sequential speed of a 6 drive RAID 10 array only as long as you use less than 640GB of data. Once you get past 640GB the speed will slow down to a 4 drive RAID 10 for writing AND reading the data from now on (for reading the data below 640GB you'll still have the ~3x speed 6 drive RAID 10 like).

So if you want speed consistency (unlikely for bulk storage) you should use similar drives for the entire storage space. Otherwise you're good.

Ok thanks for clearing that up.

I just went in and made a mirrored disk on all 6 disks and got asked if i wanted to have it as a 3 way or 2 way mirror. The speeds were good, but of course the drives have nothing on them at the moment. I could have put the 1.5TB disk in with the 4x 1TB however i think that i might just use that drive for unimportant storage.

So, as you mentioned, I will create 2 storage spaces. 1 with the 640GB drives and 1 with the 1TB drives and mirror both. Then i can create two seperate VHDs for data.

That would be a nice sound plan.

Thank you for the help!

 

[imagoon]

@imagoon Any enhancements of Storage Spaces that you know of coming in R2?

http://www.petri.co.il/new-features-windows-server-2012-r2-storage-spaces.htm

Storage tiering, parallel repairs, write back caching have been mentioned so far. I have heard but can't confirm resizing of the spaces by resizing virtual VHDX if you use those to build the storage spaces.

 

[milee]

You can't choose the physical drives to be used in a space, so you can't create 2 spaces with the drives you want in the same pool. You can instead create 2 pools each with a space (maxed out) on it. But then you loose the potential 3x seq. speed for the 640GB part.

EDIT: Sorry can't directly upload images here I see.

 

[smakme7757]

You can't choose the physical drives to be used in a space, so you can't create 2 spaces with the drives you want in the same pool. You can instead create 2 pools each with a space (maxed out) on it. But then you loose the potential 3x seq. speed for the 640GB part.

EDIT: Sorry can't directly upload images here I see.

Yea that's what i've planeed to do. 2 storage spaces.

I read here that if you used smaller disks you could only use the capacity of the smallest drive on all drives in the array even when windows displays a higher capacity of the total volume.

It's possible that this bug has been ironed out.

So, the first 640GB of the array (presuming i'm using 6 disks 4x 1TB and 2x 640GB) would recieve higher read speeds and everything after that is "reduced" to a 4 disk RAID 10 array?