What is a good SAN (or Cluster) File System for Windows XP?

[Brazen]

It is necessary for us to connect a couple Windows XP desktops to an iSCSI SAN. Does anybody know of a good open source SAN File System?

I'm looking at Dataplow's SFS (waiting on pricing now), but I would much more interested in an open source alternative.

 

[Smilin]

uh.

If you want XP to read the filesystem it is going to have to be NTFS or some flavor of FAT.

Are you sure you aren't asking about what iSCSI initiator to use? Are you using the free one from Microsoft?

 

[Brazen]

Smilin, sorry but you are mistaken...

I will be using the free initiator from Microsoft. With iSCSI you can have multiple computers connect to the same target, which shows up as a "local" harddrive on each computer. If you format this drive as NTFS or FAT then you will have problems when more than one computer is trying to use the drive at the same time. You have to format it as a SAN-aware file system. There are a few proprietary ones for Windows and Linux (ext2/3, XFS, ReiserFS and most common ones have the same problem) and one open source one that I know of for linux, called GFS or Global Files System.

 

[Smilin]

So I am mistaken when I say:

"If you want XP to read the filesystem it is going to have to be NTFS or some flavor of FAT. " ???

If you intend to have two Windows machines access the same physical (or iSCSI) drive you'll need clustering services that are not available with XP.

Having two machines access the same drive that are not communicating with each other in some form cluster is asking for trouble and it is not at all what iSCSI was meant for. If more than one machine needs access to the drive at the same time, mount the drive on a single machine and share it out. You can then leverage oplocks or some other method to allow graceful simultaneous access.

I'll stand by my earlier statement though: For XP to read the drive it's going to have to be NTFS or FAT

 

[Meursault]

Originally posted by: Smilin
So I am mistaken when I say:

"If you want XP to read the filesystem it is going to have to be NTFS or some flavor of FAT. " ???

Yes, you are. A non-NTFS/FAT filesystem can abstract that requirement away and XP doesn't know the difference.

 

[Brazen]

Originally posted by: Smilin
So I am mistaken when I say:

"If you want XP to read the filesystem it is going to have to be NTFS or some flavor of FAT. " ???

Well, yes and no. Those are the file systems that come with Windows and 90% of the time work for whatever you need. However, you can install other file systems, just like installing any other software. For instance, the Dataplow SFS I mentioned above; there is even experimental software to allow you to use ext2/3 on Windows (ext2/3 are not SAN-aware BTW).

If you intend to have two Windows machines access the same physical (or iSCSI) drive you'll need clustering services that are not available with XP.

Clustering Services is a sloppy way Microsoft tacked on to make up for their lack of a SAN-away file system.

Having two machines access the same drive that are not communicating with each other in some form cluster is asking for trouble and it is not at all what iSCSI was meant for. If more than one machine needs access to the drive at the same time, mount the drive on a single machine and share it out. You can then leverage oplocks or some other method to allow graceful simultaneous access.

Two machines accessing the same SAN LUN is asking for trouble, unless you are using a SAN file system. And this is exactly what iSCSI was meant for. We can not use a file share to due limitations in a proprietary piece of software. A SAN file system is designed to "allow graceful simultaneous access." This is commonly done on Linux using GFS. If you are familiar with VMWare ESX Server, ESX servers are almost ALWAYS set up to share SAN LUNs using VMFS.

I'll stand by my earlier statement though: For XP to read the drive it's going to have to be NTFS or FAT

Well, you might as well accept that you just learned something new then.

 

[drag]

Ya.. For a SAN via ISCSI the system should see it as a local harddrive device.. It doesn't have anything to do with any file system abstractions or whatnot, it is simply used a harddrive device.

There are two general ways that file systems can be done over a network, as per my understanding..
block-level
and
object-level

Block-level storage system exports the item out block by block.. Like harddrive sectors or something like that. Two examples of this are iSCSI and Fiberchannel. Of course fiberchannel has the advantage of being much faster, but iSCSI is much cheaper.

You would format those as you normally do any other drive. The advantages of those things are good performance especially for things that access drives in special ways like database systems. However for other types of operations more traditional network file system may still be faster.

Now if you format it NTFS it will work fine, but you won't be able to share that paticular iscsi export with anything else. NTFS is a single-system operating system and having multiple operating systems trying to access it at the same time will cause problems. NTFS is designed and optimized so that the operating system kernel has full control over the file system for best performance and best reliability. If you have multiple operating system kernels trying to run the same NTFS system at the time then there is no coordination or anything like that. Seemlingly files will move around on their own and such, which would lead to corruptions and crashing.

So in other words if you use NTFS for a ISCSI export then only one machine at a time can use it, which is great depending on what you want to do.


Object-level things are like SMB (windows file sharing) or NFS. Also network file systems.These things export their own special file system for allowing many people to connect and share files. They may be a bit slower for some things and a bit faster for others vs Iscsi (due to their more network-aware cache'ng mechanisms and such), although SMB is pretty much always just slow. Of course since these things use their own abstractions then it's designed to easily scale up to thousands of clients.


Now it's possible to use a block-level share and allow access from multiple machines to the same storage volume at the same time. For that you need a special type of file system called 'Cluster File System'. These things are generally limited to only a few dozen operating systems accessing them at the same time. They are effectively a 'native' file system that is able to handle multiple operating systems at the same time.

In Linux if you wanted to use open source systems you would generally use GFS (from Redhat) or OCFSv2 (from Oracle, currently in-kernel). For instance you can use OCFSv2 to setup a load balancing/failover situation were you can have multiple Oracle database servers sharing the same backend over fiberchannel on some fast SAN.

In Windows this is certainly very possible too with something like Veratis CFS or Polyserve, but I don't know of any open source things for Windows that will do that.

 

[Brazen]

Originally posted by: drag

but iSCSI is much cheaper.

And you don't have to run fiber all over the place... well, I guess you could call that cheaper also. I call it convenient .

For that you need a special type of file system called 'Cluster File System'.

Or a 'SAN File System.' I'm not sure which is the more common/proper term.

In Windows this is certainly very possible too with something like Veratis CFS or Polyserve, but I don't know of any open source things for Windows that will do that.

Veritas CFS or Polyserver? Any comment on which of these is good/better/cheaper/common? Edit: Well it looks like Veritas CFS is actually *nix only and Polyserve is server OSes only. I guess Dataplow will have to do.

 

[drag]

Veritas CFS or Polyserver? Any comment on which of these is good/better/cheaper/common?

Absolutely NO clue. None what-so-ever. I am only familar from the Linux/OSS side of things.

I suppose it depends on your purpose also.. For instance Oracle has their own OCFS for doing Oracle RAC setups. I wouldn't be suprised if other vendors had their own application-specific stuff.

Don't know. Hopefully some helpfull person with Windows enterprise experiance will come along and enlighten us.

edit:

Actually this seems a perfect question for 'Experts Exchange' http://www.experts-exchange.com/
I think you can find some pretty good answers there from knowledgable people.
(it's free to look up questions, it's free to answer questions, but it's not free to ask questions. You need credits and you can get those by paying for them or by successfully answering questions)

 

[Brazen]

Originally posted by: drag

Actually this seems a perfect question for 'Experts Exchange' http://www.experts-exchange.com/
I think you can find some pretty good answers there from knowledgable people.
(it's free to look up questions, it's free to answer questions, but it's not free to ask questions. You need credits and you can get those by paying for them or by successfully answering questions)

Unless I'm mistaken, don't you have to pay $100/year to sign up on Experts-exchange?

 

[drag]

Originally posted by: Brazen

Originally posted by: drag

Actually this seems a perfect question for 'Experts Exchange' http://www.experts-exchange.com/
I think you can find some pretty good answers there from knowledgable people.
(it's free to look up questions, it's free to answer questions, but it's not free to ask questions. You need credits and you can get those by paying for them or by successfully answering questions)

Unless I'm mistaken, don't you have to pay $100/year to sign up on Experts-exchange?

Nope. Although it looks that way.

They do a good job of hiding it, but if you go to their register page by clicking on the link on their front page and you look at the fine print on the bottom of the page there is a link under 'Earn your premium membership' or some such thing.

Following that will get you to their free sign up page.

Like I said they work on a system of credits.. if you get the free service then you need to answer about 12 or so questions effectively (so that the person chooses your reply as the real answer) before you can start asking questions. They call that 'earning' your 'premium' membership.

 

[Brazen]

Originally posted by: drag

Originally posted by: Brazen

Originally posted by: drag

Actually this seems a perfect question for 'Experts Exchange' http://www.experts-exchange.com/
I think you can find some pretty good answers there from knowledgable people.
(it's free to look up questions, it's free to answer questions, but it's not free to ask questions. You need credits and you can get those by paying for them or by successfully answering questions)

Unless I'm mistaken, don't you have to pay $100/year to sign up on Experts-exchange?

Nope. Although it looks that way.

They do a good job of hiding it, but if you go to their register page by clicking on the link on their front page and you look at the fine print on the bottom of the page there is a link under 'Earn your premium membership' or some such thing.

Following that will get you to their free sign up page.

Like I said they work on a system of credits.. if you get the free service then you need to answer about 12 or so questions effectively (so that the person chooses your reply as the real answer) before you can start asking questions. They call that 'earning' your 'premium' membership.

Sweet. You might find me asking a lot more questions here at ATF Just kidding!

 

[Smilin]

Originally posted by: Brazen

Originally posted by: Smilin

I'll stand by my earlier statement though: For XP to read the drive it's going to have to be NTFS or FAT

Well, you might as well accept that you just learned something new then.

Sorry, I still haven't learned anything I didn't already know but thanks for the lecture.

Your XP box is going to have to see a drive formatted with NTFS or FAT for it to work. You can run NTFS and have two clients mount the same drive without clustering, put a 3rd party filesystem driver on the client or several other options but it will just become an excercise in how to break a system. People do not typically invest the cash required for a SAN and then jury rig some crap to get access to it. Do it the right way.

 

[Brazen]

Originally posted by: Smilin

Originally posted by: Brazen

Originally posted by: Smilin

I'll stand by my earlier statement though: For XP to read the drive it's going to have to be NTFS or FAT

Well, you might as well accept that you just learned something new then.

Sorry, I still haven't learned anything I didn't already know but thanks for the lecture.

Your XP box is going to have to see a drive formatted with NTFS or FAT for it to work. You can run NTFS and have two clients mount the same drive without clustering, put a 3rd party filesystem driver on the client or several other options but it will just become an excercise in how to break a system. People do not typically invest the cash required for a SAN and then jury rig some crap to get access to it. Do it the right way.

You are only showing how little you know about this subject... Go back and read through Dag and my discussion and maybe you'll learn something. Maybe this will help, too.

 

[Smilin]

It's so cute how you keep lecturing me on SAN and iSCSI and providing me with these links to very basic information.

I did actually go back and read through your discussion with Drag and found this little quote from him: "Hopefully some helpfull person with Windows enterprise experiance will come along and enlighten us"

Just so happens I spent a year here at Microsoft in the Enterprise Platforms Support Setup team that is the final point of escalation for storage, SANs, iSCSI and whatnot. Heck I was even involved in the bugfixes that took the iSCSI initiator from 1.1 to 1.3 I'm telling you that Hooking two XP boxes to the same LUN = broke and that you should reconsider what you are trying to do, but hey whatever. You seem more interested in a good argument than an answer so good luck to ya.

 

[drag]

Originally posted by: Smilin
It's so cute how you keep lecturing me on SAN and iSCSI and providing me with these links to very basic information.

I did actually go back and read through your discussion with Drag and found this little quote from him: "Hopefully some helpfull person with Windows enterprise experiance will come along and enlighten us"

Just so happens I spent a year here at Microsoft in the Enterprise Platforms Support Setup team that is the final point of escalation for storage, SANs, iSCSI and whatnot. Heck I was even involved in the bugfixes that took the iSCSI initiator from 1.1 to 1.3 I'm telling you that Hooking two XP boxes to the same LUN = broke and that you should reconsider what you are trying to do, but hey whatever. You seem more interested in a good argument than an answer so good luck to ya.

So your saying it's a iSCSI issue?

That sharing a iSCSI target between two hosts is just a very bad idea with Windows?

Now is this just a problem with iSCSI in general, or is it a problem with Microsoft's software iSCSI initiator (say if you used a 'real' initiator it wouldn't be a big deal), or is it a problem with how Windows deals with file systems in general?

 

[Smilin]

It's not an iSCSI problem at all really. You'll just have fits with XP sharing a local drive (iSCSI or otherwise) with another XP box. Expect data corruption and a stack of 5x events in your system logs (maybe some 7,9 & 11s too)

 

[kamper]

Originally posted by: Smilin
It's not an iSCSI problem at all really. You'll just have fits with XP sharing a local drive (iSCSI or otherwise) with another XP box. Expect data corruption and a stack of 5x events in your system logs (maybe some 7,9 & 11s too)

Certainly I'm way out of my league here, but I'm confused by your explanations. Are you saying that:
1) there are no filesystems that are designed for concurrent iSCSI access from multiple machines (such as Brazen's linux example of GFS and drag's example of OCFS)?
2) assuming that 1) does exist for other operating systems, it is not possible to install a driver for such a filesystem on a windows machine (given that such things have been developed)?
It just seems to me like you're not quite arguing about the same things.

 

[Smilin]

1) Yep stuff out there that can do it.
2) Possible, but NOT recommended. The filesystems that are supported by XP have very mature drivers and third party filesystem drivers for Windows are junk (I could have a slanted point of view because I only took calls on this when things were failing).

I am not entirely sure what he is trying to do but with the limited info I have:
1. Two xp boxes hitting the same physical drive will cause problems (could be mild or catastrophic depending on what you are trying to do).
2. Better to mount the drive on a single box and share it out so that SMB oplocks can handle contention.
3. If it truly needs to be shared as a physical drive, skip XP and drop 2k or 2k3 on there and cluster it.

The only bit of info I have is he is using a SAN. That fact shifts my mindset to "reliable performance" mode instead of "duct-tape to get it working" mode. If duct-tape is acceptable, why bother to invest in a SAN?

My views and experience on this are very Microsoft-centric and this is the Microsoft-centric answer. From my experience Drag has great Linux Kung-Fu so I would lean on him for *nix answers.

 

[Brazen]

Originally posted by: kamper

Originally posted by: Smilin
It's not an iSCSI problem at all really. You'll just have fits with XP sharing a local drive (iSCSI or otherwise) with another XP box. Expect data corruption and a stack of 5x events in your system logs (maybe some 7,9 & 11s too)

Certainly I'm way out of my league here, but I'm confused by your explanations. Are you saying that:
1) there are no filesystems that are designed for concurrent iSCSI access from multiple machines (such as Brazen's linux example of GFS and drag's example of OCFS)?

Yes, such filesystems do exist for Windows. More from Wikipedia, check out the "Shared disk file system" section.

2) assuming that 1) does exist for other operating systems, it is not possible to install a driver for such a filesystem on a windows machine (given that such things have been developed)?

Yes it is possible, as the above links I provided will attest to.

There are more options than FAT or NTFS, if you are willing to shell out the monies (Dataplow SFS turns out to be $4000 per license!) or use the open source (experimental) ext2/3 file system driver on Windows. But Dataplow SFS is not a "duct-tape" solution, Microsoft Cluster Services is a duct-tape solution to Microsoft's lack of a SAN file system driver. And as Smilin admitted he has a Microsoft-centric view. Maybe I have a linux-centric view, but I admin both Windows and Linux servers, and I don't base my solutions on FUD, I base them on what is the best tools for the job.

 

[kamper]

Originally posted by: Brazen
There are more options than FAT or NTFS, if you are willing to shell out the monies (Dataplow SFS turns out to be $4000 per license!) or use the open source (experimental) ext2/3 file system driver on Windows. But Dataplow SFS is not a "duct-tape" solution, Microsoft Cluster Services is a duct-tape solution to Microsoft's lack of a SAN file system driver. And as Smilin admitted he has a Microsoft-centric view. Maybe I have a linux-centric view, but I admin both Windows and Linux servers, and I don't base my solutions on FUD, I base them on what is the best tools for the job.

So Smilin is claiming that there are no reliable 3rd party filesystem drivers for windows. It would seem he's got a lot of experience so rather than dismissing his help (I read your last post pre-edit ) why not directly ask him about Dataplow? Have you used it yourself? Is there anything other than the large license fee that indicates its quality? Smilin, is it possible that there are just options out there that you haven't come across?

 

[Brazen]

Originally posted by: Smilin

I am not entirely sure what he is trying to do but with the limited info I have:
1. Two xp boxes hitting the same physical drive will cause problems (could be mild or catastrophic depending on what you are trying to do).

As has already been stated, NOT if you use a SAN file system.

2. Better to mount the drive on a single box and share it out so that SMB oplocks can handle contention.

A very niche application (CAD software) requires block level access to the files, and an object level sharing protocol like SMB will not cut it.

3. If it truly needs to be shared as a physical drive, skip XP and drop 2k or 2k3 on there and cluster it.

THAT would be a duct-tape solution.

 

[Smilin]

Originally posted by: Brazen
It just seems that Smilin doesn't understand how file systems work, or about file system drivers. There are more options than FAT or NTFS, if you are willing to shell out the monies (Dataplow SFS turns out to be $4000 per license!) or use the open source (experimental) ext2/3 file system driver on Windows.

:roll:

At least you edited your post to take that statement out.

I'm sorry we have ended up mildly perturbed at each other here. It was my intention to steer you away from the path you are headed down, not be an iron-fisted SAN Nazi You know more about what it is you are trying to do so you are in the best position to evaluate your risks.

One last point of contention though: Microsoft clusters have over a decade of maturity and are far from a duct tape solution. There is some really, really big iron out there running our clustering solutions with five nines of uptime being quite common. Don't dismiss it with some FUD.

 

[Brazen]

Originally posted by: kamper

So Smilin is claiming that there are no reliable 3rd party filesystem drivers for windows. It would seem he's got a lot of experience so rather than dismissing his help (I read your last post pre-edit )

Ha, well, I posted it before I read his last post, in which he seemed more... amiable.

why not directly ask him about Dataplow?

The whole point of this thread was to get input on SAN file system, which Smilin staunchly denied even existed; which, of course, they undeniably do.

Have you used it yourself? Is there anything other than the large license fee that indicates its quality?

I have googled around on this subject and on Dataplow specifically and have found nothing but praise for Dataplow SFS. I have not used it personally though, and at that price, I am exploring other options.

 

[Brazen]

Originally posted by: Smilin

One last point of contention though: Microsoft clusters have over a decade of maturity and are far from a duct tape solution. There is some really, really big iron out there running our clustering solutions with five nines of uptime being quite common. Don't dismiss it with some FUD.

Touche, but it is not the answer to every problem.