802.3ad link aggregation + iSCSI

[yyrkoon]

Hello all,

I'm wondering if anyone can enlighten me concerning 802.3ad link aggregation, Basically, right now, I'm looking into buying 2 Intel Pro PT 1000 dual port GbE adapters, that sit on a 4x PCIE bus minimum. The application I'm considering these for is peer to peer(cross over cables) iSCSI Target to Initiator.

I'm wondering what kind of bottlenecks I can expect system wide on each machine, and if direct peer to peer link aggregation is even possible. I would really love to avoid an expensive switch if not needed for this situation. I do however know what link aggregation is, and does, but have no direct hands on experience with said technology. Also, If anyone has experience with this line of Intel cards, I would love to know how well they handle TCP/IP Offload.

Thanks in advance.

 

[nightowl]

802.3ad may not give you any benefit depending on the hash that is used between the host and the target. Since you will only have one MAC-address on each end and one IP address then you cannot load balance via those. Your only hope is there are multiple TCP connections going on at once and you can load balance via TCP port. If not, even if you had a switch, it still would not give you any more usable bandwidth.

 

[spidey07]

Originally posted by: nightowl
802.3ad may not give you any benefit depending on the hash that is used between the host and the target. Since you will only have one MAC-address on each end and one IP address then you cannot load balance via those. Your only hope is there are multiple TCP connections going on at once and you can load balance via TCP port. If not, even if you had a switch, it still would not give you any more usable bandwidth.

Agreed. You need to understand the layer2, 3, 4 flows going on to see if it will be any benefit at all.

The big question is why iSCSI when fiber channel works so well?

 

[yyrkoon]

The big question is why iSCSI when fiber channel works so well?

Mainly because this would be used in a home / small business environment, and I want to keep costs at a minimum. Recently I've been playing around with StarWind / MS' Initiator, and see huge potential in this technology. I was hoping for a little more "oomph" from my network, and was hoping this could help. Since I've never played around with 802.3ad, obviously I have no knowledge of Layer x networks either. I've been reading about the PCI-E 2.0, and there also seems to be promise here (think PCI-E to PCI-E bus networking. very interesting if it actually does happen).

However, there is still good news, I'm thinking I could RAID across multiple Targets (to one initiator) for increased speed as well, just have to fork out some extra cash for the hardware. All this said, I may not even need it, I was able to achieve Approx 85MB/s (RAM DISK) from Target to Initiator, peer to peer, using a cross over cable. you can read about it here if you like.

Call me a speed freak if you like, but RAID, link aggregation, and the like, has always peaked my interest.

[EDIT]
Oh, and yeah, iSCSI will work on FC networks as well (any network using TCP/IP for that matter), doesn't need proprietary equipment, and will work with any device supported on the Target machine (not just FC SCSI for example). Check out the link I gave above, maybe your imagination will show you the way If you haven't done so recently, I'd suggest reading about it, there is huge potential.

 

[yyrkoon]

So, I guess what this means if I *need* to increase my Target to Initiator bandwidth, 802.3ad will not work, and that I'll *have* to either go with FC 2.2Gbit or greater, or 10GbE equipment (currently)? No other options that I'm aware of, are there any ?

 

[theorignalAMD Overclocker]

Much more info is needed. I have done the same thing you are doing but with 4 NICs and it worked great. First, info. What is your storage setup on the server? On the client, will you have one disk or more then one? What server side software will you be using? What will your storage patterns be like? (high STR, seek, etc?)

When I had mine setup, my host software would allow you to bond ports together in different ways, round robbin, failover, etc (I cant remember the rest of the ways). That would allow you to use every link that you have. If you are using free software, good luck. I have yet to find one that is good and support bonding. I can not remember for the life of me what software I used though. If you need some more info/help, pm me or Ill check back later on.

 

[yyrkoon]

Originally posted by: theorignalamdoverclocker
Much more info is needed. I have done the same thing you are doing but with 4 NICs and it worked great. First, info. What is your storage setup on the server? On the client, will you have one disk or more then one? What server side software will you be using? What will your storage patterns be like? (high STR, seek, etc?)

When I had mine setup, my host software would allow you to bond ports together in different ways, round robbin, failover, etc (I cant remember the rest of the ways). That would allow you to use every link that you have. If you are using free software, good luck. I have yet to find one that is good and support bonding. I can not remember for the life of me what software I used though. If you need some more info/help, pm me or Ill check back later on.

Well, right now, on a single GbE connection using cross over cables, I'm achieving 85MB/s. Thats using a 700MB ram disk. Only other testing I've done, was using an img file on the Target, to the Initiator, and as far as throughput goes, only about 60% of what was achieved with the RAM disk(random access times dropped 5.x MS though!)

What I had planned on using, were two Intel Pro 1000 PT dual port server cards. According to the product sheets they DO support 802.3ad (LARC). The final Target machine would probably be implemented using Dapper 6.06 server, and the iSCSI Enterprise Target from sourceforge. I've also considered using a 8GB flash DOM for the OS. The Initiator, I planned running XP Pro, MS' Initiator 2.02. As for the connections, I had hoped that I could use cross over cables. One thing to note, is that I've read that MS' Initiator does not support Link aggregation, but thats fine, I could still use it for other things. The only problem I can see here, is that my Initiator will not have a ToE card in it.

As for anything else, hardware / OS wise, how about you tell me what you know that works, as I haven't any hands on experience here . . .