Low 10GbE throughput

[Jeff7181]

I have a HP DL380 and a backup to disk appliance. Both use Intel X520 10GbE adapters and are directly connected via fiber.

I ran a test using a utility that generates a random file on what ever destination is specified, so it takes source disk out of the equation. Running the test locally, I can see that the local physical disk on the DL380 is capable of writing at about 400 MB/sec. When I ran it against SAN attached disk, I was able to write at about 500 MB/sec, which is the limit of the 4 Gbps fiber channel HBA's we use. When I ran it against the backup to disk appliance via the 10 Gig Ethernet connection, it was only able to write at about 330 MB/sec. The vendor says the disk subsystem is capable of ingesting over 1000 MB/sec, so I should be able to almost saturate the 10GbE NIC, but I can't. I even tried multiple streams of data.

At the manufacturer's request I ran iperf and tweaked TCP window size and achieved a maximum of 440 MB/sec, which is better, but still less than half what the connection should be capable of.

Anyone have any experience tuning/troubleshooting 10GbE performance?

 

[drebo]

Larger MTU would give you better throughput on file transfers in general, depending on the protocol.

Storage networking is voodoo science, really.

 

[Jeff7181]

I should also mention that the DL380 is running Windows Server 2008 R2

 

[Jeff7181]

I found a driver setting that Intel recommended changing, then change the tcpwindow size to 64KB and with 4 streams I can max the link now. Sweet... 64.5 GB transferred in 60 seconds.

 

[Jeff7181]

Dammit... still having issues. When I use iperf to set the TCPWindowSize I can max the 10 Gbit connection. With anything else I can't.

 

[Fardringle]

iperf sends raw data directly over the connection. Actual file transfers depend on the speed of your drives as well. If your drives (on both ends) can't read and write the data that quickly then you won't be able to max out the connection. While the manufacturer of your SAN device claims that the disk array is able to collect data faster than 1000 MB/s, that doesn't necessarily mean it is true. And even if it was true, do you have any individual client machines with disk drives that are capable of sending the data that quickly? Most computers would have trouble saturating a regular gigabit connection. Not many high end servers could max out a 10 gigabit connection on their own.

The advantage of a super fast connection and disk array is that it can take data from multiple sources at the same time at the maximum speed possible for all of the clients simultaneously. To truly test the capabilities of your disk storage system, you'll probably have to throw data at it from several different systems all at the same time unless your server is set up with an incredibly fast disk array as well.

 

[freegeeks]

don't believe what the vendor is telling you. I had to install and load test a high end clustered NAS once and real-life performance was 50% of what the vendor was claiming.
Most figures that vendors throw are you are what a system can achieve under ideal circumstances and does not reflect real-life performance at all

 

[Emulex]

you don't have a switch in between?

remember 10gbe is not simple 10x faster than gigabit. IOPS with iscsi - that is more realistic

Dynamic power can affect overall i/o performance on the machine - you might play with those settings to see what gives you best results.

 

[Jeff7181]

All power saving features are disabled on the server and I couldn't find any specifically for the NIC.

No, there is no switch between. We don't have any existing 10 GbE infrastructure so we ran fiber directly from the backup server to the D2D device.

The specific volumes I'm testing with are on aggregates with a minimum of 40 spindles, some have nearly 100. I've also seen our old SAN saturate a 4 Gb fiber channel link so I'm pretty confident our new $2.5 million SAN is capable of at least that, especially with 8 Gb fiber channel HBA's.

I figured our situation was more friendly to the link than iSCSI because the IOPS are not as big a concern as throughput. It would be quite disappointing to only get 30% of the rated throughput.

 

[drebo]

If throughput is your concern, you'll need to increase your MTU size.

 

[Jeff7181]

If throughput is your concern, you'll need to increase your MTU size.

Yesterday I tried jumbo frames and a MTU of 9000. No change. I even connected a second 10GbE link and the 3 Gbps cap I seem to be hitting was split between the two. I'm going to try a couple other different methods of transferring the data today to determine whether it's the backup software. Otherwise I think I'm going to look into replacing the DL380's with DL360's since the servers are mostly idle while running backups at 3 Gbps.

 

[Mark R]

Can you get any diagnostic information off the appliance?

One thing I've run into with NASs and similar appliances, is that even though the disk susbsystem may be fantastic, there may be a bottleneck elsewhere. For instance, on a number of NASs that I've tested, the CPU is the weak spot. The system will end-up throttled with 100% CPU utilization at 50% network utilization and 20% disk time.

 

[Jeff7181]

The appliance is barely working to ingest 3 Gbps. CPU usage is below 20%, now that it has data to dedupe against, disk writes are extremely low. I'm sending about 1 TB to it every day and it's writing 20 GB of new data to disk.