• We’re currently investigating an issue related to the forum theme and styling that is impacting page layout and visual formatting. The problem has been identified, and we are actively working on a resolution. There is no impact to user data or functionality, this is strictly a front-end display issue. We’ll post an update once the fix has been deployed. Thanks for your patience while we get this sorted.

10GE upgrade question

Todd33

Todd33

Diamond Member
We recently swapped out a 1GE for a 10GE in a dual Xeon PC connected to a Panasas file server via a 10GE switch. The improvement is near zero. What are we missing? The IT guy claims that PCs can't handle 10GE, is he correct? We are using Fedora 18 with NFS3. We don't have a Panasas file system driver yet.
 
Google "NFS Tuning" - there are a number of system settings that you need to implement before you'll be able to take advantage.

It's stuff like buffer size, frame size, and a bunch of other stuff that the existing guides do a better job explaining than I could here.
 
Your pcs are most likely connected via 1gb links, so really you will see little difference with 1 pc, where you see the difference is now the server can handle the network from multiple pcs at once without being overwhelmed. You also need to make sure the disks on the server can perform well enough that your 10gb connection isnt getting wasted by slower disks
 
shocked you are seeing no improvements.

going from 1gbe to 10gbe IPOIB i get ~5 times more performance.

and thats a 4x slot on a p45 board.
 
That should be our next step, though I'm not sure how that would work with the Panasas. We might need another computer for iperf.
 
How much was your improvement? What are you trying to improve? Were you maxing out your 1GE before the upgrade? Some numbers would be useful.
 
We were getting like 50MB/s before and after the upgrade. The Panasas can feed data a lot faster than that in theory. The data is in large files mostly, 50-300MB.
 
Something else is holding it up then. If you couldn't saturate GbE previously, upgrading to 10GbE isn't going to do a thing.
 
Todd33 said:
We were getting like 50MB/s before and after the upgrade. The Panasas can feed data a lot faster than that in theory. The data is in large files mostly, 50-300MB.
Click to expand...

We need more info about the setup and usage to be able to assist. All the network bandwidth in the world won't help if you are trying to feed it using a single 5400rpm SATA 1 drive.
 
XavierMace said:
We need more info about the setup and usage to be able to assist. All the network bandwidth in the world won't help if you are trying to feed it using a single 5400rpm SATA 1 drive.
Click to expand...

It's fed by a newish storage system by these guys https://www.panasas.com/, so plenty fast.

The computer reading it is a dual Xeon system which is reading directly into RAM, no disk IO.
 
That still tells us nothing. Which model? How many blades? Looking at their current models the basic blade is a pair of spindle drives and a single SSD. Wouldn't be the first time I've seen a blade enclosure with a single blade in it.

Do you still have GbE connections on either of these boxes? Have you confirmed the storage traffic is going over the 10GbE?
 
XavierMace said:
That still tells us nothing. Which model? How many blades? Looking at their current models the basic blade is a pair of spindle drives and a single SSD. Wouldn't be the first time I've seen a blade enclosure with a single blade in it.

Do you still have GbE connections on either of these boxes? Have you confirmed the storage traffic is going over the 10GbE?
Click to expand...

It hard for me to say much about the server, other than it cost >$50k and has 80TB. I know the computer on the other end is 10GbE, I verified that. If this was computer to computer, I could run iperf, but Panasas is beyond my control.

Hypothetically speaking, assuming the file server has infinite bandwidth, where would the bottleneck be? Is it the PCI-e bus and system RAM? The switch?
 
Right now you are saying you are getting 50MB/s. That's about normal for a single GbE connection which would seem to indicate to me you don't have a 10GbE all the way back to the storage.

You're no where near the limitations PCI-e or RAM and you state you've verified the "client" system has a 10GbE connection. Therefore the issue is either the file server or the infrastructure in between. You can't rule out the file server just because it cost lots of money.

I'm assuming we are talking copper, has anyone verified the cables?
 
Todd33 said:
It hard for me to say much about the server, other than it cost >$50k and has 80TB. I know the computer on the other end is 10GbE, I verified that. If this was computer to computer, I could run iperf, but Panasas is beyond my control.

Hypothetically speaking, assuming the file server has infinite bandwidth, where would the bottleneck be? Is it the PCI-e bus and system RAM? The switch?
Click to expand...

As stated earlier, the PCI-e is absolutely not the case.

It could be network driver, it could be firmware on the switch, it could be the file server. We need to approach this as a process of elimination since we arent versed in your environment. Is it possible to try iperf to another 10G device using the same switch path?
 
To elaborate on the PCI-e not being an issue....

My SAN has a single PCIe Quad Port 4Gb Fiber HBA running to a 4Gb switch then a Quad Port 4Gb HBA on the ESX hosts. The ESX hosts are getting 1300MB/s write speeds. Obviously FC vs Ethernet is a bit of an apples to oranges, but it shows that PCIe is not the issue.
 
XavierMace said:
Right now you are saying you are getting 50MB/s. That's about normal for a single GbE connection which would seem to indicate to me you don't have a 10GbE all the way back to the storage.

You're no where near the limitations PCI-e or RAM and you state you've verified the "client" system has a 10GbE connection. Therefore the issue is either the file server or the infrastructure in between. You can't rule out the file server just because it cost lots of money.

I'm assuming we are talking copper, has anyone verified the cables?
Click to expand...

50MB/s is 1/2 of 1 gig so he isn't even at the limits of a single 1 gig connection. Something is limiting performance here. I would lean towards the storage or something in the network layer. Also all the disk in the world isn't going to mean anything if the SAN guy only gave you a lun on a single disk.
 
XavierMace said:
To elaborate on the PCI-e not being an issue....

My SAN has a single PCIe Quad Port 4Gb Fiber HBA running to a 4Gb switch then a Quad Port 4Gb HBA on the ESX hosts. The ESX hosts are getting 1300MB/s write speeds. Obviously FC vs Ethernet is a bit of an apples to oranges, but it shows that PCIe is not the issue.
Click to expand...

Your numbers aren't adding up. 4Gb would only top out at ~500MB/sec.

The ESX hosts are probably getting 1300MBps write because VMWare hosts will sometimes use unallocated RAM as a disk cache.

(Just the first thing that comes to mind.)
 
dave_the_nerd said:
Your numbers aren't adding up. 4Gb would only top out at ~500MB/sec.

The ESX hosts are probably getting 1300MBps write because VMWare hosts will sometimes use unallocated RAM as a disk cache.

(Just the first thing that comes to mind.)
Click to expand...

Read more carefully. 🙂


Quad Port, meaning 4x 4Gb/s links.
 
imagoon said:
50MB/s is 1/2 of 1 gig so he isn't even at the limits of a single 1 gig connection. Something is limiting performance here. I would lean towards the storage or something in the network layer. Also all the disk in the world isn't going to mean anything if the SAN guy only gave you a lun on a single disk.
Click to expand...

Not talking about theoretical maxes, 50-75MB/S is pretty normal for GigE file transfers.
 
You must log in or register to reply here.

TRENDING THREADS

Back
Top