Solved! Switch (Arista) And NIC (Mellanox) Compatibility Question

[TryingToVirtualizeStuff]

Hello,

My company plans on building a system with the following NIC: Mellanox MCX653106A-HDAT

NVIDIA Mellanox: End-to-End Interconnect Networking Solutions

The leading supplier of end-to-end Ethernet and InfiniBand intelligent interconnect solutions and services.

store.nvidia.com

...and I was wondering whether it would work with the following switch: Arista 7050SX-72Q

https://www.arista.com/assets/data/pdf/Datasheets/7050SX-128_64_Datasheet.pdf



Even though the switch is older (and can't handle the high throughput of the Mellanox), the plan is to eventually replace the Arista (but for now, just use the Mellanox at a lower throughput),

I assume the Arista and the Mellanox should be able to auto-negotiate with one another, but one of the Engineers on my team mentioned that QSFP+ (on the Arista) couldn't work with QSFP56 (on the Mellanox).

In case the Mellanox doesn't work with the Arista, could someone recommend the highest throughput NIC that can work with the Arista?

Any help would be greatly appreciated.

Thank you,
Nelson

 

[TryingToVirtualizeStuff]

View attachment 57409

Looks like there's a limit to how much you can push through the Arista compared to the Mellanox.

Mellanox up to 200gbps/port
View attachment 57410
Arisa up to 10gbps/port SFP+ or 40gbps/port QSFP

I would probably shoot for 2 x 40gbps bundled into a port-channel / LACP bundle to get the most out of it.

Hello Tech Junky,

Understood (thank you for the clarification).

Just to be certain though, in your post you mentioned looking for NIC's with QSFP ports.

Did you really mean QSFP+, or should I just look for QSFP (as the documentation above seems to mention QSFP+ ports)?

Thank you,
Nelson

 

[Tech Junky]

I just left off the + since most new cards are automatically +

Now POE on the other hand can be + or ++ depending on the power demand of the endpoint.

The main consideration is both sides can meet the same speed 40<>40

 

[TryingToVirtualizeStuff]

I just left off the + since most new cards are automatically +

Now POE on the other hand can be + or ++ depending on the power demand of the endpoint.

The main consideration is both sides can meet the same speed 40<>40

Hello Tech Junky (or anyone else),

I found two cards which should work, between the following two cards: St. Helens 200G4Q-XL

HotLava > Ethernet Adapters > Maximize Your Network's Bandwidth

Inquire 40/56 Gb Ethernet Adapter Cards from HotLava Systems. Eliminate I/O bottlenecks and maximize your network's bandwidth at the lowest price cost-per-port available.

www.hotlavasystems.com

...or: PE31640G4QI71 Server Adapter

Silicom Ltd. | 40G Networking Server Adapter PE31640G4QI71

Silicom’s Quad 40G Networking Server Adapter Ethernet PCI Express are designed for Servers and high-end appliances. Silicom's Quad 40Gigabit offer simple

www.silicom-usa.com

...do you prefer/recommend a specific card?

Thank you once again for all your help,
Nelson

 

[Tech Junky]

If you're planning on changing the switch then stick with the Mellanox card. It will do the 40gbps speed now and allow for 200gbps per port down the road with the new switch if it supports the speed.

These days though most any NIC should be up to the job unless you spot a ton of reviews saying it's not. I tend to aim for the cheapest option personally I don't see the need to pay $400 for an Intel based NIC when I can get a QNAP for 50% less that does the job just as well.

2 Port 40G QSFP+ Network Card Intel PCIe - Network Adapter Cards | Networking IO Products | StarTech.com

Dual-Port QSFP+ Server NIC Card - PCI Express - Intel XL710 Chip - 40G Network Interface Card - 40Gb Network Card - NIC Card - PCIe Network Card

www.startech.com

I've used Startech devices before and they work great in niche setups when no one else carries what you're looking for. $650 sounds a whole lo better than $1200-$1800 for the same functionality. Diving into pricing though there's probably a better deal to be found somewhere.

LAN-40G2SF-MLX | Flexible, fast, economical

QNAP offers various cost-effective network expansion cards for businesses and home users to easily upgrade the bandwidth of their QNAP NAS, servers and computers to accommodate high-speed data transfer and bandwidth-hungry applications.

www.qnap.com

$500

QXG-100G2SF-E810 | Ultra-high-speed 100Gbps network connectivity

The dual-port QXG-100G2SF-E810 100 GbE network expansion card with Intel® Ethernet Controller E810 supports PCIe 4.0 and provides up to 100Gbps bandwidth to overcome performance bottlenecks. With iWARP/RDMA and SR-IOV support (available soon), the QXG-100G2SF-E810 greatly boosts network...

www.qnap.com

$900

I'm using QXG-5G4T-111C for my server setup w/ NAS in Raid 10 because it allows for enough overhead and isn't a bottleneck for the array @ 400MB/s + additional traffic while moving files around. I've been using it for about a year now and it's been flawless in Linux. The only issue I've come across is certain kernel versions don't play well with it on occasion as I tend to upgrade often for the security patches as this box is my router.

 

[KentState]

QSFP+ is the old 40G standard and not future proof. You would want to look at something like QSF28. If you do go the older route, buy used since companies are dumping them for newer, more efficient technologies. I have picked up a ton of Chelsio dual port QSFP+ for home use at home. Even the Arista swtiches can be found for under $700 on ebay in 16 and 32 port configurations. If you need 10Gbe, they make fan out cables as well.