Question Are we ready for 10 Gbe networks? I just want to chitchat

[iamgenius]

Hi networking gurus.

I'm a guy who just hates slowness

Whenever there is a way to speed things up, I'll just do it. I have been toying with the idea of converting my 1Gb/s LAN into a 10 Gb/s LAN. Of course I will need to change my main router and NIC's in my machines. I save all my data into a NAS device, and this NAS backs up itself to another similar NAS in the same network. Not the very best solution but it works. I also move do a manual backup to a local drive and move files between my machines. I will be limited of course by the maximum transfer rate of my mechanical HDD's and the SATA interface transfer rate. But my nvme ssd's configured in RAID 0 can still utilize the 10 Gbps connection. It is very nice seeing huge files get transferred immediately. Are 10gbe routers/switches readily available? Are there NAS devices with 10 Gbps network connection? When I search, I still don't see them as common. Maybe we are still not ready? But 10 Gbe has been around for some time now.

I won't even mind building a NAS devices based on ssd's, heheh !

Let's chitchat

 

[SamirD]

10Gb is plenty in the used market and quite stable (and relatively cheap actually).

You can get 10Gb nics with SFPs and DACs and move at 10Gb pretty cheaply. You can also find older enterprise used 10Gb switches for decent prices.

No reason to still be on 1Gbps if you have stuff that can move at 10Gbps speeds. Just spend some money and enjoy it.

 

[Shmee]

What NAS do you use? I find that if you have a decent home built freenas or equivalent, with enough RAM, even a mechanical HDD setup can be very fast due to caching. My freenas is based on an X58 Rig with a Xeon X5660, 18GB of RAM, and a ZFS pool of 5 WD Red Pro drives. It has a 2.5GbE card, (my X570 motherboard has a 2.5GbE as well) and transfers between are still largely limited by the network speed of 2.5Gb/s. I suppose I could go with 10GbE later on, but for now what I have is fine, 2.5Gb/s is still very fast.

 

[iamgenius]

My NAS is Synology DS1618+. This one:

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjI-6uT6s_uAhXiQhUIHTlxDVQQFjABegQIARAC&url=https%3A%2F%2Fglobal.download.synology.com%2Fdownload%2FDocument%2FHardware%2FDataSheet%2FDiskStation%2F18-year%2FDS1618%2B%2Fenu%2FSynology_DS1618_Plus_Data_Sheet_enu.pdf&usg=AOvVaw3ofdRC7dbJOcKKfWkR360D

It seems like I can upgrade it to 10GbE with an add-on card, right? Can I do this while also having an SSD for caching? Looks like I'm ready to go, upgrade my two NAS's and buy 10GbE cards for my rigs, and change the router.

 

[SamirD]

My NAS is Synology DS1618+. This one:

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjI-6uT6s_uAhXiQhUIHTlxDVQQFjABegQIARAC&url=https%3A%2F%2Fglobal.download.synology.com%2Fdownload%2FDocument%2FHardware%2FDataSheet%2FDiskStation%2F18-year%2FDS1618%2B%2Fenu%2FSynology_DS1618_Plus_Data_Sheet_enu.pdf&usg=AOvVaw3ofdRC7dbJOcKKfWkR360D

It seems like I can upgrade it to 10GbE with an add-on card, right? Can I do this while also having an SSD for caching? Looks like I'm ready to go, upgrade my two NAS's and buy 10GbE cards for my rigs, and change the router.

Yep, and you don't have to change the router--you can get cards that can use sfps and use dacs (direct attach cables) or just get a 10Gb switch and some sfps and pre-terminated fibre.

 

[iamgenius]

Yep, and you don't have to change the router--you can get cards that can use sfps and use dacs (direct attach cables) or just get a 10Gb switch and some sfps and pre-terminated fibre.

This is where I will need some help. Why is it that I don't have to change the router? I have my main switch connected to my router. If I change the switch and later on have machines connected to it...Will a machine connected to the new switch communicate with another connected directly to the router using a 10Gbps connection? Or you mean all devices that need to utilize the 10Gbps connection will need to connect to the new switch. I'm under the impression that my 1Gbps router will be the limiting factor.

I have a very simple network. A wireless router that is serving ip addresses via its dhcp and also working as switch + another 1Gbps switch to extend the network. Some devices are hooked by cables and the rest (Mobile phones and laptops) are connected wirelessly.

I haven't used sfps and dacs before but I will read about them will be ready to try them out.

 

[aigomorla]

Why is it that I don't have to change the router

Because unless you absolutely need every last PC on that 10gbe connection, you can do a direct single PC to NAS.
Again unless you want every last PC in your network to have that 10gbe, which honestly i think its sort of pointless.

Also wireless will never even get close to saturating 1gbe let alone a 10gbe unless your like 3 foot from a wifi-6 connection.
Typically you want that 10gbe only on 1 pc to a NAS, and not just that, you also want your NAS to be fast enough on the I/O department in disk speed to handle a 10gbe, which means u have some form of RAID on that NAS otherwise again, your disks will be your weakest link.

 

[iamgenius]

Because unless you absolutely need every last PC on that 10gbe connection, you can do a direct single PC to NAS.
Again unless you want every last PC in your network to have that 10gbe, which honestly i think its sort of pointless.

Also wireless will never even get close to saturating 1gbe let alone a 10gbe unless your like 3 foot from a wifi-6 connection.
Typically you want that 10gbe only on 1 pc to a NAS, and not just that, you also want your NAS to be fast enough on the I/O department in disk speed to handle a 10gbe, which means u have some form of RAID on that NAS otherwise again, your disks will be your weakest link.

The wireless devices connect to the network mostly for internet access, so there is no need for anything higher than 1Gpbs as my internet connection speed from ISP is 500 Mbps. I get from 30 to 320 Mbps depending on distance and which AP I'm connected to.

I want to utilize 10Gbps for my wired rigs. I have three and they are where I generate my data and download stuff. What they generate goes into the first NAS. What's inside the first NAS gets backed up everyday to the 2nd NAS that is slightly in a different location.

Let's skip sfps and DACs, then my question will be : If I equip each of my desktop machines and my 2 NASs with a 10 Gbps NIC, hook them all up to a 10 Gbps switch using cat6 or cat7 cables, will they communicate in 10Gbps speeds? The switch will connect to my router of course for internet access and DHCP server. This bit I still don't fully understand, please bear with.

And you are right about NAS disk speed, but caching can help here I think?

And I still need to read about sfps and DACs and explore their option. I think I will just draw a simple diagram for my network so that you can better help me.

Thanks to all.

 

[aigomorla]

Let's skip sfps and DACs, then my question will be : If I equip each of my desktop machines and my 2 NASs with a 10 Gbps NIC, hook them all up to a 10 Gbps switch using cat6 or cat7 cables, will they communicate in 10Gbps speeds? The switch will connect to my router of course for internet access and DHCP server. This bit I still don't fully understand, please bear with.

yes if you have a 10gbe switch, all the pc's connected to that switch will have 10gbe as they will communicate though that switch.
But also keep note, if none of the PC's can handle that bandwith, its again pointless.

Single magnetic disks cap at around 170-200mb/s.
Single SATA SSD Caps at around 500mb/s
Single Gen.3 nVME can saturate a 10gbe, but you run into issues of storage space and if you RAID these guys for more storage space, you then run into an issue of PCI-E lanes.

RAID-10 will give you around 400-800mb's depending on how many disks you are running and what controller they are on.
Here is my RAID-10 speed

And if you run a RAM Disk, you can saturate that 10gb/e with speeds like this:

But again, if the source and destination can not handle the read and writes speeds on that 10gbe.... well, its sort of a silly and expensive network upgrade to go full 10GBase-T as the switchs are VERY EXPENSIVE over going SFP+.

This is why some people find 2.5GB/e more sensible, as things are a bit cheaper, but not yet floor cheap like 1gbe.
The 2.5gbe switch will still cost you, but most high end desktop boards now offer a 2.5gbe nic.

 

[iamgenius]

Will something like this work:

https://www.amazon.com/NETGEAR-Ethernet-Unmanaged-Multi-Gig-Protection/dp/B076642YPN/ref=pd_ybh_a_12?_encoding=UTF8&psc=1&refRID=3Z4RPCWRJSPXS0KAF8C5

??

If it does, then I will need some of

https://www.amazon.com/Synology-10Gb-Ethernet-Adapter-E10G18-T1/dp/B07G9N9KJT/ref=pd_ybh_a_5?_encoding=UTF8&psc=1&refRID=VY7JAW7V7M6G456W55K4

and

https://www.amazon.com/TRENDnet-Supports-Standard-Low-Profile-TEG-10GECTX/dp/B01N5AOWW6/ref=pd_ybh_a_13?_encoding=UTF8&psc=1&refRID=RV060QNB2N7BJWRY7JBS

That won't be too expensive, and I don't really have to saturate the link.

Will be back, got to run somewhere!

 

[Shmee]

Keep in mind that switch only has 2 10G ports.

 

[iamgenius]

Keep in mind that switch only has 2 10G ports.

I saw that and was thinking it is fine, until I realized I will need at least 4 ports if I follow my previous post logic. I will need to dig again. Thanks.


Edit: Here, this one should do(NETGEAR XS505M):

https://www.amazon.com/NETGEAR-Ethernet-Unmanaged-Rackmount-Protection/dp/B075Q5C3Z4/ref=pd_ybh_a_17?_encoding=UTF8&refRID=TQP3NP5W5XVKT7DDWVS5&th=1

I'm fine with ~350 $

Edit 2:

https://www.amazon.com/Buffalo-Multi-Gigabit-Business-Switch-BS-MP2008/dp/B06XX4H997

Buffalo 12 Port 10gbe Gigabit Switch for sale online | eBay

Find many great new & used options and get the best deals for Buffalo 12 Port 10gbe Gigabit Switch at the best online prices at eBay! Free shipping for many products!

www.ebay.com

They are hard to find. I see what you guys are saying. An all-out 10 Gbps will kill the wallet. I will come back after I read about how to implement SFP+ . I have to compare the two methods.

Thanks for everybody who contributed in this thread.

 

[iamgenius]

So I went and read about sfp+ and DACs plus some bits from here and there and I'm now more confused than ever. Please help me clear this confusion.

So, let's keep this to the minimum. I just want to connect one PC to my Main NAS device with a 10 Gbps link just like aigomorla suggested. That will require me to get a sfp+ card for my NAS and my PC. Something like this:

https://www.amazon.com/10Gtek-E10G41BTDAG1P5-Ethernet-Converged-X520-DA1/dp/B01LZRSQM9

And for my NAS this:

Download Center - download | Synology Inc.

Centralize data storage and backup, streamline file collaboration, optimize video management, and secure network deployment to facilitate data management.

www.synology.com

And now I take it that I can connect between the two by using a DAC cable like this one:

https://www.amazon.com/10G-SFP-DAC-Cable-SFP-H10GB-CU2M/dp/B00U8BL09Q

OR

You can use fiber optic cables like this?

https://www.amazon.com/Fiber-Patch-Cable-Single-Duplex/dp/B00NFZ86NU/ref=sr_1_6?dchild=1&keywords=sfp%2B+with+pre-terminated+fibre+cables&qid=1612880880&sr=8-6

I'm missing something here: When are you supposed to use copper or fiber? I think this has to do with the presence of the transceiver ?

This should establish a 10 Gbps link between the two devices? What are the setup requirements software wise? And the old connection to the 1Gbps switch should stay intact?

Also, my NAS has one free PCIe 3.0 slot, so if I use that for the sfp+ card, where else can install SSD drives for SSD caching? I think I can use the USB 3.0 port but that is only 5 Gbps, so it won't saturate the link.

 

[iamgenius]

I guess this answers my last question(Thanks Synology):

E10M20-T1 | Synology Inc.

M.2 SSD & 10GbE combo adapter card for performance acceleration

www.synology.com

 

[iamgenius]

Please don't leave me alone on this! I'm getting exited and don't want to lose this excitement !

 

[iamgenius]

Foolish me. How can I forget about youtube

I will come back after I fully understand this video:

Youtube is your friend.

 

[JackMDS]

Hmm... A Video from more than TwoYears ago is a Good solution today???

 

[iamgenius]

Hmm... A Video from more than TwoYears ago is a Good solution today???

Why not? It should work. Isn't it? It is the first video that came up and I'm viewing others. I need to understand this sfp+ / DAC/ fiber stuff fully.

 

[mxnerd]

Sometimes I really don't understand JackMDS's logic.

==

Be aware that E10M20-T1 is RJ45 copper (10/5/2.5/1Gbps and 100Mbps ).

https://global.download.synology.com/download/Document/Hardware/DataSheet/Accessories/20-year/E10M20-T1/enu/Synology_E10M20_T1_Data_Sheet_enu.pdf

Compatibility

Compatibility List | Synology Inc.

Synology strives to improve product compatibility. Check out our compatibility list before purchasing the peripherals you need.

www.synology.com

 

[mxnerd]

Dual SFP+, of course you can use single port.

Compatibility List | Synology Inc.

Synology strives to improve product compatibility. Check out our compatibility list before purchasing the peripherals you need.

www.synology.com

Mellanox ConnectX-3 Pro Dual Port Ethernet Adapter Network Card - MCX312B-XCCT for sale online | eBay

Find many great new & used options and get the best deals for Mellanox ConnectX-3 Pro Dual Port Ethernet Adapter Network Card - MCX312B-XCCT at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[iamgenius]

Sometimes I really don't understand JackMDS's logic.

==

Be aware that E10M20-T1 is RJ45 copper (10/5/2.5/1Gbps and 100Mbps ).

https://global.download.synology.com/download/Document/Hardware/DataSheet/Accessories/20-year/E10M20-T1/enu/Synology_E10M20_T1_Data_Sheet_enu.pdf

Compatibility

Compatibility List | Synology Inc.

Synology strives to improve product compatibility. Check out our compatibility list before purchasing the peripherals you need.

www.synology.com

I missed that. Thanks to you. But I can still use it if I go with the switch solution.

 

[mxnerd]

Trendnet-Teg-40128 https://www.trendnet.com/products/10G-web-smart-switches/12-Port/TEG-40128WS
seems a very flexible managed 10GBe (copper + optics) switch

Trendnet Teg-40128 (8)10gbe Ethernet & (4) SFP+ web smart managed switch | eBay

Find many great new & used options and get the best deals for Trendnet Teg-40128 (8)10gbe Ethernet & (4) SFP+ web smart managed switch at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Good reviews at Amazon.ca

TRENDnet TEG-40128 12-Port 10G Web Smart Switch with 8 X 10Gbase-T Ports Including 2 X 10G SFP+ Slots : Amazon.ca: Tools & Home Improvement

TRENDnet TEG-40128 12-Port 10G Web Smart Switch with 8 X 10Gbase-T Ports Including 2 X 10G SFP+ Slots : Amazon.ca: Tools & Home Improvement

www.amazon.ca

Brand new price $800

https://www.amazon.com/TRENDnet-10GBASE-T-Switching-Protection-TEG-40128/dp/B01EOPTKBQ

 

[iamgenius]

Trendnet-Teg-40128 https://www.trendnet.com/products/10G-web-smart-switches/12-Port/TEG-40128WS
seems a very flexible managed 10GBe (copper + optics) switch

Trendnet Teg-40128 (8)10gbe Ethernet & (4) SFP+ web smart managed switch | eBay

Find many great new & used options and get the best deals for Trendnet Teg-40128 (8)10gbe Ethernet & (4) SFP+ web smart managed switch at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Good reviews at Amazon.ca

TRENDnet TEG-40128 12-Port 10G Web Smart Switch with 8 X 10Gbase-T Ports Including 2 X 10G SFP+ Slots : Amazon.ca: Tools & Home Improvement

TRENDnet TEG-40128 12-Port 10G Web Smart Switch with 8 X 10Gbase-T Ports Including 2 X 10G SFP+ Slots : Amazon.ca: Tools & Home Improvement

www.amazon.ca

Brand new price $800

https://www.amazon.com/TRENDnet-10GBASE-T-Switching-Protection-TEG-40128/dp/B01EOPTKBQ

Nice option. Thanks for searching for me. The switches I indicated earlier should work as well. I still need to find out how to add 10 Gbps capability to my NAS while also utilizing SSD's for caching. I don't want to use the large bays of course. It seems like the E10M20-T1 is the only option but I need to skip on optical. I'm also worried that it won't make much difference:

Synology E10M20-T1 M.2 SSD & 10GbE Card Review

Synology came up with an easy way to add 10GbE connectivity & NVMe cache to some of their NAS devices through the Synology E10M20-T1 AIC.

www.storagereview.com

 

[mxnerd]

Managed switch gives you the advantage to do link aggregation so if you use dual port sfp+ adapter then you can upgrade the speed to 20GBe.

with 6 SSDs in RAID 0, the max your NAS can achieve is 3,600MB/s , which is over 20GBe. with 6 HDDs in RAID, it probably will be hovering around 10GBe.

It really depends on what you need and how you are going to setup your NAS and network. It's your choice.

 

[iamgenius]

Managed switch gives you the advantage to do link aggregation so if you use dual port sfp+ adapter then you can upgrade the speed to 20GBe.

with 6 SSDs in RAID 0, the max your NAS can achieve is 3,600MB/s , which is over 20GBe. with 6 HDDs in RAID, it probably will be hovering around 10GBe.

It really depends on what you need and how you are going to setup your NAS and network. It's your choice.

Do you know of a way to measure NAS performance (transfer rate)? Something like HD tune? I couldn't find a tool that does it, not in the package manager for my synology either.