networks issue: can only ping with one NIC on private network

[Gooberlx2]

I have a host server with 4 NICs. Two (eth0 and eth1) are bonded to an external network, and working fine. The other two are each static on a private, unmanaged switch for iSCSI traffic.

Basically, I'm trying to get multipathing properly setup with an equallogic iSCSI array. However, it seems only one NIC is truly active, though both say they're up.

For example:
1.after a network restart, both NICs (eth2 and eth3) are up.
2. Only eth2 will ping out to the iSCSI group, eth3 will fail.
3. ifdown eth2, eth3 will begin to work.
4. ifup eth2, eth3 will ping, eth2 will fail
5. ifdown eth3, eth2 will ping
6. rinse, repeat

I'm not terribly knowedgable about networks. Server is running SLES11, I set things up in Yast.

Any ideas on this? Here are some various config outputs:

ifconfig:

bond0 Link encap:Ethernet HWaddr 00:22:19:5D:0F:46
inet addr:XXX.XXX.252.59 Bcast:XXX.XXX.252.255 Mask:255.255.255.0
inet6 addr: fe80::222:19ff:fe5d:f46/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:38807 errors:0 dropped:0 overruns:0 frame:0
TX packets:50571 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3156959 (3.0 Mb) TX bytes:64312025 (61.3 Mb)

eth0 Link encap:Ethernet HWaddr 00:22:19:5D:0F:46
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:19289 errors:0 dropped:0 overruns:0 frame:0
TX packets:25190 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1571305 (1.4 Mb) TX bytes:32007732 (30.5 Mb)
Interrupt:36 Memory:d6000000-d6012700

eth1 Link encap:Ethernet HWaddr 00:22:19:5D:0F:46
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:19518 errors:0 dropped:0 overruns:0 frame:0
TX packets:25381 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1585654 (1.5 Mb) TX bytes:32304293 (30.8 Mb)
Interrupt:48 Memory:d8000000-d8012700

eth2 Link encap:Ethernet HWaddr 00:22:19:5D:0F:4A
inet addr:192.168.130.10 Bcast:192.168.130.255 Mask:255.255.255.0
inet6 addr: fe80::222:19ff:fe5d:f4a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:498 (498.0 b) TX bytes:976 (976.0 b)
Interrupt:32 Memory:da000000-da012700

eth3 Link encap:Ethernet HWaddr 00:22:19:5D:0F:4C
inet addr:192.168.130.20 Bcast:192.168.130.255 Mask:255.255.255.0
inet6 addr: fe80::222:19ff:fe5d:f4c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:10 errors:0 dropped:0 overruns:0 frame:0
TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:868 (868.0 b) TX bytes:1232 (1.2 Kb)
Interrupt:42 Memory:dc000000-dc012700

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:29560 errors:0 dropped:0 overruns:0 frame:0
TX packets:29560 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:56996219 (54.3 Mb) TX bytes:56996219 (54.3 Mb)

cat /etc/sysconfig/network/ifcfg-eth2:

BOOTPROTO='static'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR='192.168.130.10/24'
MTU=''
NAME='NetXtreme II BCM5709 Gigabit Ethernet'
NETMASK=''
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'
PREFIXLEN='24'

cat /etc/sysconfig/network/ifcfg-eth3:

BOOTPROTO='static'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR='192.168.130.20/24'
MTU=''
NAME='NetXtreme II BCM5709 Gigabit Ethernet'
NETMASK=''
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'

route -n:

Kernel IP routing table
Destination........Gateway................Genmask..........Flags..Metric..Ref..Use...Iface
192.168.130.0....0.0.0.0..................255.255.255.0...U.......0.........0.....0.......eth2
192.168.130.0....0.0.0.0..................255.255.255.0...U.......0.........0.....0.......eth3
XXX.XXX.252.0..0.0.0.0..................255.255.255.0...U.......0.........0.....0.......bond0
169.254.0.0.......0.0.0.0..................255.255.0.0.......U.......0.........0.....0.......eth2
127.0.0.0...........0.0.0.0..................255.0.0.0..........U.......0.........0.....0.......lo
0.0.0.0..............XXX.XXX.252.252...0.0.0.0..............UG.....0.........0.....0.......bond0

 

[her209]

Methinks its because you have eth2 and eth3 in the same network.

 

[Gooberlx2]

Originally posted by: her209
Methinks its because you have eth2 and eth3 in the same network.

I can't do that? I thought it was okay if they had different IPs.

So, if I set them up on different subnet, like:
eth2 = 192.168.100.1
eth3 = 192.168.200.1

how do I get them to talk to the iSCSI group at 192.168.130.100

...or am I totally missing it? (probably)

 

[xSauronx]

Originally posted by: her209
Methinks its because you have eth2 and eth3 in the same network.

he should have no problem using both adapters in the same subnet if they have different IPs, afaik.

maybe it has something to do with startmode=auto and usercontrol=no

id check the suse documentation on network interface configuration, it should be a pretty straightforward thing to figure out, but ive never played with configuration in suse or anything else using the /etc/sysconfig/network/ifcfg scripts so im not sure what does what in there, ive just had simple stuff to deal with via /etc/network/interfaces

 

[Gooberlx2]

Originally posted by: xSauronx

Originally posted by: her209
Methinks its because you have eth2 and eth3 in the same network.

he should have no problem using both adapters in the same subnet if they have different IPs, afaik.

maybe it has something to do with startmode=auto and usercontrol=no

id check the suse documentation on network interface configuration, it should be a pretty straightforward thing to figure out, but ive never played with configuration in suse or anything else using the /etc/sysconfig/network/ifcfg scripts so im not sure what does what in there, ive just had simple stuff to deal with via /etc/network/interfaces

Apparently startmode is specifying how the device activates. Automatically as soon as it's available, and there are other options like, hotplug and off.

usercontrol, I think, allows a regular user to bring the device up and down.

This is frustrating the hell outta me. I did do an arp -v once and the same hwaddress was given to both eth2 and eth3 on some connections to 192.168.130.100...I dunno if that's anything special though. The hwaddress listed didn't coincide with either's mac address.

 

[Gooberlx2]

I found this:
http://serverfault.com/questio...ways-go-out-a-single-n (tried both first and third solutions listed, stuck with 3rd 'cause of guy's explanation)

Which seems to allow both NICs to ping and connect to the array properly, though the NICs can't ping each other. Maybe that's by design or there are additional things to worry about. Atleast it seems to be a step in the right direction.

 

[Gooberlx2]

Huh...well...everything WAS working fine, then I noticed the firewall wasn't turned on. I turned it on temporarily, saw that eth3 could no longer ping the array and set it back. Now eth3 can ping the array, but it's not actually being used during transfers. Seems like turning on and off the firewall fucked up my multipathing....god damnit.

Regardless, not having a firewall is definitely NOT ideal. The solution above is likely a shitty workaround, but I just don't know enough about how this stuff works, routing tables, iptables, arptables, etc...

If only I had a real router for all this to be behind...but I don't.

 

[Crusty]

Can you put some more detail about the physical layer in your network setup?
How is your iSCSI array connected to the network?

 

[Gooberlx2]

Originally posted by: Crusty
Can you put some more detail about the physical layer in your network setup?
How is your iSCSI array connected to the network?

Maybe a diagram would help?

iSCSIDiagram.jpg

 

[Crusty]

Alright, that's not going to work unless you put static routes for each NIC to the target iSCSI destination that NIC will be talking to(I wouldn't recommend doing this).

If you're trying to increase redundancy you need to use different switches for each iSCSI network segment(and they should be different IMO) and use the built in multipathing with iSCSI. If you're trying to increase throughput then you can use LACP similar to how your other NIC's are bonded, which in turn can be used with multipathing to provide a faster redundant network.

For example, your iSCSI target can be presented on both
192.168.130.3
192.168.131.3

Now just make sure each NIC in your client server connected to that network has an address in that network and you'll have no problems pinging that IP. Then it's just about setting up iSCSI in an Active/Active or Active/Standby depending on needs.

If you're still having problems with pings you'll need a packet tracer to really figure out what's going on so you can see traffic in/outbound at both endpoints.

 

[Gooberlx2]

Originally posted by: Crusty
Alright, that's not going to work unless you put static routes for each NIC to the target iSCSI destination that NIC will be talking to(I wouldn't recommend doing this).

If you're trying to increase redundancy you need to use different switches for each iSCSI network segment(and they should be different IMO) and use the built in multipathing with iSCSI. If you're trying to increase throughput then you can use LACP similar to how your other NIC's are bonded, which in turn can be used with multipathing to provide a faster redundant network.

For example, your iSCSI target can be presented on both
192.168.130.3
192.168.131.3

Now just make sure each NIC in your client server connected to that network has an address in that network and you'll have no problems pinging that IP. Then it's just about setting up iSCSI in an Active/Active or Active/Standby depending on needs.

If you're still having problems with pings you'll need a packet tracer to really figure out what's going on so you can see traffic in/outbound at both endpoints.

Well, the server only talks to the array on the group IP. It doesn't actually talk to the iSCSI NICs...I only put their IPs down for completeness-sake. I may actually have to do those static routes. I'm not sure how though. Any simple tutorials for network retards like me?

 

[Crusty]

Well now I'm just confused. How are your iSCSI targets setup so that they can take advantage of the 3 NICs in the iSCSI box?

 

[Gooberlx2]

Originally posted by: Crusty
Well now I'm just confused. How are your iSCSI targets setup so that they can take advantage of the 3 NICs in the iSCSI box?

Yeah, it's how equallogic does it. 192.168.130.100 is the IP address the initiator is pointed at. The array then assigns each connection from the server to its own NICs.

Each NIC on the server should have a connection for each NIC and volume on the array. So if I have two volumes on the array, its management page should show me 4 total connections from the server. In the server's initiator, each volume shows up twice. That's the multipathing.

One of the NICs on the array is wasted, currently.

 

[Crusty]

Well if you can't reconfigure the array then you either need to specify the network interface used when you configure the initiator or put in some static routes.

You can do something like

route add 192.168.130.100 netmask 255.255.255.0 dev eth2

I believe. That should send traffic destined for 192.168.130.100 out eth2, just swap numbers and interfaces for the other one.

 

[Gooberlx2]

Originally posted by: Crusty
Well if you can't reconfigure the array then you either need to specify the network interface used when you configure the initiator or put in some static routes.

You can do something like

route add 192.168.130.100 netmask 255.255.255.0 dev eth2

I believe. That should send traffic destined for 192.168.130.100 out eth2, just swap numbers and interfaces for the other one.

Awesome, thanks. I will try that as soon as a few large transfers are finished. It works right now...it's just not ideal.

 

[Gooberlx2]

Sorry for the thread necro. This is a looong time follow-up, but I was helping someone else on this issue this week, and thought someone here might have use of the solution.

I originally mucked around with the arp and rp_filter settings as suggested in that earlier servfault.com link. While that allows pings from the eth devices to the group ip, traffic still routed through the first device in the subnet (eth2), and the devices couldn't ping each other.

I found this document and this post (more recently when helping my debian using colleague), and used them as examples (used 0.0.0.0 for the gateway and 192.168.130.0 for the network).

The interfaces can now ping both each other and the iSCSI group IP. More importantly, multipathing now works where I have increased throughput and failover with the iSCSI array, since traffic is routing through the correct interfaces.