Ubuntu Desktop 20.04.3 LTS: strange network behaviour

[glc650]

Hi,

I have a fresh (few days old) install of Ubuntu Desktop 20.04.3 on an Intel 11 NUC (NUC11PAHi7). The WiFi interface is disabled in the BIOS so the only physical network interface on this machine is the NUC's built-in 1gbe NIC (enp88s0). This NIC is on two different subnets (VLANs). Here is my netplan config:

# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: networkd
  ethernets:
    enp88s0:
      mtu: 9000
      dhcp4: false
  vlans:
    enp88s0.100:
      mtu: 9000
      dhcp4: false
      addresses: [10.0.0.3/24]
      gateway4: 10.0.0.254
      nameservers:
        addresses: [1.1.1.1,1.0.0.1]
      id: 100
      link: enp88s0
    enp88s0.101:
      mtu: 9000
      dhcp4: false
      addresses: [10.1.0.3/24]
      nameservers:
        addresses: [1.1.1.1,1.0.0.1]
      id: 101
      link: enp88s0

ip r output:
default via 10.0.0.254 dev enp88s0.100 proto static
10.0.0.0/24 dev enp88s0.100 proto kernel scope link src 10.0.0.3
10.1.0.0/24 dev enp88s0.101 proto kernel scope link src 10.1.0.3

For some reason I am unable to SSH/VNC into this machine from the 10.1.0.0/24 network to the 10.0.0.3 interface (specifically from my MacBook Pro's WiFi at 10.1.0.32). It just times out (the firewall sitting in-between these subnets has this traffic permitted & ufw is disabled on Ubuntu). SSH/VNC work just fine when on the same subnet (so 10.1.0.32 to 10.1.0.3 or 10.0.0.0/24 to 10.0.0.3).

Thanks,

->g.

 

[Tech Junky]

You don't have a gateway set for enp88s0.101

 

[glc650]

You don't have a gateway set for enp88s0.101

Netplan apply compains if I have two default gateways set so it is only set on .100 as that is my preferred (for Internet access) subnet anyway. And even with 2 GWs set the problem is still there.

 

[Tech Junky]

I use the OG /etc/network/interfaces option because NP is kind of a PITA to deal with.


auto lo:50
iface lo:50 inet static
address 192.168.0.50

auto br0
iface br0 inet manual
iface br0 inet static
address 192.168.0.1

auto br0
iface br0 inet manual
iface br0 inet static
address 192.168.0.2

While I'm not running VL's using the OG method can allow for more options such as assigning virtual interfaces / multiple static options. So, Lo50 is my management IP, bro0 is the gateway and the secondary is what I'm using for Pihole.

https://techviewleo.com/how-to-configure-vlan-interface-on-debian/ This sort of option should work and if you add the first one then just use nano on the /etc/iproute2/rt_tables

 

[Fallen Kell]

Also, you might want to setup "policy based routing" sometimes called "source based routing". Whenever you have multiple NICs that can eventually route out and talk to the same networks, you really need to do this. You are implying that this can happen on your network as you stated you are using a system on 10.1.0.0/24 net attempting to connect to your host on the 10.0.0.0/24 net interface. Policy/source based routing basically tells the system to respond via the same NIC that received the packet even if another NIC has a shorter pathway to the destination.

So one thing that might be happening is that your system attempts to create a TCP connection from the 10.1.0.0/24 network over to the 10.0.0.3 ip address, it sends the initial packets to create the handshake to 10.0.0.3, but the response comes back from 10.1.0.3, not 10.0.0.3 because it is shorter for the system to respond directly to the 10.1.0.0/24 network interface, but the initializing system on the 10.0.0.0/24 network sees this response as a un-initialized TCP connection attempt from an unknown host as it is expecting the response to be from 10.0.0.3, not 10.1.0.3.

 

[glc650]

Also, you might want to setup "policy based routing" sometimes called "source based routing". Whenever you have multiple NICs that can eventually route out and talk to the same networks, you really need to do this. You are implying that this can happen on your network as you stated you are using a system on 10.1.0.0/24 net attempting to connect to your host on the 10.0.0.0/24 net interface. Policy/source based routing basically tells the system to respond via the same NIC that received the packet even if another NIC has a shorter pathway to the destination.

So one thing that might be happening is that your system attempts to create a TCP connection from the 10.1.0.0/24 network over to the 10.0.0.3 ip address, it sends the initial packets to create the handshake to 10.0.0.3, but the response comes back from 10.1.0.3, not 10.0.0.3 because it is shorter for the system to respond directly to the 10.1.0.0/24 network interface, but the initializing system on the 10.0.0.0/24 network sees this response as a un-initialized TCP connection attempt from an unknown host as it is expecting the response to be from 10.0.0.3, not 10.1.0.3.

Trying to wrap my head around a config for source routing based on examples I've seen online (netplan site & elsewhere) but nothing is working:

# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: networkd
  ethernets:
    enp88s0:
      mtu: 9000
      dhcp4: false
  vlans:
    enp88s0.100:
      mtu: 9000
      dhcp4: false
      addresses: [10.0.0.3/24]
      gateway4: 10.0.0.254
      nameservers:
        addresses: [1.1.1.1,1.0.0.1]
      id: 100
      link: enp88s0
      routes:
       - to: 10.1.0.0/24
         via: 10.0.0.254
         table: 100
      routing-policy:
       - from: 10.1.0.0/24
         table: 100
    enp88s0.101:
      mtu: 9000
      dhcp4: false
      addresses: [10.1.0.3/24]
      #gateway4: 10.1.0.254
      nameservers:
        addresses: [1.1.1.1,1.0.0.1]
      id: 101
      link: enp88s0

All I want is for my Macbook when it is on WiFi only (10.1.0.32, vlan 101) to be able to SSH & VNC to ubuntu on the vl100 interface (enp88s0.100, 10.0.0.3).

 

[Fallen Kell]

So I am linking the underlying documentation on this below:

https://lartc.org/howto/lartc.rpdb.multiple-links.html

Please note that the above is primarily talking about a system that is acting as a router from the local network to two ISPs, but the underlying issue is the same when it comes to responding back to traffic that was received from one ISP or the other, ensuring the response goes back out the same NIC it came in. And while your system isn't a "router" per say, it does need to know how to respond to the various packets it receives over the appropriate NIC just as in the example in the above documentation. So you don't need to worry about the parts dealing with load balancing, as that is not your issue (and in fact you probably do not want that, but want the wired system to primarily communicate over the 10.0.0.0/24 net as it seems the 10.1.0.0/24 is your wireless net and you don't want to send traffic there unless it needs to be there as it would possibly eat into your available/usable wireless bandwidth).

From what I understand of your network setup, I believe you need to do the following in your /etc/iproute2/rt_tables:
P1_NET = 10.0.0.0/24
P2_NET = 10.1.0.0/24
IF1 = enp88s0.100
IF2 = enp88s0.101
IP1 = 10.0.0.3
IP2 = 10.1.0.3
P1 = 10.0.0.254
P2 = 10.1.0.254

ip route add $P1_NET dev $IF1 src $IP table T1
ip route add default via $P1 table T1
ip route add $P2_NET dev $IF2 src $IP2 table T2
ip route add default via $P2 table T2

ip route add $P1_NET dev $IF1 src $IP1
ip route add $P2_NET dev $IF2 src $IP2

ip route add default via $P1

ip rule add from $IP1 table T1
ip rule add from $IP2 table T2


You might need to actually replace the variables with the values, I don't remember if you can set variables in that file (but it was easier to type it this way since the example used the variables).

Again, I could be wrong. I have not had to set this up in years. I mostly dealt with this situation by running Solaris at the time as it behaves like this out of the box (but Solaris is no longer a solution I would recommend on anyone ever since Oracle took ownership from Sun Microsystems and subsequently fired most of the development staff a few years later).

The real magic is telling your system that it needs to have two route tables, T1 and T2, and that all communications from NIC1 needs to use T1, and all communications from NIC2 needs to use T2. This keeps all the network connections responding on the same NIC that received the initial request.

 

[mv2devnull]

On RHEL 'rp_filter' has been on strict mode for a while. Does Ubuntu have it too? https://www.theurbanpenguin.com/rp_filter-and-lpic-3-linux-security/
When strict rp_filter did appear in RHEL, this "if I talk to you with your other address, you ignore me" did happen.
Keeping strict and adding policy routing is the way to go though.

Alas, RHEL uses different style for network config and I've never used Ubuntu, so can't help with syntax.

PS. If destination is "any address of this machine", then it is "input" in the firewall. "Forward filtering" occurs only for packets that will be routed to someone else. Hence firewall in the Ubuntu should not be an issue.