I need help with server setup for networking

[Missing Ghost]

Hi to all the people that are nice enough to try to help me.
My question is quite complex, I will try to make it as simple as possible.
I have one server that runs Debian GNU/linux.
It is the center of a medium network, the Internet is plugged into it.
eth0 is the Internet
eth1 to eth 5 are LAN connections.
eth6 is an IP-based SAN.


eth1 to 5 are linked together using a bridge called bluelan
eth0 and bluelan have packets routed between them, with NAT.
eth6 has totally no routing to other interfaces, so all the packets that get received by it are destinated to the server.

The problem is that the computers that are plugged into different ports of bluelan can't see each other. Like, the computers that are plugged into eth3 can't access the computers into eth4 for example. In my understanding of a bridge (it's like a network switch), it should work.

For references, here is a few startup scripts that I run.

/etc/network/interfaces

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# iptables startup scripts
pre-up /etc/network/if-pre-up.d/iptables-start.sh
post-down /etc/network/if-post-down.d/iptables-stop.sh

# The Internet network interface
auto eth0
iface eth0 inet dhcp

# The bluelan bridge

auto bluelan
iface bluelan inet static
address 192.168.0.1
netmask 255.255.255.0
broadcast 192.168.0.255
network 192.168.0.0
bridge_ports eth2 eth3 eth1 eth4 eth5
bridge_stp off
bridge_fd 3
bridge_maxwait 5

# The redlan downlink

auto eth6
iface eth6 inet static
address 192.168.2.1
netmask 255.255.255.0
broadcast 192.168.2.255
network 192.168.2.0
mtu 1500

#EOF

/etc/network/if-pre-up.d/iptables-start.sh

#!/bin/sh
# "iptables-start.sh"
#
########----------------------------------#############
# rules reset
iptables -F
iptables -t nat -F
iptables -t mangle -F
#
# policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
#
#
#########-----------INPUT RULES------------#############
#drop all inputs on the lan from the hosts that are also on the san
iptables -A INPUT -i bluelan -p tcp -m multiport --dport 137,138,139 -s 192.168.0.20 -j DROP
iptables -A INPUT -i bluelan -p udp -m multiport --dport 137,138,139 -s 192.168.0.20 -j DROP
iptables -A INPUT -i bluelan -p tcp -m multiport --dport 137,138,139 -s 192.168.0.30 -j DROP
iptables -A INPUT -i bluelan -p udp -m multiport --dport 137,138,139 -s 192.168.0.30 -j DROP
#accept all other inputs from lan
iptables -A INPUT -i bluelan -j ACCEPT
#---------------------------------------------------------------------------
#accept samba (netbios) related inputs from the san
iptables -A INPUT -i eth6 -p tcp -m multiport --dport 137,138,139 -j ACCEPT
iptables -A INPUT -i eth6 -p udp -m multiport --dport 137,138,139 -j ACCEPT
#---------------------------------------------------------------------------
#accept all inputs from loopback
iptables -A INPUT -i lo -j ACCEPT
#---------------------------------------------------------------------------
#accept all related and established states as input from the Internet
iptables -A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#ICMP stuff
iptables -A INPUT -p icmp -m state --state RELATED -j ACCEPT
#accept all inputs to servers that must be reachable from the Internet
#SSH
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
#SVN
iptables -A INPUT -p tcp --dport 3690 -j ACCEPT
#log all invalid states from the Internet
iptables -A INPUT -i eth0 -m state --state INVALID -j LOG
#
########-----------PREROUTING RULES--------###########
#port forwards
iptables -t nat -A PREROUTING -p tcp --dport 81 -j DNAT --to-destination 192.168.0.2:81
iptables -t nat -A PREROUTING -p tcp --dport 3389 -j DNAT --to-destination 192.168.0.20:3389
iptables -t nat -A PREROUTING -p tcp --dport 3955 -j DNAT --to-destination 192.168.0.20:3955
iptables -t nat -A PREROUTING -p udp --dport 3955 -j DNAT --to-destination 192.168.0.20:3955
#########----------FORWARD RULES-----------############
#accept all the traffic from the lan to the internet
iptables -A FORWARD -i bluelan -o eth0 -j ACCEPT

#accept all related and established states as forward from the Internet
iptables -A FORWARD -i eth0 -o bluelan -m state --state RELATED,ESTABLISHED -j ACCEPT
#accept all web traffic to 192.168.0.2 server
iptables -A FORWARD -p tcp --dport 81 -j ACCEPT
#accept all rdc traffic to 192.168.0.20
iptables -A FORWARD -p tcp --dport 3389 -j ACCEPT
iptables -A FORWARD -p udp --dport 3389 -j ACCEPT
#accept traffic for bittorrent to 192.168.0.20
iptables -A FORWARD -p tcp --dport 3955 -j ACCEPT
iptables -A FORWARD -p tcp --dport 3955 -j ACCEPT
#log all invalid states from the Internet
iptables -A FORWARD -i eth0 -m state --state INVALID -j LOG
#########----------POSTROUTING RULES-------############
#nat for the lan on the Internet
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

#EOF

That was quite a long post, I wish somebody will read it. Previously the server was all openBSD based. Bridges and firewalls on this OS are a lot more simple and I got them to work within one day. I need linux because openBSD no longer fits the job because it is not compatible with some new pieces of hardware I needed and some software. Samba was full of memory leaks, there is no LVM, the only file system is FFS, and for a lot of other reasons I really need to switch to linux. On openBSD everything could go over the bridge with no problems. Thanks for your help

 

[cleverhandle]

Before I think too much, is there some reason why you're bridging eth0-5 rather than plugging into a switch? It just seems really easy to introduce problems this way.

 

[Missing Ghost]

eth1 to eth5 are all connected to switches already, the server is the center. It's wired like that for performance purposes.

 

[cleverhandle]

Originally posted by: Missing Ghost
eth1 to eth5 are all connected to switches already, the server is the center. It's wired like that for performance purposes.

Hmm... can't say I really get that. You're not getting any more bandwidth out to the SAN than a single wire, so it seems much more direct to get one more switch and have a single interface on the server.

And unfortunately, nothing jumps out at me past that. It's got to be a NAT/iptables problem somewhere, but my experience is mostly with OpenBSD for this. As you're seeing, it's a lot easier to write complex rulesets in PF. Hopefully, some good iptables people will come by in a bit. Sorry...

 

[Brazen]

bridging is notoriously resource intensive. Way worse than routing, IIRC. You would probably get better performance ditching the bridge and using a seperate switch. If you are just wanting the speed of having 5 aggregate ethernet connections, then get you a 12 port hub, plug 5 of those ports into your linux box and set up nic bonding (also called nic teaming).

Alternatively, if you ditched the bridge and routed between those interfaces, you would get much better performance. And the performance would be two-fold: your linux box would use less cpu power and you cut down on broadcast traffic. You have an internace DNS server right?

 

[Missing Ghost]

I would like it if somebody could try to guess what the problem is instead of discussing my choices. I don't care if bridging is ressource intensive, I only want it to Just Work. Is there something in my setup that for real should not work? I tried everything I could.

 

[cleverhandle]

I think that the iptables mailing list is likely to get you better advice than you're going to find here.

 

[Brazen]

Your setup is so complicated that it is going to be a nightmare to troubleshoot, and needlessly so. You are going to have a hard time finding a knowledgeable network admin who is familiar with such a setup, because a knowledgeable admin would not set it up like that. I know these responses don't get your bridge working, but that's better than no help at all. Take it or leave it.

 

[futuristicmonkey]

That's not how to bridge.

This is:

ifconfig ethx 0.0.0.0 (repeat for each interface)
ifconfig ethx down
brctl addbr br0
brctl br0 stp off (i think this is the command, do some research)
brctl addif br0 ethx (repeat this for each interface)
ifconfig br0 up 192.168.0.1 (or whatever you want the ip address of your entire bridge to be


You might wanna research this stuff. I also forgot the name of the package for brctl. This setup works for me, but I must stress, again, do some research

 

[cleverhandle]

Originally posted by: futuristicmonkey
You might wanna research this stuff. I also forgot the name of the package for brctl. This setup works for me, but I must stress, again, do some research

Debian's networking system is doing all the work with brctl for him, based on the information he provided in /etc/network/interfaces. He shouldn't have any need to use brctl directly. The problem is somewhere in the iptables configuration.

 

[Brazen]

Originally posted by: cleverhandle

Originally posted by: futuristicmonkey
You might wanna research this stuff. I also forgot the name of the package for brctl. This setup works for me, but I must stress, again, do some research

Debian's networking system is doing all the work with brctl for him, based on the information he provided in /etc/network/interfaces. He shouldn't have any need to use brctl directly. The problem is somewhere in the iptables configuration.

Well, at first glance, his firewall config looks fine to me, but I'm no expert. The OP might want to install Webmin, which will let him have a nicer look at his firewall rules, and maybe even wipe out all his rules and start over, using webmin to remove and add the new rules in an organized fashion.

The OP, could also try flushing his rules ("iptables -F") and then see if the bridged interfaces will communicate. If it works, then that would pretty much confirm that it is indeed some firewall rule.

 

[Missing Ghost]

yes so :
iptables -A FORWARD -i bluelan -o bluelan -j ACCEPT
fixed the problem. It also shows me how linux networking is lame, all the bridge data goes through the whole networking stack instead of like it does on *BSD.

But there is still a problem while using the lan: accessing local services using the Internet IP address.

computer somewhere on the Internet enters my Internet IP address -> it works
computer somewhere on the LAN enters my Internet IP address -> it doesn't work
computer somewhere on the LAN enters my LAN IP address -> it works

example: http://192.168.0.2:81/somepage works, the Internet equivalent won't work except if I access it from another building.
That means if I go on a web page with a picture hosted internally the pictures won't display.

 

[cleverhandle]

Originally posted by: Missing Ghost
But there is still a problem while using the lan: accessing local services using the Internet IP address.

Unless I've missed something added recently, this won't work with either iptables or PF. There's no simple way to do this at the packet filtering level, because even if you rdr the packets to the correct address, the client machines will then be receiving packets with a source address that they don't expect.

This is better solved with an internal or split-view DNS.

 

[Missing Ghost]

 

[spidey07]

Originally posted by: Missing Ghost
eth1 to eth5 are all connected to switches already, the server is the center. It's wired like that for performance purposes.

that's bad. that's really, really bad. don't do that.

Unless you can offer a good reason (and there are none) to bridge those interfaces, don't do that. You are creating a network loop.

You're making this much harder than it needs to be.

 

[Missing Ghost]

Please tell me where the loop is??? A bridge between interfaces is a PC that behaves like a switch. So this is the same setup as if you plugged 5 switches into a 5 port switch. There is nothing wrong with that. And I get all the advantages that I can control everything, add prioritization, etc.

 

[spidey07]

I thought you had them attached to the same broadcast domain (a LAN). That would be the loop. but if they are on different broadcast domains you're fine.