FreeBSD bridge?

[Soybomb]

FreeBSD 5.3 firewall with 3 interfaces running ipfilter and ipnat:

fxp0 - Interface with a static public internet IP address (123.456.789.1/24)
fxp1 - interface with static private address for LAN (10.10.10.1/24)
fxp2 - unused (planned 987.654.321.1/24)

I use ipnat to rewrite requests coming in on fxp0 from the internet to servers on the lan on fxp1. It works great.

Now I have a new lan in the building that also has an internet connection. I want traffic destined for the internet address of the firewall to go over the lan instead of the internet. Its a long story but I have to do this at the firewall and not at the router. Similarly all traffic from the original LAN on fxp1 should be delivered locally instead of through the building. To do this I'll plug the switch for the 2nd LAN into fxp2.

I've tried using map statements in ipnat.rules but that doesn't seem to work quite right. If I just use the bridge functionality of freebsd on fxp0 and fxp2 would that effectively allow me to send traffic from a device on the new lan to an address on fxp0 over fxp2? How would traffic from lan 2 being sent to fxp0's address not be forced through lan 2's gateway since they're on different networks?

Would the bridging functionality take place before the traffic goes through ipnat and ipfilter so all of my existing rules for each would work? Keep in mind lan 1 and lan 2 have different ISP's.

 

[n0cmonkey]

Will this new network be NATed? If so, are you going to use the internal IPs to access lan2 from lan1?

If the firewall is the default router for the machines on LAN1, the machines will send traffic destined to LAN2 to the firewall. The firewall will send that traffic out fxp2, since it is on the LAN2 network. I think.

 

[cleverhandle]

Interesting question...

Originally posted by: Soybomb

fxp0 - Interface with a static public internet IP address (123.456.789.1/24)
fxp1 - interface with static private address for LAN (10.10.10.1/24)
fxp2 - unused (planned 987.654.321.1/24)

Now I have a new lan in the building that also has an internet connection. I want traffic destined for the internet address of the firewall to go over the lan instead of the internet. Its a long story but I have to do this at the firewall and not at the router.

OK, I understand what you want. Not why you want it, but I know how long stories can be...

Similarly all traffic from the original LAN on fxp1 should be delivered locally instead of through the building.

I'm not sure I understand what you're getting at here. "Locally" and "through the building" sound like the same thing to me.

To do this I'll plug the switch for the 2nd LAN into fxp2.

I've tried using map statements in ipnat.rules but that doesn't seem to work quite right. If I just use the bridge functionality of freebsd on fxp0 and fxp2 would that effectively allow me to send traffic from a device on the new lan to an address on fxp0 over fxp2?

I don't think so, precisely because...

How would traffic from lan 2 being sent to fxp0's address not be forced through lan 2's gateway since they're on different networks?

Yup, that's the problem. Traffic to a different subnet is going to go through the gateway by definition. If you set up a bridge, it will learn the MAC of fxp0 (I think? fxp0 is external technically...). But the client isn't going to ARP for the firewall unless the IP address is on it's own subnet. It's just going to ARP for the gateway. So the bridge will never get involved because it will never hear an ARP request for fxp0's IP.

The situation's a little convoluted even before the NAT's get involved, so I wouldn't bet my life on that analysis. But it makes sense to me at the moment. I think you'll need to get the routers involved so long as you have different subnets. For starters, the gateway for LAN 2 would need a static route for LAN 1 pointed through fxp2's IP. And then you'll need the reciprocal arrangement for LAN 1's gateway. That should work, I think.

I'm guessing there's some good (or at least unavoidable) reason why we need separate Internet connections here? Everything would be a lot simpler with one router acting as a single gateway for both LAN's.

 

[Soybomb]

The new net will not be nat'ed.

Makes sense about traffic going from lan1 to lan2 since it goes through the firewall. How about traffic going from lan 2 to lan 1?

The BSD handbook on bridging: "The bridge works by learning the MAC layer addresses (Ethernet addresses) of the devices on each of its network interfaces. It forwards traffic between two networks only when its source and destination are on different networks."

How does that work? So lets say a client on lan 2 sends traffic to the ip address assigned to fxp0. Would it arp for a mac for that IP address which then could be passed on via the bridge or would it send the traffic to the gateway since the destination isn't on its network, in which case I'd have to do static routes on the LAN2 devices?

 

[Soybomb]

Originally posted by: cleverhandle
Interesting question...

Originally posted by: Soybomb

fxp0 - Interface with a static public internet IP address (123.456.789.1/24)
fxp1 - interface with static private address for LAN (10.10.10.1/24)
fxp2 - unused (planned 987.654.321.1/24)

Now I have a new lan in the building that also has an internet connection. I want traffic destined for the internet address of the firewall to go over the lan instead of the internet. Its a long story but I have to do this at the firewall and not at the router.

OK, I understand what you want. Not why you want it, but I know how long stories can be...

Similarly all traffic from the original LAN on fxp1 should be delivered locally instead of through the building.

I'm not sure I understand what you're getting at here. "Locally" and "through the building" sound like the same thing to me.

To do this I'll plug the switch for the 2nd LAN into fxp2.

I've tried using map statements in ipnat.rules but that doesn't seem to work quite right. If I just use the bridge functionality of freebsd on fxp0 and fxp2 would that effectively allow me to send traffic from a device on the new lan to an address on fxp0 over fxp2?

I don't think so, precisely because...

How would traffic from lan 2 being sent to fxp0's address not be forced through lan 2's gateway since they're on different networks?

Yup, that's the problem. Traffic to a different subnet is going to go through the gateway by definition. If you set up a bridge, it will learn the MAC of fxp0 (I think? fxp0 is external technically...). But the client isn't going to ARP for the firewall unless the IP address is on it's own subnet. It's just going to ARP for the gateway. So the bridge will never get involved because it will never hear an ARP request for fxp0's IP.

The situation's a little convoluted even before the NAT's get involved, so I wouldn't bet my life on that analysis. But it makes sense to me at the moment. I think you'll need to get the routers involved so long as you have different subnets. For starters, the gateway for LAN 2 would need a static route for LAN 1 pointed through fxp2's IP. And then you'll need the reciprocal arrangement for LAN 1's gateway. That should work, I think.

I'm guessing there's some good (or at least unavoidable) reason why we need separate Internet connections here? Everything would be a lot simpler with one router acting as a single gateway for both LAN's.

I can do a static route on the router for LAN 2 pointing all traffic from lan 2 destined to lan 1 to the ip address on fxp2. I was wanting to avoid that initially, but now it might be better to just work around those problems. Not a chance on anything for lan1's router however.

The reason why just 1 internet connection is they both use a lot of bandwidth and want to keep their individual usage seperate, but have the stability of a connection locally instead of relying on the internet for transit between their LANs.

 

[n0cmonkey]

Originally posted by: Soybomb
The new net will not be nat'ed.

Makes sense about traffic going from lan1 to lan2 since it goes through the firewall. How about traffic going from lan 2 to lan 1?

As long as they're using the real IP addresses of the servers, it'll do the same thing.

The BSD handbook on bridging: "The bridge works by learning the MAC layer addresses (Ethernet addresses) of the devices on each of its network interfaces. It forwards traffic between two networks only when its source and destination are on different networks."

How does that work? So lets say a client on lan 2 sends traffic to the ip address assigned to fxp0. Would it arp for a mac for that IP address which then could be passed on via the bridge or would it send the traffic to the gateway since the destination isn't on its network, in which case I'd have to do static routes on the LAN2 devices?

If FreeBSD's bridging is anything like OpenBSD's, it's weird. IIRC, you pair up a couple of network cards, don't give them IP addresses, and configure bridging. The bridge just passes through stuff destined for LAN2 to LAN2 from LAN1 (and back). Broadcast stuff does not make it through. I think it just uses a list of MAC addresses to determine which side of the bridge the packet should be on.

EDIT: Wait, I'm not sure about the LAN2 to LAN1. Does LAN2 have a different default router?

 

[cleverhandle]

Originally posted by: Soybomb
I can do a static route on the router for LAN 2 pointing all traffic from lan 2 destined to lan 1 to the ip address on fxp2. I was wanting to avoid that initially, but now it might be better to just work around those problems. Not a chance on anything for lan1's router however.

Hmm... but the return path shouldn't be a problem. The firewall has an interface on LAN 2, so its replies should go directly through that interface. So I think that should be cool. (This is what n0c said.)

Originally posted by: n0cmonkey

Originally posted by: Soybomb
How about traffic going from lan 2 to lan 1?

As long as they're using the real IP addresses of the servers, it'll do the same thing.

Are you sure about that part? The firewall isn't the gateway for LAN 2. As far as I can tell, it should never (without a static route at the LAN 2 router) forward any packets destined for LAN 1.

 

[n0cmonkey]

Originally posted by: cleverhandle

Originally posted by: Soybomb
I can do a static route on the router for LAN 2 pointing all traffic from lan 2 destined to lan 1 to the ip address on fxp2. I was wanting to avoid that initially, but now it might be better to just work around those problems. Not a chance on anything for lan1's router however.

Hmm... but the return path shouldn't be a problem. The firewall has an interface on LAN 2, so its replies should go directly through that interface. So I think that should be cool. (This is what n0c said.)

Originally posted by: n0cmonkey

Originally posted by: Soybomb
How about traffic going from lan 2 to lan 1?

As long as they're using the real IP addresses of the servers, it'll do the same thing.

Are you sure about that part? The firewall isn't the gateway for LAN 2. As far as I can tell, it should never (without a static route at the LAN 2 router) forward any packets destined for LAN 1.

Read my entire post.

I added that in an edit a couple of minutes after posting, since I wasn't sure whether the firewall was the defaultrouter for the hosts on LAN2 or not.

 

[Soybomb]

No currently the firewall is the gateway for machines on lan 1, the firewall's gateway is router 1. The gateway for machines on lan2 is router2.

 

[cleverhandle]

Originally posted by: n0cmonkey
Read my entire post.

Sure, just let me know when your entire post is posted. (I replied before the edit.)

Originally posted by: Soybomb
No currently the firewall is the gateway for machines on lan 1, the firewall's gateway is router 1. The gateway for machines on lan2 is router2.

Right. So just one static route set on the LAN 2 router should put you where you want to be. Since the fxp0 address is the only one you're concerned with, I'd keep the route restricted to 123.456.789.1/32, though I can't immediately see any problems with setting the route to the whole /24.

 

[Soybomb]

Originally posted by: cleverhandle

Originally posted by: n0cmonkey
Read my entire post.

Sure, just let me know when your entire post is posted. (I replied before the edit.)

Originally posted by: Soybomb
No currently the firewall is the gateway for machines on lan 1, the firewall's gateway is router 1. The gateway for machines on lan2 is router2.

Right. So just one static route set on the LAN 2 router should put you where you want to be. Since the fxp0 address is the only one you're concerned with, I'd keep the route restricted to 123.456.789.1/32, though I can't immediately see any problems with setting the route to the whole /24.

Assuming the bridge functionality will do this at least

 

[cleverhandle]

Originally posted by: Soybomb
Assuming the bridge functionality will do this at least

What-huh? You don't want a bridge at all - I thought we established that. Everything you need will be handled by standard routing.

 

[n0cmonkey]

Originally posted by: cleverhandle

Originally posted by: n0cmonkey
Read my entire post.

Sure, just let me know when your entire post is posted. (I replied before the edit.)

7:36 comes before 7:17?

 

[cleverhandle]

Originally posted by: n0cmonkey
7:36 comes before 7:17?

Maybe. Replying is a process after all...

Click reply at 7:16, ponder for a few minutes, play with the cat, ponder a few more minutes, catch a smoke, type the reply, play with the other cat, click submit.

Why, yes! I do have ADD - what made you ask?

 

[n0cmonkey]

Originally posted by: cleverhandle

Originally posted by: n0cmonkey
7:36 comes before 7:17?

Maybe. Replying is a process after all...

Click reply at 7:16, ponder for a few minutes, play with the cat, ponder a few more minutes, catch a smoke, type the reply, play with the other cat, click submit.

Why, yes! I do have ADD - what made you ask?

Quit smoking, it'll do wonders.

<-Quit in Jan. Still playing with cats though.

 

[Soybomb]

Originally posted by: cleverhandle

Originally posted by: Soybomb
Assuming the bridge functionality will do this at least

What-huh? You don't want a bridge at all - I thought we established that. Everything you need will be handled by standard routing.

Hmmm I must have fallen off the train somewhere. If I don't bridge I don't see how it would work. Traffic leaves LAN2 destined for LAN1. It goes to router2. The static would have have to be an lan2 address on fxp2 right. If so I'm back to my original problem of making all my nat crap work right on the net interface.

 

[n0cmonkey]

Originally posted by: Soybomb

Originally posted by: cleverhandle

Originally posted by: Soybomb
Assuming the bridge functionality will do this at least

What-huh? You don't want a bridge at all - I thought we established that. Everything you need will be handled by standard routing.

Hmmm I must have fallen off the train somewhere. If I don't bridge I don't see how it would work. Traffic leaves LAN2 destined for LAN1. It goes to router2. The static would have have to be an lan2 address on fxp2 right. If so I'm back to my original problem of making all my nat crap work right on the net interface.

fxp2 shouldn't need any kind of NAT.

The bridging would have a very similar problem, since the traffic isn't headed towards the bridge at all. You'd generally have a bridge or a router, IIRC.

 

[Soybomb]

I currently use the nat functionality to take traffic in on fxp0 and point it to a server on the fxp1 network (LAN1). If traffic is coming in on fxp2 from lan2, destined for an ip address on fxp0, how will it get rewritten as the proper lan1 private ip address as it would as if it came off the internet?

Hope I'm not being too thick tonight

 

[n0cmonkey]

Originally posted by: Soybomb
I currently use the nat functionality to take traffic in on fxp0 and point it to a server on the fxp1 network (LAN1). If traffic is coming in on fxp2 from lan2, destined for an ip address on fxp0, how will it get rewritten as the proper lan1 private ip address as it would as if it came off the internet?

Hope I'm not being too thick tonight

That's why you'd use the internal IP address when coming from LAN2.

 

[cleverhandle]

Originally posted by: Soybomb
Hmmm I must have fallen off the train somewhere.

No, I think I missed something in your description.

If I don't bridge I don't see how it would work. Traffic leaves LAN2 destined for LAN1. It goes to router2. The static would have have to be an lan2 address on fxp2 right. If so I'm back to my original problem of making all my nat crap work right on the net interface.

From your earlier statements, I was under the impression that hosts in LAN 2 were using fxp0 as a destination - i.e. that they were only speaking to the firewall machine, not to the NAT'ed machines behind it. But this appears not to be the case - i.e. hosts in LAN 2 need to communicate with the NAT'ed servers in LAN 1. Correct?

If so, life is a bit harder. I don't think a bridge will work, but I'm not sure. You'd be asking the same interface (fxp2) to both pass packets like a bridge on the forward path from LAN 2 and route packets like a router on the return. Maybe... but really weird. Sounds very prone to doing things that are not what you intend to do. And I think the external interface will be even worse.

I think it would be easier to simply forward the same ports on fxp2 that you already are on fxp0. Then a client on LAN 2 sends to a destination on LAN 1, that packet hits Router 2 which has a static route sending it to fxp2, which forwards the port to the appropriate host in the NAT net out through fxp1. The source address has never been rewritten so far, since the forward path never goes "out" on fxp0. That's fine, because the LAN 1 server sends the reply through the gateway which is directly connected to LAN 2 (where the source address is located).

I'm guessing that the root of the problem is DNS - your servers have DNS records pointing to the fxp0 IP, which is then forwarding ports appropriately. A cleaner solution would be to use two different views (internal and external) on the DNS server. Then the LAN 2 clients could use the proper private IP's for LAN 1 servers automatically. But you probably don't have that level of control over your DNS.

edit: Your recent posts confirm what I was guessing about NAT above. Sorry it takes me so long to type.

edit 2: I'm becoming less sure of that port forwarding on fxp2 stuff. The client will think it's talking to the IP of fxp0, but the reply will never get rewritten that way. Sounds messed up. This is quickly sounding like a setup I wouldn't want to administer with a ten-foot pole. I think that split DNS or a more sane subnetting/gateway setup are really the only solutions here.

 

[Soybomb]

noc: internal IP's are no good though because then I have dns problems, I really want to use the externals

clever you're right, I want the hosts on lan2 to go through fxp0 just like data coming in from the internet and go through the nat layer to hosts on lan1. Right now what I'm using is actually like you describe with forward the same ports on fxp2 that I do on fxp0. The part that I left out however is fxp0 actually has multiple IPs assigned to so then I run into problems with multiple port conflicts. Also heres a sample rule to redirect say smtp from an ip address on the outside to an ip on lan1:
rdr 123.456.789.1/32 port 25 -> 10.10.10.100 port 25
So now on each client on lan2 I have to have a hosts file entry pointing the hostname for the fxp0 ipaddy to the fxp2 ipaddy. Among other things that makes resolving domains where the authoritative name server is on lan 1 hard. I thought bridging might be the best bet, but after hearing about the no broadcast thing expecially and how it handles it, I'm starting to think the same, that without both routers, its just super nasty.

 

[cleverhandle]

Originally posted by: Soybomb
The part that I left out however is fxp0 actually has multiple IPs assigned to so then I run into problems with multiple port conflicts.

Gah... it gets worse and worse.

Time to think about some serious restructuring. This is a trainwreck. Even if you can hack something together for the moment, it's going to come back and kick you in the *ss with security or maintainability eventually.

 

[Soybomb]

Originally posted by: cleverhandle

Originally posted by: Soybomb
The part that I left out however is fxp0 actually has multiple IPs assigned to so then I run into problems with multiple port conflicts.

Gah... it gets worse and worse.

Time to think about some serious restructuring. This is a trainwreck. Even if you can hack something together for the moment, it's going to come back and kick you in the *ss with security or maintainability eventually.

I'd love to but there is simply no restructuring possible. I think its either going to be partially working kludgey solution or traffic over the internet.

 

[cleverhandle]

I'll take traffic over the Internet for sure. A little slow I can live with. A network that's too complex for me to understand and trust I can't. Especially when I'm the one running it.