Elvar started with all the correct elements to setup a functioning multrouting gateway. Two connections from two different providers, eth1, eth2. Running on an Ubuntu box with eth0 as internal. Both internet connections working on their own.

But alas. Whenever both eth1 and eth2 were active on the host, outgoing packets just would not go out. I don’t know if incoming packets were being replied to as we were unable to check that.

If just eth1 or eth2 was active than everything traversed ok. But we wanted it to work with both connections.

After a LONG time of diagnosing we noticed there were two default routes in iproute2 (ip’s fudged, but you get the idea):

firewall# ip route show
...
default via 1.1.2.1 dev eth2  metric 100
default via 1.1.3.1 dev eth1  metric 100

My firewall often has two default routes listed on the main table (ppp0, and ppp1) until the cleanup script fixes it. Without any negative side effect. I may just be lucky though.

Upon removing the eth2 line from table main, everything started working correctly. Incoming, outgoing, forwarding, balancing.

I also noticed the output of ifconfig eth2 looked a bit screwed too, but there was not much we could do about that as it was assigned by dhcp.

eth2      Link encap:Ethernet  HWaddr 00:11:22:33:44:55
    inet addr:1.1.2.1  Bcast:255.255.255.255  Mask:255.255.255.0

See it? No not the MAC address. The broadcast address. A quick ipcalc 1.1.2.1/24 gives me a broadcast of 1.1.2.255. But once we removed the eth2 default route line, it all started working again and didn’t get to dig into it to see if the broadcast actually affected it.

So that’s all. Just remember there are a lot more things to go wrong in a multigateway setup, including things outside of iptables.

I cannot remember what site I used as a reference when creating the diagram. The original was an ascii chart though. This is created in dia and exported to pdf too. Both attached for convenience. CC licensed as long as the original chart I copied allows it.

Packet flow examples:

All packets to or from localhost travel down the left side of the chart.

From external destination localhost: PREROUTING -> INPUT -> [Local Process] -> OUTPUT -> POSTROUTING.

From localhost destination external: OUTPUT -> POSTROUTING -> [External Host] -> PREROUTING -> INPUT.

All forwarded packets travel the right side of the chart and travel all three tables coming in and then again going out.

From external dest internal: PREROUTING -> FORWARD -> POSTROUTING.

Internal response to external: PREROUTING -> FORWARD -> POSTROUTING.

and so forth.

This is what caught me for a while. Forwarded packet travel the right side route and get out of (or in to) the network. The response then gets generated and does not start from FORWARD or POSTROUTING, but from PREROUTING again. So all NEW forwarded packets need to be marked in PREROUTING and the mark saved. Not new packets need that mark restored in PREROUTING, and all packets need that mark restored in POSTROUTING. (As demonstrated by my previous post).

Locally generated traffic only sees OUTPUT and POSTROUTING before hitting the network and needs to be marked before hitting POSTROUTING, hence the OUTPUT chain rules in my previous post.

I will accept changes to the chart too if anybody wants modifications made.

iptables routing.dia

iptables routing.pdf

This only balances outgoing traffic as incoming traffic balanced via DNS RR and the firewall just returns the traffic on the interface it arrived on as per the previous post.

On the whole, I prefer the iptables solution. It seems to balance the traffic better. ip route balances outgoing connections based on nexthop of the route to that host is not already in it’s routing cache. While iptables balances traffic by alternate outgoing connections. The only downside I have seen is occasional connection drops to the BlackBerry servers.

After 24 hours of iptables balancing:

ppp0      Link encap:Point-to-Point Protocol
          RX bytes:1186783900 (1.1 GB)  TX bytes:1290603327 (1.2 GB)
ppp1      Link encap:Point-to-Point Protocol
          RX bytes:1109227490 (1.1 GB)  TX bytes:1140565429 (1.1 GB)

This is using inclusion rules for determining balanced traffic. These are the rules that ended up on the production server:

# Load balancing rules (Split 50/50 between fwmark 1/2)
iptables -t mangle -A balance1 -d 192.168.0.0/16      -j RETURN
iptables -t mangle -A balance1 -d 10.0.0.0/8          -j RETURN
iptables -t mangle -A balance1 -m connmark ! --mark 0 -j RETURN
iptables -t mangle -A balance1 -m state --state ESTABLISHED,RELATED -j RETURN
iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 0 -j CONNMARK --set-mark 1
iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 1 -j CONNMARK --set-mark 2

# Check to see if we have already marked a packet
iptables -t mangle -A PREROUTING  -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT      -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark

# Mark incoming connections to return on the interface they came in on
iptables -t mangle -A PREROUTING          -i ppp0                     -m state --state NEW  -j CONNMARK --set-mark 1
iptables -t mangle -A PREROUTING          -i ppp1                     -m state --state NEW  -j CONNMARK --set-mark 2

# New outgoing packets
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  22 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  25 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  80 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport 443 -m state --state NEW  -j balance1
iptables -t mangle -A OUTPUT                       -p tcp --dport  80 -m state --state NEW  -j balance1

# Choose our route and save the mark
iptables -t mangle -A PREROUTING  -m connmark --mark 1 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING  -m connmark --mark 2 -j MARK --set-mark 2
iptables -t mangle -A PREROUTING  -m state --state NEW -m connmark ! --mark 0  -j CONNMARK --save-mark

That’s all of the CONNMARK and MARK related rules I use.

The new outgoing packets section is where I choose what packets should be balanced and accounts for about 95% of our outgoing traffic.

The balance1 chain just has some checks at the beginning to catch further traffic that should not be balanced in case some rule gets messed up.

Of the new outgoing packets rules, the PREROUTE lines are for forwarded traffic and the OUTPUT rule is for traffic generated on that host by a transparent squid proxy.

Hope somebody finds that useful one day.