Odd iproute2 gateway behavior with two gateways

After helping a frustrated yet patient Elvar I thought I better post about the situation we came across incase anybody else finds themselves in the same situation.

Elvar started with all the correct elements to setup a functioning multrouting gateway. Two connections from two different providers, eth1, eth2. Running on an Ubuntu box with eth0 as internal. Both internet connections working on their own.

But alas. Whenever both eth1 and eth2 were active on the host, outgoing packets just would not go out. I don’t know if incoming packets were being replied to as we were unable to check that.

If just eth1 or eth2 was active than everything traversed ok. But we wanted it to work with both connections.

After a LONG time of diagnosing we noticed there were two default routes in iproute2 (ip’s fudged, but you get the idea):

firewall# ip route show
default via dev eth2  metric 100
default via dev eth1  metric 100

My firewall often has two default routes listed on the main table (ppp0, and ppp1) until the cleanup script fixes it. Without any negative side effect. I may just be lucky though.

Upon removing the eth2 line from table main, everything started working correctly. Incoming, outgoing, forwarding, balancing.

I also noticed the output of ifconfig eth2 looked a bit screwed too, but there was not much we could do about that as it was assigned by dhcp.

eth2      Link encap:Ethernet  HWaddr 00:11:22:33:44:55
    inet addr:  Bcast:  Mask:

See it? No not the MAC address. The broadcast address. A quick ipcalc gives me a broadcast of But once we removed the eth2 default route line, it all started working again and didn’t get to dig into it to see if the broadcast actually affected it.

So that’s all. Just remember there are a lot more things to go wrong in a multigateway setup, including things outside of iptables.

Posted in Server, Ubuntu | Tagged , , , | Comments closed

Iptables packet flow diagram

Here is the diagram I keep stuck to my wall for reference when dealing with iptables.

I cannot remember what site I used as a reference when creating the diagram. The original was an ascii chart though. This is created in dia and exported to pdf too. Both attached for convenience. CC licensed as long as the original chart I copied allows it.

Packet flow examples:

All packets to or from localhost travel down the left side of the chart.

From external destination localhost: PREROUTING -> INPUT -> [Local Process] -> OUTPUT -> POSTROUTING.

From localhost destination external: OUTPUT -> POSTROUTING -> [External Host] -> PREROUTING -> INPUT.

All forwarded packets travel the right side of the chart and travel all three tables coming in and then again going out.

From external dest internal: PREROUTING -> FORWARD -> POSTROUTING.

Internal response to external: PREROUTING -> FORWARD -> POSTROUTING.

and so forth.

This is what caught me for a while. Forwarded packet travel the right side route and get out of (or in to) the network. The response then gets generated and does not start from FORWARD or POSTROUTING, but from PREROUTING again. So all NEW forwarded packets need to be marked in PREROUTING and the mark saved. Not new packets need that mark restored in PREROUTING, and all packets need that mark restored in POSTROUTING. (As demonstrated by my previous post).

Locally generated traffic only sees OUTPUT and POSTROUTING before hitting the network and needs to be marked before hitting POSTROUTING, hence the OUTPUT chain rules in my previous post.

I will accept changes to the chart too if anybody wants modifications made.

iptables routing.dia

iptables routing.pdf

Posted in Uncategorized | Tagged , , , | Comments closed

Multi gateway balancing with iptables

I have been testing load balancing via both iptables and ip route nexthop for a couple of days now.  They both work pretty well too.

This only balances outgoing traffic as incoming traffic balanced via DNS RR and the firewall just returns the traffic on the interface it arrived on as per the previous post.

On the whole, I prefer the iptables solution. It seems to balance the traffic better. ip route balances outgoing connections based on nexthop of the route to that host is not already in it’s routing cache. While iptables balances traffic by alternate outgoing connections. The only downside I have seen is occasional connection drops to the BlackBerry servers.

After 24 hours of iptables balancing:

ppp0      Link encap:Point-to-Point Protocol
          RX bytes:1186783900 (1.1 GB)  TX bytes:1290603327 (1.2 GB)
ppp1      Link encap:Point-to-Point Protocol
          RX bytes:1109227490 (1.1 GB)  TX bytes:1140565429 (1.1 GB)


This is using inclusion rules for determining balanced traffic. These are the rules that ended up on the production server:


# Load balancing rules (Split 50/50 between fwmark 1/2)
iptables -t mangle -A balance1 -d      -j RETURN
iptables -t mangle -A balance1 -d          -j RETURN
iptables -t mangle -A balance1 -m connmark ! --mark 0 -j RETURN
iptables -t mangle -A balance1 -m state --state ESTABLISHED,RELATED -j RETURN
iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 0 -j CONNMARK --set-mark 1
iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 1 -j CONNMARK --set-mark 2

# Check to see if we have already marked a packet
iptables -t mangle -A PREROUTING  -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT      -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark

# Mark incoming connections to return on the interface they came in on
iptables -t mangle -A PREROUTING          -i ppp0                     -m state --state NEW  -j CONNMARK --set-mark 1
iptables -t mangle -A PREROUTING          -i ppp1                     -m state --state NEW  -j CONNMARK --set-mark 2

# New outgoing packets
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  22 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  25 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  80 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport 443 -m state --state NEW  -j balance1
iptables -t mangle -A OUTPUT                       -p tcp --dport  80 -m state --state NEW  -j balance1

# Choose our route and save the mark
iptables -t mangle -A PREROUTING  -m connmark --mark 1 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING  -m connmark --mark 2 -j MARK --set-mark 2
iptables -t mangle -A PREROUTING  -m state --state NEW -m connmark ! --mark 0  -j CONNMARK --save-mark


That’s all of the CONNMARK and MARK related rules I use.

The new outgoing packets section is where I choose what packets should be balanced and accounts for about 95% of our outgoing traffic.

The balance1 chain just has some checks at the beginning to catch further traffic that should not be balanced in case some rule gets messed up.

Of the new outgoing packets rules, the PREROUTE lines are for forwarded traffic and the OUTPUT rule is for traffic generated on that host by a transparent squid proxy.

Hope somebody finds that useful one day.



Posted in Server, Ubuntu | Tagged , , , , , | Comments closed

Multi gateway routing with iptables and iproute2

Notes on multi gateway routing with iptables and iproute2, suggestions and corrections gladly accepted. My notes may be incomplete or just plain wrong, I pieced them together after getting it working.

Running on Ubuntu-9.10 with two internet connections ppp0 and ppp1 both with static IP’s from two different internet providers in Australia (iiNet and Internode).


Extra ip route tables per gateway.

Add tables to /etc/iproute2/rt_tables. Table names and numbers can be anything as long as they are consistent later on.

echo -e "101 connection1n102 connection2" | sudo tee -a /etc/iproute2/rt_tables

Add routes to the extra rule tables. Copy the local routes from the main table then add the default gateway specific to this connection. Replace the vars at the beginning with your relevant settings.

ip route flush table $TABLE1
ip route flush table $TABLE2
ip route show table main | grep -Ev '(^default|ppp)' | while read ROUTE ; do
    ip route add table $TABLE1 $ROUTE
    ip route add table $TABLE2 $ROUTE
ip route add table $TABLE1 $GW1 dev $DEV1 src $IP1
ip route add table $TABLE2 $GW2 dev $DEV2 src $IP2
ip route add table $TABLE1 default via $GW1
ip route add table $TABLE2 default via $GW2

ip route output:

~# ip route show dev ppp0  proto kernel  scope link  src dev ppp1  proto kernel  scope link  src dev eth0  proto kernel  scope link  src
default via dev ppp0
~# ip route show table connection1 dev ppp0  proto kernel  scope link  src dev eth0  proto kernel  scope link  src
default via dev ppp0

~# ip route show table connection2 dev ppp1  proto kernel  scope link  src dev eth0  proto kernel  scope link  src
default via dev ppp1

Add the ip rules:

ip rule add from lookup connection1
ip rule add from lookup connection2
ip rule add fwmark 1 lookup connection1
ip rule add fwmark 2 lookup connection2

Add the iptables rules for SNAT:

iptables -A POSTROUTING -o ppp0 -j SNAT --to-source
iptables -A POSTROUTING -o ppp1 -j SNAT --to-source

And finally add the rules for marking the connection they should be going out on. The first PREROUTING rule is for packets we forward to be returned via the interface they were received on. The OUTPUT rule is for packets handled on this PC to be returned on the correct interface too. We only want to mark new packets and restore marks on established connections else the packets

-A PREROUTING          -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
-A OUTPUT              -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
-A PREROUTING -i ppp0  -m state --state NEW                 -j CONNMARK --set-mark 1
-A PREROUTING -i ppp1  -m state --state NEW                 -j CONNMARK --set-mark 2
-A PREROUTING -m connmark --mark 1                          -j MARK --set-mark 1
-A PREROUTING -m connmark --mark 2                          -j MARK --set-mark 2
-A PREROUTING -m state --state NEW -m connmark ! --mark 0   -j CONNMARK --save-mark

Selective routing:

To send all outgoing traffic on a specific table:

-A PREROUTING -i eth0 -m state --state NEW -p tcp --dport  80 -j CONNMARK --set-mark 2
-A PREROUTING -i eth0 -m state --state NEW -p tcp --dport 443 -j CONNMARK --set-mark 2





Posted in Server, Ubuntu | Tagged , , | Comments closed

Ubuntu 9.10 impressions, extra packages.

I updated my main laptop to Ubuntu 9.10 on it’s release. It called for a fresh install as this laptop get’s a lot of crap installed left right and centre and an upgrade was not going to clean it up. (Apache, Nginx and Lighttpd, tomcat, netbeans all installed and starting on bootup, argh, plus many self compiled packages I installed without keeping track of.)

The only bugs I have been hit by was #446146. A Huawei E169 USB here modem only shows up as a memory card reader. Quick install of one of the kernel packages listed fixed this for me though.

A PC that after upgrade was trying to detect the harddrives as part of fakeraid setup and would fail to boot. Booted with the previous kernel, removed dmraid, and rebooted and she’s all good.

And gscan2pdf needed three packages from Jaunty to save as PDF again:

  • libmagickcore1_6.4.5.4.dfsg1-1ubuntu3.1_amd64.deb
  • libmagickwand1_6.4.5.4.dfsg1-1ubuntu3.1_amd64.deb
  • perlmagick_6.4.5.4.dfsg1-1ubuntu3.1_amd64.deb

I am really impressed with the direction that to boot theming is taking, well done Scott and team. Nvidia black redraw issues have gone away too. Sound is working better for me. Easier to select the speakers I want to output to. Havn’t tried Ubuntu One yet as most of my family is already using dropbox. Will probably try it for backing up some app data though.

The nvidia-settings tool would not save my settings for two monitor to xorg.conf as it could not parse it. So I instead just replaced the Screen Section in xorg.conf with the following and the extra monitor is auto detected and expanded to now:

Section "Screen"
    Identifier    "Default Screen"
    DefaultDepth    24
    Option         "TwinView" "1"
    Option         "TwinViewXineramaInfoOrder" "DFP, CRT"
    Option         "metamodes" "DFP: nvidia-auto-select +0+0, CRT: nvidia-auto-select +1280+0"
     SubSection     "Display"
        Depth       24

I still have to work out how to slow down my mouse further with X as the slowest I can set it to in Gnome is still to fast at times.

And incase anybody like to compare notes, these are the packages I have installed/removed in the first couple of days (some from PPA’s, etc).





I removed the list of fonts to see 1) what would happen, 2) my default font list was cluttered & 3) I can’t understand any of those languages anyway.


Posted in Desktop, Ubuntu | Tagged , , | Comments closed