Multi gateway routing with iptables and iproute2

Notes on multi gateway routing with iptables and iproute2, suggestions and corrections gladly accepted. My notes may be incomplete or just plain wrong, I pieced them together after getting it working.

Running on Ubuntu-9.10 with two internet connections ppp0 and ppp1 both with static IP’s from two different internet providers in Australia (iiNet and Internode).

Preperation:

Extra ip route tables per gateway.

Add tables to /etc/iproute2/rt_tables. Table names and numbers can be anything as long as they are consistent later on.

echo -e "101 connection1n102 connection2" | sudo tee -a /etc/iproute2/rt_tables

Add routes to the extra rule tables. Copy the local routes from the main table then add the default gateway specific to this connection. Replace the vars at the beginning with your relevant settings.

#!/bin/sh
DEV1=ppp0
IP1=100.0.1.1
GW1=100.0.1.254
TABLE2=connection2
DEV2=ppp1
IP2=100.0.2.1
GW2=100.0.2.254
ip route flush table $TABLE1
ip route flush table $TABLE2
ip route show table main | grep -Ev '(^default|ppp)' | while read ROUTE ; do
    ip route add table $TABLE1 $ROUTE
    ip route add table $TABLE2 $ROUTE
done
ip route add table $TABLE1 $GW1 dev $DEV1 src $IP1
ip route add table $TABLE2 $GW2 dev $DEV2 src $IP2
ip route add table $TABLE1 default via $GW1
ip route add table $TABLE2 default via $GW2

ip route output:

~# ip route show
100.0.1.254 dev ppp0  proto kernel  scope link  src 100.0.1.1
100.0.2.254 dev ppp1  proto kernel  scope link  src 100.0.2.1
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.1
default via 100.0.1.254 dev ppp0
~# ip route show table connection1
100.0.1.254 dev ppp0  proto kernel  scope link  src 100.0.1.1
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.1
default via 100.0.1.254 dev ppp0

~# ip route show table connection2
100.0.2.254 dev ppp1  proto kernel  scope link  src 100.0.2.1
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.1
default via 100.0.2.254 dev ppp1

Add the ip rules:

ip rule add from 100.0.1.1 lookup connection1
ip rule add from 100.0.2.1 lookup connection2
ip rule add fwmark 1 lookup connection1
ip rule add fwmark 2 lookup connection2

Add the iptables rules for SNAT:

iptables -A POSTROUTING -o ppp0 -j SNAT --to-source 100.0.1.1
iptables -A POSTROUTING -o ppp1 -j SNAT --to-source 100.0.2.1

And finally add the rules for marking the connection they should be going out on. The first PREROUTING rule is for packets we forward to be returned via the interface they were received on. The OUTPUT rule is for packets handled on this PC to be returned on the correct interface too. We only want to mark new packets and restore marks on established connections else the packets

-A PREROUTING          -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
-A OUTPUT              -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
-A PREROUTING -i ppp0  -m state --state NEW                 -j CONNMARK --set-mark 1
-A PREROUTING -i ppp1  -m state --state NEW                 -j CONNMARK --set-mark 2
-A PREROUTING -m connmark --mark 1                          -j MARK --set-mark 1
-A PREROUTING -m connmark --mark 2                          -j MARK --set-mark 2
-A PREROUTING -m state --state NEW -m connmark ! --mark 0   -j CONNMARK --save-mark

Selective routing:

To send all outgoing traffic on a specific table:

-A PREROUTING -i eth0 -m state --state NEW -p tcp --dport  80 -j CONNMARK --set-mark 2
-A PREROUTING -i eth0 -m state --state NEW -p tcp --dport 443 -j CONNMARK --set-mark 2

 

References:

http://www.clintoneast.com/articles/multihomed.php

http://linux-ip.net/html/adv-multi-internet.html

This entry was posted in Server, Ubuntu and tagged , , . Bookmark the permalink. Both comments and trackbacks are currently closed.

5 Comments

  1. Randy Wallace
    November 30, 2009 9:42 am

    A few years ago, I did a project overseas which involved providing internet to a LAN. At the time, I chose to use a Linux Server / Router to do 99% of the work. Our internet connection was a farm of Internet Satellite connections (price and dish-size was a factor, which led to this solution). I ended up writing a Python Script which performed a myriad of tasks to include: balancing subscribers across internet connections, traffic shaping rules manipulation (via tc), firewall port blocking, etc… You may, then, be interested in seeing the rules created by my script: http://greamin.com/server/PastWork/UbuntuGateway.html . Subsequently, on that site, there is also the script, and the configuration file, which made my life remarkably easy! It worked *beautifully* and provided a high level of service per subscriber, regardless of their network demands!

    • November 30, 2009 11:41 am

      Thanks Randy, some good examples of real world use in there. From what I can see users were partitioned per satellite? I have not looked at failover or QoS yet. I was thinking of just checking interfaces status via cron and reconfiguring based on that.
      My biggest problem was that one of my providers blocks outgoing packets not marked as being from my IP. So if a packet ended up going via the wrong provider they were just being dropped.
      This most affected returning of incoming packets as outgoing connections always ended up back on the connection they were sent from.
      For load balancing ‘ip route default nexthop’ seems to work ok, but stutters quite bad when a connection goes down.
      As does ‘iptables -m statistic –mode nth –every 2′ balancing. But doing it in iptables for outgoing connections seems to balance better and provide better throughput.
      I guess that might be the subject of a new post.

      • Randy Wallace
        November 30, 2009 5:38 pm

        Yes, you’re correct; I found that a true ‘multi-home’ gateway for one subscriber turned into a housekeeping nightmare. The problem gets worse, though: without a single IP for traffic to ‘relate’ to, HTTPS and other secured technologies would wreak havoc. Subsequently, there was not way possible to determine which satellite deserved which port(s) traffic, etc… (i.e. ‘fair’ load balancing across dishes was nearly impossible) Thus, we assigned a set number of subscribers to each satellite. This also helped to mitigate problems concerning outage(s).

        Upon testing multiple different scenario(s) using multi-homing, I found that they were all to unreliable, especially when handling the loads of traffic we were passing (and frequent outages). Thus, in the end, I was looking into the Click Modular Router for writing ‘the perfect routing code’ for my application. I never got around to it, though…

        You can only imagine what kinds of issues I encountered using Satellite Internet; avoid it AT ALL COSTS! (We had no choice at the time, I was deployed in Iraq)

    • Michael Mercier
      August 7, 2013 9:56 pm

      I just found this old post and the Python script that you mentioned in your post is really interesting me, but the link is broken. Could you post a new valid link or send the script to me directly on
      michael.mercier@NOSPAMinria.fr (without the NOSPAM of course)?

      Thanks for this useful post anyway.

  2. B Raveendra Reddy
    May 11, 2011 3:07 pm

    ip route add table $TABLE1 $GW1 dev $DEV1 src $IP1

    This gives an error “RTNETLINK answers: Invalid argument”. Could you please correct it?

    Lastly, thanks for good work.

One Trackback