Multi gateway balancing with iptables

I have been testing load balancing via both iptables and ip route nexthop for a couple of days now.  They both work pretty well too.

This only balances outgoing traffic as incoming traffic balanced via DNS RR and the firewall just returns the traffic on the interface it arrived on as per the previous post.

On the whole, I prefer the iptables solution. It seems to balance the traffic better. ip route balances outgoing connections based on nexthop of the route to that host is not already in it’s routing cache. While iptables balances traffic by alternate outgoing connections. The only downside I have seen is occasional connection drops to the BlackBerry servers.

After 24 hours of iptables balancing:

ppp0      Link encap:Point-to-Point Protocol
          RX bytes:1186783900 (1.1 GB)  TX bytes:1290603327 (1.2 GB)
ppp1      Link encap:Point-to-Point Protocol
          RX bytes:1109227490 (1.1 GB)  TX bytes:1140565429 (1.1 GB)

 

This is using inclusion rules for determining balanced traffic. These are the rules that ended up on the production server:

 

# Load balancing rules (Split 50/50 between fwmark 1/2)
iptables -t mangle -A balance1 -d 192.168.0.0/16      -j RETURN
iptables -t mangle -A balance1 -d 10.0.0.0/8          -j RETURN
iptables -t mangle -A balance1 -m connmark ! --mark 0 -j RETURN
iptables -t mangle -A balance1 -m state --state ESTABLISHED,RELATED -j RETURN
iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 0 -j CONNMARK --set-mark 1
iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 1 -j CONNMARK --set-mark 2

# Check to see if we have already marked a packet
iptables -t mangle -A PREROUTING  -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT      -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark

# Mark incoming connections to return on the interface they came in on
iptables -t mangle -A PREROUTING          -i ppp0                     -m state --state NEW  -j CONNMARK --set-mark 1
iptables -t mangle -A PREROUTING          -i ppp1                     -m state --state NEW  -j CONNMARK --set-mark 2

# New outgoing packets
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  22 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  25 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  80 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport 443 -m state --state NEW  -j balance1
iptables -t mangle -A OUTPUT                       -p tcp --dport  80 -m state --state NEW  -j balance1

# Choose our route and save the mark
iptables -t mangle -A PREROUTING  -m connmark --mark 1 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING  -m connmark --mark 2 -j MARK --set-mark 2
iptables -t mangle -A PREROUTING  -m state --state NEW -m connmark ! --mark 0  -j CONNMARK --save-mark

 

That’s all of the CONNMARK and MARK related rules I use.

The new outgoing packets section is where I choose what packets should be balanced and accounts for about 95% of our outgoing traffic.

The balance1 chain just has some checks at the beginning to catch further traffic that should not be balanced in case some rule gets messed up.

Of the new outgoing packets rules, the PREROUTE lines are for forwarded traffic and the OUTPUT rule is for traffic generated on that host by a transparent squid proxy.

Hope somebody finds that useful one day.

 

 

This entry was posted in Server, Ubuntu and tagged , , , , , . Bookmark the permalink. Both comments and trackbacks are currently closed.

6 Comments

  1. sysenm
    December 6, 2009 10:42 am

    Hello

    Thank you for such a wonderful article/guide.

    I have started working on load balancing and i am trying to achieve something similar to what you have described here but i would like to mark my traffic (using iptables and dport) so that i can have more control. Here is the scenario:
    eth0 – static adress to internal lan
    eth1 – isp 1
    eth2 – isp 2

    1 – http, dns, ftp, ssl, ssh are marked 1 (based on destination port)
    2 – chat and voip are marked 2
    3 – ip ru add fwmark 1 table 1
    ip ru add fwmark 2 table 2
    4 – ip route add default via eth1 table 1
    ip route add default via eth2 table 2
    5 – add multiple default routes in main. ( . . . nexthop via eth1 nexthop via eth2)

    The problem i am facing is that any unmarked traffic is alternating between eth0 and eth1. Is this suppose to be normal ?
    Is it possible to do without adding any kind of default route in the main ? What kinds of problem can be faced if i have too much un-marked traffic ??

    Waiting for your help!

    Thanks again!

    • December 6, 2009 11:22 am

      Ok. I’ll start with the questions in order:

      If eth0 and eth1 are not typos and you are referring to external bound traffic, then ‘No’. It is not normal that external bound traffic be alternated between an internal lan dev (which should not happen as eth1 and eth2 should be the only devices with a gateway) and an isp. What could be causing it? Do not know without seeing a lot more detail as it is not a problem I have encountered so far.

      No default route in main? Yes. But. There still has to be a default route in the main table just make it a dummy route though as seen at: http://www.clintoneast.com/articles/multihomed.php
      ifconfig dummy0 1.2.3.4
      ip route add default dev dummy0

      To much un-marked traffic? No problem. It will just be redirected to the default route in the main table.

      Ok. What you want sounds very similar to what I have here just using ip route nexthops for load balancing instead of iptables though.
      I am going to suggest (the process I went through) that you start off with the routes as separated as possible first and use iptables for debugging your traffic. For example, when I was testing nexthop balancing I used (in addition to table 1 & 2) fwmark 3 / table 3 and gave it a nexthop default route. Then in iptables just tested for any NEW un-MARKed traffic and gave it mark 3. ie:

      [code]iptables -t mangle -A PREROUTING -m state –state NEW -m connmark ! –mark 0 -j CONNMARK –set-mark 3[/code]

      Somewhere just after testing for traffic that I want sent to a specific table. (This is also just for forwarded traffic. For traffic from this server use the OUTPUT chain).

      After getting that in place I test it out to make sure traffic is going out and getting returned and check the packet counts in iptables:

      [code]iptables -t mangle -L PREROUTING -nv[/code]

      You should see the pkts column increase each time a request is forwarded.

      If you then use the iptables balancing rules ala ‘balance1’ chain in the post, you can then do [code]iptables -t mangle -L balance1 -nv[/code] to see the number of packets being directed to each interface.

      That help at all? When I was trying to work out where packets were going I went all the way to logging all packets between the interfaces in mangle if per interface for just port 80 traffic:
      [code]
      -A PREROUTING -s EXTERNALIP -p tcp -m tcp –dport 80 -j LOG –log-prefix "[d/PREROUTING dport]: "
      -A POSTROUTING -s EXTERNALIP -p tcp -m tcp –dport 80 -j LOG –log-prefix "[d/POSTROUTING dport]: "
      -A INPUT -s EXTERNALIP -p tcp -m tcp –dport 80 -j LOG –log-prefix "[d/INPUT dport]: "
      -A OUTPUT -s EXTERNALIP -p tcp -m tcp –dport 80 -j LOG –log-prefix "[d/OUTPUT dport]: "
      -A FORWARD -s EXTERNALIP -p tcp -m tcp –dport 80 -j LOG –log-prefix "[d/FORWARD dport]: "

      -A PREROUTING -d EXTERNALIP -p tcp -m tcp –sport 80 -j LOG –log-prefix "[d/PREROUTING sport]: "
      -A POSTROUTING -d EXTERNALIP -p tcp -m tcp –sport 80 -j LOG –log-prefix "[d/POSTROUTING sport]: "
      -A INPUT -d EXTERNALIP -p tcp -m tcp –sport 80 -j LOG –log-prefix "[d/INPUT sport]: "
      -A OUTPUT -d EXTERNALIP -p tcp -m tcp –sport 80 -j LOG –log-prefix "[d/OUTPUT sport]: "
      -A FORWARD -d EXTERNALIP -p tcp -m tcp –sport 80 -j LOG –log-prefix "[d/FORWARD sport]: "
      [/code]

      Then monitor syslog as I generate traffic to debug what was going on.

  2. sysenm
    December 7, 2009 2:19 am

    Thanks dude!

    Here is a more cleaner explanation now with a simpler set of commands..

    eth0 – DHCP server is running on it, serving clients of lan
    eth1 – is connected to isp1 with ip 192.168.0.101 and gateway 192.168.0.1
    eth2 – is connected to isp2 with ip 192.168.2.133 and gateway 192.168.2.1

    ip ru add fwmark 1 table 1
    ip route add default via 192.168.0.1 dev eth1 table 1

    iptables -t mangle -A PREROUTING -p tcp –dport 80 -j MARK –set-mark 1
    iptables -t mangle -A PREROUTING -p tcp –dport 53 -j MARK –set-mark 1

    iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
    iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE

    ip r del default main

    echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
    echo 0 > /proc/sys/net/ipv4/conf/eth2/rp_filter

    ifconfig dummy0 1.2.3.4
    ip route add default dev dummy0

    The above scenario does not work. The dummy is not helping. Why a default route is necessory in main ? and secondly how can i mark all the packets that are unmarked ?

    Thanks for a quick reply.

  3. Elvar
    January 20, 2010 6:13 pm

    In order to get this working, what has to be done with routing tables and what not? I’ve tried http://lartc.org/howto/lartc.rpdb.multiple-links.html which works fine for outbound connections but it screws up my inbound port redirections because sometimes they go in eth1 and come out eth2 which obviously doesn’t work. I’d really like to get your solution working.

    Regards

  4. John ORourke
    May 18, 2010 8:59 pm

    Just 2 notes for people trying this:

    1. In the rules under ‘# Choose our route and save the mark’, don’t use those rules because they just copy the connection mark to the packet mark, then copy that back to the connection mark again. Replace them with these, and that also allows you to remove the ‘# Check to see if we have already marked a packet’ section because these rules do the same thing:

    # Choose our route and save the mark (copies the connection mark to the packet mark)
    iptables -t mangle -A PREROUTING -m connmark ! –mark 0 -j CONNMARK –restore-mark
    iptables -t mangle -A OUTPUT -m connmark ! –mark 0 -j CONNMARK –restore-mark

    2. I found the same issue as the commenter above – with no default route in the ‘main’ routing table, a lot of things on the local machine didn’t work. I tried the dummy route as above, that didn’t work either. When I looked, it was sending packets out to the LAN with a source address of the dummy interface! I think this might be specific to my setup – I’m balancing outgoing connections between 2 gateways on the same interface / same LAN.

    Here’s my full setup, where the LAN is 172.26.16.0/24 and the gateways are 172.26.16.1 and 172.26.16.2:

    IP TABLES SCRIPT:
    Note – this does not include, but should work with, any NAT/masquerading rules you have.

    # flush existing rules:
    iptables -t mangle -F OUTPUT
    iptables -t mangle -F PREROUTING

    # Load balancing rules (Split 50/50 between fwmark 1/2)
    iptables -t mangle -F balance1 >/dev/null 2>&1 # flush balance1 if exists
    iptables -t mangle -X balance1 >/dev/null 2>&1 # delete balance1 if exists
    iptables -t mangle -N balance1
    iptables -t mangle -A balance1 -d 172.26.16.0/24 -j RETURN
    iptables -t mangle -A balance1 -m connmark ! –mark 0 -j RETURN
    iptables -t mangle -A balance1 -m state –state ESTABLISHED,RELATED -j RETURN
    iptables -t mangle -A balance1 -m statistic –mode nth –every 2 –packet 0 -j CONNMARK –set-mark 1
    iptables -t mangle -A balance1 -m statistic –mode nth –every 2 –packet 1 -j CONNMARK –set-mark 2

    # New outgoing connections
    iptables -t mangle -A PREROUTING -s 172.26.16.0/24 -m state –state NEW -j balance1
    iptables -t mangle -A OUTPUT -m state –state NEW -j balance1

    # Choose our route and save the mark
    iptables -t mangle -A PREROUTING -m connmark ! –mark 0 -j CONNMARK –restore-mark
    iptables -t mangle -A OUTPUT -m connmark ! –mark 0 -j CONNMARK –restore-mark

    IP ROUTING/RULES SCRIPT:

    ip route del default table main
    ip route del default table 1
    ip route del 172.26.16.0/24 dev eth0 table 1
    ip route del 172.26.16.0/24 dev eth0 table 2
    ip route del default table 2
    ip rule del fwmark 1
    ip rule del fwmark 2
    ip rule add fwmark 1 table 1
    ip rule add fwmark 2 table 2
    ip route add default via 172.26.16.1 table 1
    ip route add default via 172.26.16.3 table 2
    ip route add default via 172.26.16.1 table main
    ip route flush cache

    Hope that helps someone, and thanks for this article agentk – it was a massive help!

  5. Sumera
    August 14, 2010 8:40 am

    Hello Evey One !!!
    i am very new in Linux .. ubuntu and squid type stuff… we have a little network of 30 workstations..

    i have one machine installed unbuntu 9.10 and squid 2.74 with 4 Ethernet cards.

    eth0 ……ISP-1
    eth1…….LAN-1
    eth2…….ISP-2
    eth3…….LAN2

    ( LAN1 : 192.168.0.1 )===> 20 Work Stations
    ( LAN2 : 192.168.2.1 )===> 10 Work Stations

    i just need to make it possible that LAN 1 will use the internet from WAN-1 and LAN-2 will use the Internet From WAN-2 ..

    although i want that if WAN-1 fails then only LAN1 should be routed to WAN-2… WAN-2 is pretty fine it does not go down mostly… Please Help me step by step i will be very thank full to you

One Trackback

  • By links for 2009-12-01 « Bloggitation on December 2, 2009 at 4:08 pm

    […] Multi gateway balancing with iptables (tags: iptables security linux sysadmin) Possibly related posts: (automatically generated)Tags on Blogs Make for Great SEOTags to friends Categories: Links Comments (0) Trackbacks (0) Leave a comment Trackback […]