Multi gateway balancing with iptables

6 Comments

1. sysenm
   December 6, 2009 10:42 am

   Hello

   Thank you for such a wonderful article/guide.

   I have started working on load balancing and i am trying to achieve something similar to what you have described here but i would like to mark my traffic (using iptables and dport) so that i can have more control. Here is the scenario:
   eth0 – static adress to internal lan
   eth1 – isp 1
   eth2 – isp 2

   1 – http, dns, ftp, ssl, ssh are marked 1 (based on destination port)
   2 – chat and voip are marked 2
   3 – ip ru add fwmark 1 table 1
   ip ru add fwmark 2 table 2
   4 – ip route add default via eth1 table 1
   ip route add default via eth2 table 2
   5 – add multiple default routes in main. ( . . . nexthop via eth1 nexthop via eth2)

   The problem i am facing is that any unmarked traffic is alternating between eth0 and eth1. Is this suppose to be normal ?
   Is it possible to do without adding any kind of default route in the main ? What kinds of problem can be faced if i have too much un-marked traffic ??

   Waiting for your help!

   Thanks again!

   • agentk
     December 6, 2009 11:22 am

     Ok. I’ll start with the questions in order:

     If eth0 and eth1 are not typos and you are referring to external bound traffic, then ‘No’. It is not normal that external bound traffic be alternated between an internal lan dev (which should not happen as eth1 and eth2 should be the only devices with a gateway) and an isp. What could be causing it? Do not know without seeing a lot more detail as it is not a problem I have encountered so far.

     No default route in main? Yes. But. There still has to be a default route in the main table just make it a dummy route though as seen at: http://www.clintoneast.com/articles/multihomed.php
     ifconfig dummy0 1.2.3.4
ip route add default dev dummy0

     To much un-marked traffic? No problem. It will just be redirected to the default route in the main table.

     Ok. What you want sounds very similar to what I have here just using ip route nexthops for load balancing instead of iptables though.
     I am going to suggest (the process I went through) that you start off with the routes as separated as possible first and use iptables for debugging your traffic. For example, when I was testing nexthop balancing I used (in addition to table 1 & 2) fwmark 3 / table 3 and gave it a nexthop default route. Then in iptables just tested for any NEW un-MARKed traffic and gave it mark 3. ie:

     [code]iptables -t mangle -A PREROUTING -m state –state NEW -m connmark ! –mark 0 -j CONNMARK –set-mark 3[/code]

     Somewhere just after testing for traffic that I want sent to a specific table. (This is also just for forwarded traffic. For traffic from this server use the OUTPUT chain).

     After getting that in place I test it out to make sure traffic is going out and getting returned and check the packet counts in iptables:

     [code]iptables -t mangle -L PREROUTING -nv[/code]

     You should see the pkts column increase each time a request is forwarded.

     If you then use the iptables balancing rules ala ‘balance1’ chain in the post, you can then do [code]iptables -t mangle -L balance1 -nv[/code] to see the number of packets being directed to each interface.

     That help at all? When I was trying to work out where packets were going I went all the way to logging all packets between the interfaces in mangle if per interface for just port 80 traffic:
     [code]
     -A PREROUTING -s EXTERNALIP -p tcp -m tcp –dport 80 -j LOG –log-prefix "[d/PREROUTING dport]: "
     -A POSTROUTING -s EXTERNALIP -p tcp -m tcp –dport 80 -j LOG –log-prefix "[d/POSTROUTING dport]: "
     -A INPUT -s EXTERNALIP -p tcp -m tcp –dport 80 -j LOG –log-prefix "[d/INPUT dport]: "
     -A OUTPUT -s EXTERNALIP -p tcp -m tcp –dport 80 -j LOG –log-prefix "[d/OUTPUT dport]: "
     -A FORWARD -s EXTERNALIP -p tcp -m tcp –dport 80 -j LOG –log-prefix "[d/FORWARD dport]: "

     -A PREROUTING -d EXTERNALIP -p tcp -m tcp –sport 80 -j LOG –log-prefix "[d/PREROUTING sport]: "
     -A POSTROUTING -d EXTERNALIP -p tcp -m tcp –sport 80 -j LOG –log-prefix "[d/POSTROUTING sport]: "
     -A INPUT -d EXTERNALIP -p tcp -m tcp –sport 80 -j LOG –log-prefix "[d/INPUT sport]: "
     -A OUTPUT -d EXTERNALIP -p tcp -m tcp –sport 80 -j LOG –log-prefix "[d/OUTPUT sport]: "
     -A FORWARD -d EXTERNALIP -p tcp -m tcp –sport 80 -j LOG –log-prefix "[d/FORWARD sport]: "
     [/code]

     Then monitor syslog as I generate traffic to debug what was going on.

2. sysenm
   December 7, 2009 2:19 am

   Thanks dude!

   Here is a more cleaner explanation now with a simpler set of commands..

   eth0 – DHCP server is running on it, serving clients of lan
   eth1 – is connected to isp1 with ip 192.168.0.101 and gateway 192.168.0.1
   eth2 – is connected to isp2 with ip 192.168.2.133 and gateway 192.168.2.1

   ip ru add fwmark 1 table 1
   ip route add default via 192.168.0.1 dev eth1 table 1

   iptables -t mangle -A PREROUTING -p tcp –dport 80 -j MARK –set-mark 1
   iptables -t mangle -A PREROUTING -p tcp –dport 53 -j MARK –set-mark 1

   iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
   iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE

   ip r del default main

   echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
   echo 0 > /proc/sys/net/ipv4/conf/eth2/rp_filter

   ifconfig dummy0 1.2.3.4
   ip route add default dev dummy0

   The above scenario does not work. The dummy is not helping. Why a default route is necessory in main ? and secondly how can i mark all the packets that are unmarked ?

   Thanks for a quick reply.

3. Elvar
   January 20, 2010 6:13 pm

   In order to get this working, what has to be done with routing tables and what not? I’ve tried http://lartc.org/howto/lartc.rpdb.multiple-links.html which works fine for outbound connections but it screws up my inbound port redirections because sometimes they go in eth1 and come out eth2 which obviously doesn’t work. I’d really like to get your solution working.

   Regards

4. John ORourke
   May 18, 2010 8:59 pm

   Just 2 notes for people trying this:

   1. In the rules under ‘# Choose our route and save the mark’, don’t use those rules because they just copy the connection mark to the packet mark, then copy that back to the connection mark again. Replace them with these, and that also allows you to remove the ‘# Check to see if we have already marked a packet’ section because these rules do the same thing:

   # Choose our route and save the mark (copies the connection mark to the packet mark)
   iptables -t mangle -A PREROUTING -m connmark ! –mark 0 -j CONNMARK –restore-mark
   iptables -t mangle -A OUTPUT -m connmark ! –mark 0 -j CONNMARK –restore-mark

   2. I found the same issue as the commenter above – with no default route in the ‘main’ routing table, a lot of things on the local machine didn’t work. I tried the dummy route as above, that didn’t work either. When I looked, it was sending packets out to the LAN with a source address of the dummy interface! I think this might be specific to my setup – I’m balancing outgoing connections between 2 gateways on the same interface / same LAN.

   Here’s my full setup, where the LAN is 172.26.16.0/24 and the gateways are 172.26.16.1 and 172.26.16.2:

   IP TABLES SCRIPT:
   Note – this does not include, but should work with, any NAT/masquerading rules you have.

   # flush existing rules:
   iptables -t mangle -F OUTPUT
   iptables -t mangle -F PREROUTING

   # Load balancing rules (Split 50/50 between fwmark 1/2)
   iptables -t mangle -F balance1 >/dev/null 2>&1 # flush balance1 if exists
   iptables -t mangle -X balance1 >/dev/null 2>&1 # delete balance1 if exists
   iptables -t mangle -N balance1
   iptables -t mangle -A balance1 -d 172.26.16.0/24 -j RETURN
   iptables -t mangle -A balance1 -m connmark ! –mark 0 -j RETURN
   iptables -t mangle -A balance1 -m state –state ESTABLISHED,RELATED -j RETURN
   iptables -t mangle -A balance1 -m statistic –mode nth –every 2 –packet 0 -j CONNMARK –set-mark 1
   iptables -t mangle -A balance1 -m statistic –mode nth –every 2 –packet 1 -j CONNMARK –set-mark 2

   # New outgoing connections
   iptables -t mangle -A PREROUTING -s 172.26.16.0/24 -m state –state NEW -j balance1
   iptables -t mangle -A OUTPUT -m state –state NEW -j balance1

   # Choose our route and save the mark
   iptables -t mangle -A PREROUTING -m connmark ! –mark 0 -j CONNMARK –restore-mark
   iptables -t mangle -A OUTPUT -m connmark ! –mark 0 -j CONNMARK –restore-mark

   IP ROUTING/RULES SCRIPT:

   ip route del default table main
   ip route del default table 1
   ip route del 172.26.16.0/24 dev eth0 table 1
   ip route del 172.26.16.0/24 dev eth0 table 2
   ip route del default table 2
   ip rule del fwmark 1
   ip rule del fwmark 2
   ip rule add fwmark 1 table 1
   ip rule add fwmark 2 table 2
   ip route add default via 172.26.16.1 table 1
   ip route add default via 172.26.16.3 table 2
   ip route add default via 172.26.16.1 table main
   ip route flush cache

   Hope that helps someone, and thanks for this article agentk – it was a massive help!

5. Sumera
   August 14, 2010 8:40 am

   Hello Evey One !!!
   i am very new in Linux .. ubuntu and squid type stuff… we have a little network of 30 workstations..

   i have one machine installed unbuntu 9.10 and squid 2.74 with 4 Ethernet cards.

   eth0 ……ISP-1
   eth1…….LAN-1
   eth2…….ISP-2
   eth3…….LAN2

   ( LAN1 : 192.168.0.1 )===> 20 Work Stations
   ( LAN2 : 192.168.2.1 )===> 10 Work Stations

   i just need to make it possible that LAN 1 will use the internet from WAN-1 and LAN-2 will use the Internet From WAN-2 ..

   although i want that if WAN-1 fails then only LAN1 should be routed to WAN-2… WAN-2 is pretty fine it does not go down mostly… Please Help me step by step i will be very thank full to you