Multi gateway balancing with iptables
I have been testing load balancing via both iptables and ip route nexthop for a couple of days now. They both work pretty well too.
This only balances outgoing traffic as incoming traffic balanced via DNS RR and the firewall just returns the traffic on the interface it arrived on as per the previous post.
On the whole, I prefer the iptables solution. It seems to balance the traffic better. ip route balances outgoing connections based on nexthop of the route to that host is not already in it’s routing cache. While iptables balances traffic by alternate outgoing connections. The only downside I have seen is occasional connection drops to the BlackBerry servers.
After 24 hours of iptables balancing:
ppp0 Link encap:Point-to-Point Protocol RX bytes:1186783900 (1.1 GB) TX bytes:1290603327 (1.2 GB) ppp1 Link encap:Point-to-Point Protocol RX bytes:1109227490 (1.1 GB) TX bytes:1140565429 (1.1 GB)
This is using inclusion rules for determining balanced traffic. These are the rules that ended up on the production server:
# Load balancing rules (Split 50/50 between fwmark 1/2) iptables -t mangle -A balance1 -d 192.168.0.0/16 -j RETURN iptables -t mangle -A balance1 -d 10.0.0.0/8 -j RETURN iptables -t mangle -A balance1 -m connmark ! --mark 0 -j RETURN iptables -t mangle -A balance1 -m state --state ESTABLISHED,RELATED -j RETURN iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 0 -j CONNMARK --set-mark 1 iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 1 -j CONNMARK --set-mark 2 # Check to see if we have already marked a packet iptables -t mangle -A PREROUTING -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark iptables -t mangle -A OUTPUT -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark # Mark incoming connections to return on the interface they came in on iptables -t mangle -A PREROUTING -i ppp0 -m state --state NEW -j CONNMARK --set-mark 1 iptables -t mangle -A PREROUTING -i ppp1 -m state --state NEW -j CONNMARK --set-mark 2 # New outgoing packets iptables -t mangle -A PREROUTING -i eth0 -p tcp --dport 22 -m state --state NEW -j balance1 iptables -t mangle -A PREROUTING -i eth0 -p tcp --dport 25 -m state --state NEW -j balance1 iptables -t mangle -A PREROUTING -i eth0 -p tcp --dport 80 -m state --state NEW -j balance1 iptables -t mangle -A PREROUTING -i eth0 -p tcp --dport 443 -m state --state NEW -j balance1 iptables -t mangle -A OUTPUT -p tcp --dport 80 -m state --state NEW -j balance1 # Choose our route and save the mark iptables -t mangle -A PREROUTING -m connmark --mark 1 -j MARK --set-mark 1 iptables -t mangle -A PREROUTING -m connmark --mark 2 -j MARK --set-mark 2 iptables -t mangle -A PREROUTING -m state --state NEW -m connmark ! --mark 0 -j CONNMARK --save-mark
That’s all of the CONNMARK and MARK related rules I use.
The new outgoing packets section is where I choose what packets should be balanced and accounts for about 95% of our outgoing traffic.
The balance1 chain just has some checks at the beginning to catch further traffic that should not be balanced in case some rule gets messed up.
Of the new outgoing packets rules, the PREROUTE lines are for forwarded traffic and the OUTPUT rule is for traffic generated on that host by a transparent squid proxy.
Hope somebody finds that useful one day.
Hello
Thank you for such a wonderful article/guide.
I have started working on load balancing and i am trying to achieve something similar to what you have described here but i would like to mark my traffic (using iptables and dport) so that i can have more control. Here is the scenario:
eth0 – static adress to internal lan
eth1 – isp 1
eth2 – isp 2
1 – http, dns, ftp, ssl, ssh are marked 1 (based on destination port)
2 – chat and voip are marked 2
3 – ip ru add fwmark 1 table 1
ip ru add fwmark 2 table 2
4 – ip route add default via eth1 table 1
ip route add default via eth2 table 2
5 – add multiple default routes in main. ( . . . nexthop via eth1 nexthop via eth2)
The problem i am facing is that any unmarked traffic is alternating between eth0 and eth1. Is this suppose to be normal ?
Is it possible to do without adding any kind of default route in the main ? What kinds of problem can be faced if i have too much un-marked traffic ??
Waiting for your help!
Thanks again!
Ok. I’ll start with the questions in order:
If eth0 and eth1 are not typos and you are referring to external bound traffic, then ‘No’. It is not normal that external bound traffic be alternated between an internal lan dev (which should not happen as eth1 and eth2 should be the only devices with a gateway) and an isp. What could be causing it? Do not know without seeing a lot more detail as it is not a problem I have encountered so far.
No default route in main? Yes. But. There still has to be a default route in the main table just make it a dummy route though as seen at: http://www.clintoneast.com/articles/multihomed.php
ifconfig dummy0 1.2.3.4ip route add default dev dummy0
To much un-marked traffic? No problem. It will just be redirected to the default route in the main table.
Ok. What you want sounds very similar to what I have here just using ip route nexthops for load balancing instead of iptables though.
I am going to suggest (the process I went through) that you start off with the routes as separated as possible first and use iptables for debugging your traffic. For example, when I was testing nexthop balancing I used (in addition to table 1 & 2) fwmark 3 / table 3 and gave it a nexthop default route. Then in iptables just tested for any NEW un-MARKed traffic and gave it mark 3. ie:
Somewhere just after testing for traffic that I want sent to a specific table. (This is also just for forwarded traffic. For traffic from this server use the OUTPUT chain).
After getting that in place I test it out to make sure traffic is going out and getting returned and check the packet counts in iptables:
You should see the pkts column increase each time a request is forwarded.
If you then use the iptables balancing rules ala ‘balance1′ chain in the post, you can then do
to see the number of packets being directed to each interface.
That help at all? When I was trying to work out where packets were going I went all the way to logging all packets between the interfaces in mangle if per interface for just port 80 traffic:
Then monitor syslog as I generate traffic to debug what was going on.
Thanks dude!
Here is a more cleaner explanation now with a simpler set of commands..
eth0 – DHCP server is running on it, serving clients of lan
eth1 – is connected to isp1 with ip 192.168.0.101 and gateway 192.168.0.1
eth2 – is connected to isp2 with ip 192.168.2.133 and gateway 192.168.2.1
ip ru add fwmark 1 table 1
ip route add default via 192.168.0.1 dev eth1 table 1
iptables -t mangle -A PREROUTING -p tcp –dport 80 -j MARK –set-mark 1
iptables -t mangle -A PREROUTING -p tcp –dport 53 -j MARK –set-mark 1
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE
ip r del default main
echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth2/rp_filter
ifconfig dummy0 1.2.3.4
ip route add default dev dummy0
The above scenario does not work. The dummy is not helping. Why a default route is necessory in main ? and secondly how can i mark all the packets that are unmarked ?
Thanks for a quick reply.
In order to get this working, what has to be done with routing tables and what not? I’ve tried http://lartc.org/howto/lartc.rpdb.multiple-links.html which works fine for outbound connections but it screws up my inbound port redirections because sometimes they go in eth1 and come out eth2 which obviously doesn’t work. I’d really like to get your solution working.
Regards