i have one machine installed unbuntu 9.10 and squid 2.74 with 4 Ethernet cards.

eth0 ……ISP-1
eth1…….LAN-1
eth2…….ISP-2
eth3…….LAN2

( LAN1 : 192.168.0.1 )===> 20 Work Stations
( LAN2 : 192.168.2.1 )===> 10 Work Stations

i just need to make it possible that LAN 1 will use the internet from WAN-1 and LAN-2 will use the Internet From WAN-2 ..

although i want that if WAN-1 fails then only LAN1 should be routed to WAN-2… WAN-2 is pretty fine it does not go down mostly… Please Help me step by step i will be very thank full to you

1. In the rules under ‘# Choose our route and save the mark’, don’t use those rules because they just copy the connection mark to the packet mark, then copy that back to the connection mark again. Replace them with these, and that also allows you to remove the ‘# Check to see if we have already marked a packet’ section because these rules do the same thing:

# Choose our route and save the mark (copies the connection mark to the packet mark)
iptables -t mangle -A PREROUTING -m connmark ! –mark 0 -j CONNMARK –restore-mark
iptables -t mangle -A OUTPUT -m connmark ! –mark 0 -j CONNMARK –restore-mark

2. I found the same issue as the commenter above – with no default route in the ‘main’ routing table, a lot of things on the local machine didn’t work. I tried the dummy route as above, that didn’t work either. When I looked, it was sending packets out to the LAN with a source address of the dummy interface! I think this might be specific to my setup – I’m balancing outgoing connections between 2 gateways on the same interface / same LAN.

Here’s my full setup, where the LAN is 172.26.16.0/24 and the gateways are 172.26.16.1 and 172.26.16.2:

IP TABLES SCRIPT:
Note – this does not include, but should work with, any NAT/masquerading rules you have.

# flush existing rules:
iptables -t mangle -F OUTPUT
iptables -t mangle -F PREROUTING

# Load balancing rules (Split 50/50 between fwmark 1/2)
iptables -t mangle -F balance1 >/dev/null 2>&1  # flush balance1 if exists
iptables -t mangle -X balance1 >/dev/null 2>&1  # delete balance1 if exists
iptables -t mangle -N balance1
iptables -t mangle -A balance1 -d 172.26.16.0/24 -j RETURN
iptables -t mangle -A balance1 -m connmark ! --mark 0 -j RETURN
iptables -t mangle -A balance1 -m state --state ESTABLISHED,RELATED -j RETURN
iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 0 -j CONNMARK --set-mark 1
iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 1 -j CONNMARK --set-mark 2

# New outgoing connections
iptables -t mangle -A PREROUTING -s 172.26.16.0/24 -m state --state NEW -j balance1
iptables -t mangle -A OUTPUT -m state --state NEW -j balance1

# Choose our route and save the mark
iptables -t mangle -A PREROUTING -m connmark ! --mark 0 -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -m connmark ! --mark 0 -j CONNMARK --restore-mark

IP ROUTING/RULES SCRIPT:

ip route del default table main
ip route del default table 1
ip route del 172.26.16.0/24 dev eth0 table 1
ip route del 172.26.16.0/24 dev eth0 table 2
ip route del default table 2
ip rule del fwmark 1
ip rule del fwmark 2
ip rule add fwmark 1 table 1
ip rule add fwmark 2 table 2
ip route add default via 172.26.16.1 table 1
ip route add default via 172.26.16.3 table 2
ip route add default via 172.26.16.1 table main
ip route flush cache

Hope that helps someone, and thanks for this article agentk – it was a massive help!

Regards

Here is a more cleaner explanation now with a simpler set of commands..

eth0 – DHCP server is running on it, serving clients of lan
eth1 – is connected to isp1 with ip 192.168.0.101 and gateway 192.168.0.1
eth2 – is connected to isp2 with ip 192.168.2.133 and gateway 192.168.2.1

ip ru add fwmark 1 table 1
ip route add default via 192.168.0.1 dev eth1 table 1

iptables -t mangle -A PREROUTING -p tcp –dport 80 -j MARK –set-mark 1
iptables -t mangle -A PREROUTING -p tcp –dport 53 -j MARK –set-mark 1

iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE

ip r del default main

echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth2/rp_filter

ifconfig dummy0 1.2.3.4
ip route add default dev dummy0

The above scenario does not work. The dummy is not helping. Why a default route is necessory in main ? and secondly how can i mark all the packets that are unmarked ?

Thanks for a quick reply.

If eth0 and eth1 are not typos and you are referring to external bound traffic, then ‘No’. It is not normal that external bound traffic be alternated between an internal lan dev (which should not happen as eth1 and eth2 should be the only devices with a gateway) and an isp. What could be causing it? Do not know without seeing a lot more detail as it is not a problem I have encountered so far.

No default route in main? Yes. But. There still has to be a default route in the main table just make it a dummy route though as seen at: http://www.clintoneast.com/articles/multihomed.php
ifconfig dummy0 1.2.3.4
ip route add default dev dummy0

To much un-marked traffic? No problem. It will just be redirected to the default route in the main table.

Ok. What you want sounds very similar to what I have here just using ip route nexthops for load balancing instead of iptables though.
I am going to suggest (the process I went through) that you start off with the routes as separated as possible first and use iptables for debugging your traffic. For example, when I was testing nexthop balancing I used (in addition to table 1 & 2) fwmark 3 / table 3 and gave it a nexthop default route. Then in iptables just tested for any NEW un-MARKed traffic and gave it mark 3. ie:

iptables -t mangle -A PREROUTING  -m state --state NEW -m connmark ! --mark 0  -j CONNMARK --set-mark 3

Somewhere just after testing for traffic that I want sent to a specific table. (This is also just for forwarded traffic. For traffic from this server use the OUTPUT chain).

After getting that in place I test it out to make sure traffic is going out and getting returned and check the packet counts in iptables:

iptables -t mangle -L PREROUTING -nv

You should see the pkts column increase each time a request is forwarded.

If you then use the iptables balancing rules ala ‘balance1′ chain in the post, you can then do

iptables -t mangle -L balance1 -nv

to see the number of packets being directed to each interface.

That help at all? When I was trying to work out where packets were going I went all the way to logging all packets between the interfaces in mangle if per interface for just port 80 traffic:

-A PREROUTING  -s EXTERNALIP -p tcp -m tcp --dport 80 -j LOG --log-prefix "[d/PREROUTING  dport]: "
-A POSTROUTING -s EXTERNALIP -p tcp -m tcp --dport 80 -j LOG --log-prefix "[d/POSTROUTING dport]: "
-A INPUT       -s EXTERNALIP -p tcp -m tcp --dport 80 -j LOG --log-prefix "[d/INPUT       dport]: "
-A OUTPUT      -s EXTERNALIP -p tcp -m tcp --dport 80 -j LOG --log-prefix "[d/OUTPUT      dport]: "
-A FORWARD     -s EXTERNALIP -p tcp -m tcp --dport 80 -j LOG --log-prefix "[d/FORWARD     dport]: "

-A PREROUTING  -d EXTERNALIP -p tcp -m tcp --sport 80 -j LOG --log-prefix "[d/PREROUTING  sport]: "
-A POSTROUTING -d EXTERNALIP -p tcp -m tcp --sport 80 -j LOG --log-prefix "[d/POSTROUTING sport]: "
-A INPUT       -d EXTERNALIP -p tcp -m tcp --sport 80 -j LOG --log-prefix "[d/INPUT       sport]: "
-A OUTPUT      -d EXTERNALIP -p tcp -m tcp --sport 80 -j LOG --log-prefix "[d/OUTPUT      sport]: "
-A FORWARD     -d EXTERNALIP -p tcp -m tcp --sport 80 -j LOG --log-prefix "[d/FORWARD     sport]: "

Then monitor syslog as I generate traffic to debug what was going on.

Thank you for such a wonderful article/guide.

I have started working on load balancing and i am trying to achieve something similar to what you have described here but i would like to mark my traffic (using iptables and dport) so that i can have more control. Here is the scenario:
eth0 – static adress to internal lan
eth1 – isp 1
eth2 – isp 2

1 – http, dns, ftp, ssl, ssh are marked 1 (based on destination port)
2 – chat and voip are marked 2
3 – ip ru add fwmark 1 table 1
ip ru add fwmark 2 table 2
4 – ip route add default via eth1 table 1
ip route add default via eth2 table 2
5 – add multiple default routes in main. ( . . . nexthop via eth1 nexthop via eth2)

The problem i am facing is that any unmarked traffic is alternating between eth0 and eth1. Is this suppose to be normal ?
Is it possible to do without adding any kind of default route in the main ? What kinds of problem can be faced if i have too much un-marked traffic ??

Waiting for your help!

Thanks again!