Iptables packet flow diagram
Here is the diagram I keep stuck to my wall for reference when dealing with iptables.
I cannot remember what site I used as a reference when creating the diagram. The original was an ascii chart though. This is created in dia and exported to pdf too. Both attached for convenience. CC licensed as long as the original chart I copied allows it.
Packet flow examples:
All packets to or from localhost travel down the left side of the chart.
From external destination localhost: PREROUTING -> INPUT -> [Local Process] -> OUTPUT -> POSTROUTING.
From localhost destination external: OUTPUT -> POSTROUTING -> [External Host] -> PREROUTING -> INPUT.
All forwarded packets travel the right side of the chart and travel all three tables coming in and then again going out.
From external dest internal: PREROUTING -> FORWARD -> POSTROUTING.
Internal response to external: PREROUTING -> FORWARD -> POSTROUTING.
and so forth.
This is what caught me for a while. Forwarded packet travel the right side route and get out of (or in to) the network. The response then gets generated and does not start from FORWARD or POSTROUTING, but from PREROUTING again. So all NEW forwarded packets need to be marked in PREROUTING and the mark saved. Not new packets need that mark restored in PREROUTING, and all packets need that mark restored in POSTROUTING. (As demonstrated by my previous post).
Locally generated traffic only sees OUTPUT and POSTROUTING before hitting the network and needs to be marked before hitting POSTROUTING, hence the OUTPUT chain rules in my previous post.
I will accept changes to the chart too if anybody wants modifications made.
Multi gateway balancing with iptables
I have been testing load balancing via both iptables and ip route nexthop for a couple of days now. They both work pretty well too.
This only balances outgoing traffic as incoming traffic balanced via DNS RR and the firewall just returns the traffic on the interface it arrived on as per the previous post.
On the whole, I prefer the iptables solution. It seems to balance the traffic better. ip route balances outgoing connections based on nexthop of the route to that host is not already in it’s routing cache. While iptables balances traffic by alternate outgoing connections. The only downside I have seen is occasional connection drops to the BlackBerry servers.
After 24 hours of iptables balancing:
ppp0 Link encap:Point-to-Point Protocol RX bytes:1186783900 (1.1 GB) TX bytes:1290603327 (1.2 GB) ppp1 Link encap:Point-to-Point Protocol RX bytes:1109227490 (1.1 GB) TX bytes:1140565429 (1.1 GB)
This is using inclusion rules for determining balanced traffic. These are the rules that ended up on the production server:
# Load balancing rules (Split 50/50 between fwmark 1/2) iptables -t mangle -A balance1 -d 192.168.0.0/16 -j RETURN iptables -t mangle -A balance1 -d 10.0.0.0/8 -j RETURN iptables -t mangle -A balance1 -m connmark ! --mark 0 -j RETURN iptables -t mangle -A balance1 -m state --state ESTABLISHED,RELATED -j RETURN iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 0 -j CONNMARK --set-mark 1 iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 1 -j CONNMARK --set-mark 2 # Check to see if we have already marked a packet iptables -t mangle -A PREROUTING -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark iptables -t mangle -A OUTPUT -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark # Mark incoming connections to return on the interface they came in on iptables -t mangle -A PREROUTING -i ppp0 -m state --state NEW -j CONNMARK --set-mark 1 iptables -t mangle -A PREROUTING -i ppp1 -m state --state NEW -j CONNMARK --set-mark 2 # New outgoing packets iptables -t mangle -A PREROUTING -i eth0 -p tcp --dport 22 -m state --state NEW -j balance1 iptables -t mangle -A PREROUTING -i eth0 -p tcp --dport 25 -m state --state NEW -j balance1 iptables -t mangle -A PREROUTING -i eth0 -p tcp --dport 80 -m state --state NEW -j balance1 iptables -t mangle -A PREROUTING -i eth0 -p tcp --dport 443 -m state --state NEW -j balance1 iptables -t mangle -A OUTPUT -p tcp --dport 80 -m state --state NEW -j balance1 # Choose our route and save the mark iptables -t mangle -A PREROUTING -m connmark --mark 1 -j MARK --set-mark 1 iptables -t mangle -A PREROUTING -m connmark --mark 2 -j MARK --set-mark 2 iptables -t mangle -A PREROUTING -m state --state NEW -m connmark ! --mark 0 -j CONNMARK --save-mark
That’s all of the CONNMARK and MARK related rules I use.
The new outgoing packets section is where I choose what packets should be balanced and accounts for about 95% of our outgoing traffic.
The balance1 chain just has some checks at the beginning to catch further traffic that should not be balanced in case some rule gets messed up.
Of the new outgoing packets rules, the PREROUTE lines are for forwarded traffic and the OUTPUT rule is for traffic generated on that host by a transparent squid proxy.
Hope somebody finds that useful one day.
Multi gateway routing with iptables and iproute2
Notes on multi gateway routing with iptables and iproute2, suggestions and corrections gladly accepted. My notes may be incomplete or just plain wrong, I pieced them together after getting it working.
Running on Ubuntu-9.10 with two internet connections ppp0 and ppp1 both with static IP’s from two different internet providers in Australia (iiNet and Internode).
Preperation:
Extra ip route tables per gateway.
Add tables to /etc/iproute2/rt_tables. Table names and numbers can be anything as long as they are consistent later on.
echo -e "101 connection1\n102 connection2" | sudo tee -a /etc/iproute2/rt_tables
Add routes to the extra rule tables. Copy the local routes from the main table then add the default gateway specific to this connection. Replace the vars at the beginning with your relevant settings.
#!/bin/sh
DEV1=ppp0
IP1=100.0.1.1
GW1=100.0.1.254
TABLE2=connection2
DEV2=ppp1
IP2=100.0.2.1
GW2=100.0.2.254
ip route flush table $TABLE1
ip route flush table $TABLE2
ip route show table main | grep -Ev '(^default|ppp)' | while read ROUTE ; do
ip route add table $TABLE1 $ROUTE
ip route add table $TABLE2 $ROUTE
done
ip route add table $TABLE1 $GW1 dev $DEV1 src $IP1
ip route add table $TABLE2 $GW2 dev $DEV2 src $IP2
ip route add table $TABLE1 default via $GW1
ip route add table $TABLE2 default via $GW2
ip route output:
~# ip route show 100.0.1.254 dev ppp0 proto kernel scope link src 100.0.1.1 100.0.2.254 dev ppp1 proto kernel scope link src 100.0.2.1 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.1 default via 100.0.1.254 dev ppp0
~# ip route show table connection1 100.0.1.254 dev ppp0 proto kernel scope link src 100.0.1.1 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.1 default via 100.0.1.254 dev ppp0 ~# ip route show table connection2 100.0.2.254 dev ppp1 proto kernel scope link src 100.0.2.1 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.1 default via 100.0.2.254 dev ppp1
Add the ip rules:
ip rule add from 100.0.1.1 lookup connection1 ip rule add from 100.0.2.1 lookup connection2 ip rule add fwmark 1 lookup connection1 ip rule add fwmark 2 lookup connection2
Add the iptables rules for SNAT:
iptables -A POSTROUTING -o ppp0 -j SNAT --to-source 100.0.1.1 iptables -A POSTROUTING -o ppp1 -j SNAT --to-source 100.0.2.1
And finally add the rules for marking the connection they should be going out on. The first PREROUTING rule is for packets we forward to be returned via the interface they were received on. The OUTPUT rule is for packets handled on this PC to be returned on the correct interface too. We only want to mark new packets and restore marks on established connections else the packets
-A PREROUTING -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark -A OUTPUT -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark -A PREROUTING -i ppp0 -m state --state NEW -j CONNMARK --set-mark 1 -A PREROUTING -i ppp1 -m state --state NEW -j CONNMARK --set-mark 2 -A PREROUTING -m connmark --mark 1 -j MARK --set-mark 1 -A PREROUTING -m connmark --mark 2 -j MARK --set-mark 2 -A PREROUTING -m state --state NEW -m connmark ! --mark 0 -j CONNMARK --save-mark
Selective routing:
To send all outgoing traffic on a specific table:
-A PREROUTING -i eth0 -m state --state NEW -p tcp --dport 80 -j CONNMARK --set-mark 2 -A PREROUTING -i eth0 -m state --state NEW -p tcp --dport 443 -j CONNMARK --set-mark 2
References:
PostBooks install tutorial for Ubuntu Linux 7.10
PostBooks is an excellent cross-platform commercial open source ERP / accounting software suite. These are my old instructions from installing PostBooks in Ubuntu-7.10.
This document will give you setup instructions for installing PostBooks from scratch. Ie, no previous install of postgresql or PostBooks on the system. You will need to read between the lines if your install does not fit that description. (more…)
agentk
