Elvar started with all the correct elements to setup a functioning multrouting gateway. Two connections from two different providers, eth1, eth2. Running on an Ubuntu box with eth0 as internal. Both internet connections working on their own.

But alas. Whenever both eth1 and eth2 were active on the host, outgoing packets just would not go out. I don’t know if incoming packets were being replied to as we were unable to check that.

If just eth1 or eth2 was active than everything traversed ok. But we wanted it to work with both connections.

After a LONG time of diagnosing we noticed there were two default routes in iproute2 (ip’s fudged, but you get the idea):

firewall# ip route show
...
default via 1.1.2.1 dev eth2  metric 100
default via 1.1.3.1 dev eth1  metric 100

My firewall often has two default routes listed on the main table (ppp0, and ppp1) until the cleanup script fixes it. Without any negative side effect. I may just be lucky though.

Upon removing the eth2 line from table main, everything started working correctly. Incoming, outgoing, forwarding, balancing.

I also noticed the output of ifconfig eth2 looked a bit screwed too, but there was not much we could do about that as it was assigned by dhcp.

eth2      Link encap:Ethernet  HWaddr 00:11:22:33:44:55
    inet addr:1.1.2.1  Bcast:255.255.255.255  Mask:255.255.255.0

See it? No not the MAC address. The broadcast address. A quick ipcalc 1.1.2.1/24 gives me a broadcast of 1.1.2.255. But once we removed the eth2 default route line, it all started working again and didn’t get to dig into it to see if the broadcast actually affected it.

So that’s all. Just remember there are a lot more things to go wrong in a multigateway setup, including things outside of iptables.

I cannot remember what site I used as a reference when creating the diagram. The original was an ascii chart though. This is created in dia and exported to pdf too. Both attached for convenience. CC licensed as long as the original chart I copied allows it.

Packet flow examples:

All packets to or from localhost travel down the left side of the chart.

From external destination localhost: PREROUTING -> INPUT -> [Local Process] -> OUTPUT -> POSTROUTING.

From localhost destination external: OUTPUT -> POSTROUTING -> [External Host] -> PREROUTING -> INPUT.

All forwarded packets travel the right side of the chart and travel all three tables coming in and then again going out.

From external dest internal: PREROUTING -> FORWARD -> POSTROUTING.

Internal response to external: PREROUTING -> FORWARD -> POSTROUTING.

and so forth.

This is what caught me for a while. Forwarded packet travel the right side route and get out of (or in to) the network. The response then gets generated and does not start from FORWARD or POSTROUTING, but from PREROUTING again. So all NEW forwarded packets need to be marked in PREROUTING and the mark saved. Not new packets need that mark restored in PREROUTING, and all packets need that mark restored in POSTROUTING. (As demonstrated by my previous post).

Locally generated traffic only sees OUTPUT and POSTROUTING before hitting the network and needs to be marked before hitting POSTROUTING, hence the OUTPUT chain rules in my previous post.

I will accept changes to the chart too if anybody wants modifications made.

iptables routing.dia

iptables routing.pdf

This only balances outgoing traffic as incoming traffic balanced via DNS RR and the firewall just returns the traffic on the interface it arrived on as per the previous post.

On the whole, I prefer the iptables solution. It seems to balance the traffic better. ip route balances outgoing connections based on nexthop of the route to that host is not already in it’s routing cache. While iptables balances traffic by alternate outgoing connections. The only downside I have seen is occasional connection drops to the BlackBerry servers.

After 24 hours of iptables balancing:

ppp0      Link encap:Point-to-Point Protocol
          RX bytes:1186783900 (1.1 GB)  TX bytes:1290603327 (1.2 GB)
ppp1      Link encap:Point-to-Point Protocol
          RX bytes:1109227490 (1.1 GB)  TX bytes:1140565429 (1.1 GB)

This is using inclusion rules for determining balanced traffic. These are the rules that ended up on the production server:

# Load balancing rules (Split 50/50 between fwmark 1/2)
iptables -t mangle -A balance1 -d 192.168.0.0/16      -j RETURN
iptables -t mangle -A balance1 -d 10.0.0.0/8          -j RETURN
iptables -t mangle -A balance1 -m connmark ! --mark 0 -j RETURN
iptables -t mangle -A balance1 -m state --state ESTABLISHED,RELATED -j RETURN
iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 0 -j CONNMARK --set-mark 1
iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 1 -j CONNMARK --set-mark 2

# Check to see if we have already marked a packet
iptables -t mangle -A PREROUTING  -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT      -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark

# Mark incoming connections to return on the interface they came in on
iptables -t mangle -A PREROUTING          -i ppp0                     -m state --state NEW  -j CONNMARK --set-mark 1
iptables -t mangle -A PREROUTING          -i ppp1                     -m state --state NEW  -j CONNMARK --set-mark 2

# New outgoing packets
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  22 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  25 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  80 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport 443 -m state --state NEW  -j balance1
iptables -t mangle -A OUTPUT                       -p tcp --dport  80 -m state --state NEW  -j balance1

# Choose our route and save the mark
iptables -t mangle -A PREROUTING  -m connmark --mark 1 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING  -m connmark --mark 2 -j MARK --set-mark 2
iptables -t mangle -A PREROUTING  -m state --state NEW -m connmark ! --mark 0  -j CONNMARK --save-mark

That’s all of the CONNMARK and MARK related rules I use.

The new outgoing packets section is where I choose what packets should be balanced and accounts for about 95% of our outgoing traffic.

The balance1 chain just has some checks at the beginning to catch further traffic that should not be balanced in case some rule gets messed up.

Of the new outgoing packets rules, the PREROUTE lines are for forwarded traffic and the OUTPUT rule is for traffic generated on that host by a transparent squid proxy.

Hope somebody finds that useful one day.

Running on Ubuntu-9.10 with two internet connections ppp0 and ppp1 both with static IP’s from two different internet providers in Australia (iiNet and Internode).

Preperation:

Extra ip route tables per gateway.

Add tables to /etc/iproute2/rt_tables. Table names and numbers can be anything as long as they are consistent later on.

echo -e "101 connection1\n102 connection2" | sudo tee -a /etc/iproute2/rt_tables

Add routes to the extra rule tables. Copy the local routes from the main table then add the default gateway specific to this connection. Replace the vars at the beginning with your relevant settings.

#!/bin/sh
DEV1=ppp0
IP1=100.0.1.1
GW1=100.0.1.254
TABLE2=connection2
DEV2=ppp1
IP2=100.0.2.1
GW2=100.0.2.254
ip route flush table $TABLE1
ip route flush table $TABLE2
ip route show table main | grep -Ev '(^default|ppp)' | while read ROUTE ; do
    ip route add table $TABLE1 $ROUTE
    ip route add table $TABLE2 $ROUTE
done
ip route add table $TABLE1 $GW1 dev $DEV1 src $IP1
ip route add table $TABLE2 $GW2 dev $DEV2 src $IP2
ip route add table $TABLE1 default via $GW1
ip route add table $TABLE2 default via $GW2

ip route output:

~# ip route show
100.0.1.254 dev ppp0  proto kernel  scope link  src 100.0.1.1
100.0.2.254 dev ppp1  proto kernel  scope link  src 100.0.2.1
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.1
default via 100.0.1.254 dev ppp0

~# ip route show table connection1
100.0.1.254 dev ppp0  proto kernel  scope link  src 100.0.1.1
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.1
default via 100.0.1.254 dev ppp0

~# ip route show table connection2
100.0.2.254 dev ppp1  proto kernel  scope link  src 100.0.2.1
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.1
default via 100.0.2.254 dev ppp1

Add the ip rules:

ip rule add from 100.0.1.1 lookup connection1
ip rule add from 100.0.2.1 lookup connection2
ip rule add fwmark 1 lookup connection1
ip rule add fwmark 2 lookup connection2

Add the iptables rules for SNAT:

iptables -A POSTROUTING -o ppp0 -j SNAT --to-source 100.0.1.1
iptables -A POSTROUTING -o ppp1 -j SNAT --to-source 100.0.2.1

And finally add the rules for marking the connection they should be going out on. The first PREROUTING rule is for packets we forward to be returned via the interface they were received on. The OUTPUT rule is for packets handled on this PC to be returned on the correct interface too. We only want to mark new packets and restore marks on established connections else the packets

-A PREROUTING          -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
-A OUTPUT              -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
-A PREROUTING -i ppp0  -m state --state NEW                 -j CONNMARK --set-mark 1
-A PREROUTING -i ppp1  -m state --state NEW                 -j CONNMARK --set-mark 2
-A PREROUTING -m connmark --mark 1                          -j MARK --set-mark 1
-A PREROUTING -m connmark --mark 2                          -j MARK --set-mark 2
-A PREROUTING -m state --state NEW -m connmark ! --mark 0   -j CONNMARK --save-mark

Selective routing:

To send all outgoing traffic on a specific table:

-A PREROUTING -i eth0 -m state --state NEW -p tcp --dport  80 -j CONNMARK --set-mark 2
-A PREROUTING -i eth0 -m state --state NEW -p tcp --dport 443 -j CONNMARK --set-mark 2

References:

http://www.clintoneast.com/articles/multihomed.php

http://linux-ip.net/html/adv-multi-internet.html

The only bugs I have been hit by was #446146. A Huawei E169 USB here modem only shows up as a memory card reader. Quick install of one of the kernel packages listed fixed this for me though.

A PC that after upgrade was trying to detect the harddrives as part of fakeraid setup and would fail to boot. Booted with the previous kernel, removed dmraid, and rebooted and she’s all good.

And gscan2pdf needed three packages from Jaunty to save as PDF again:

• libmagickcore1_6.4.5.4.dfsg1-1ubuntu3.1_amd64.deb
• libmagickwand1_6.4.5.4.dfsg1-1ubuntu3.1_amd64.deb
• perlmagick_6.4.5.4.dfsg1-1ubuntu3.1_amd64.deb

I am really impressed with the direction that to boot theming is taking, well done Scott and team. Nvidia black redraw issues have gone away too. Sound is working better for me. Easier to select the speakers I want to output to. Havn’t tried Ubuntu One yet as most of my family is already using dropbox. Will probably try it for backing up some app data though.

The nvidia-settings tool would not save my settings for two monitor to xorg.conf as it could not parse it. So I instead just replaced the Screen Section in xorg.conf with the following and the extra monitor is auto detected and expanded to now:

Section "Screen"
    Identifier    "Default Screen"
    DefaultDepth    24
    Option         "TwinView" "1"
    Option         "TwinViewXineramaInfoOrder" "DFP, CRT"
    Option         "metamodes" "DFP: nvidia-auto-select +0+0, CRT: nvidia-auto-select +1280+0"
     SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

I still have to work out how to slow down my mouse further with X as the slowest I can set it to in Gnome is still to fast at times.

And incase anybody like to compare notes, these are the packages I have installed/removed in the first couple of days (some from PPA’s, etc).

Installed:

cowbell
bash-completion
vim
vim-gnome
screen
mc
gwibber
inkscape
chromium
virtualbox-3.0
gvim
fontypython
nautilus-dropbox
conky
wine1.2
cups-pdf
gnome-do
shutter
libnss3-tools
gstm
gscan2pdf
nmap
thewidgetfactory
agave

Removed:

latex-xft-fonts
ttf-thai-tlwg
ttf-kacst
ttf-indic-fonts-core
ttf-lao
ttf-wqy-zenhei
ttf-vlgothic
ttf-unfonts-core
f-spot

I removed the list of fonts to see 1) what would happen, 2) my default font list was cluttered & 3) I can’t understand any of those languages anyway.

There are some areas that Plone really does well such as the content type handling and content workflow. But lack of documentation in other areas has left me pulling my hair out at times, such as ldap user and group integration with openldap and active directory, debugging modules and some new gotcha each time something is updated in a buildout.

Liferay has also had a steep learning curve, but I found it no where as bad as Plone. No messing in the ZMI for setting up ldap auth, changing database backends or setting portal defaults. Just one config file. I have found the speed surprising compared to Plone on the same hardware too. With java webapps I have tried in the past I have been left with the impression of them being bloated, memory hungry and slow. But I am slowly coming around thanks to Alfresco and Liferay.

It took a long time to get the liferay-portal-5.2.2.war running in the ubuntu jaunty tomcat6 environment and it still had a couple of issues so I just went back to running the liferay-timcat6 bundle which does run well without issue. I guess that is the price I pay for avoiding java for so long.

To make it suitable for my needs the interface needed to be cleaned up and I decided to just create a fresh theme which can be seen in the screenshot below. The theme is called Winter Sky and is adapted from MMOZINE, and is available from the Liferay community plugin section.

Other additions over the classic liferay theme are multiple level navigation menus and print styling css targets. Which were just a couple of extras that made things much more usable for my victims…. oh.. target users.

There are still a few visual glitches to fix, but everything seems usable and I will post the link to the Liferay project page once it is approved.

Solution:

Use a separate user with it’s own install of python2.4 and any libraries needed for plone.

These are not complete instructions on how to get plone running again. But more or less just notes to point most people in the right path.

Command history:

echo 'export PATH=$HOME/python/2.4/bin:$PATH' | tee -a ~/.bashrc
source ~/.bashrc
mkdir -p ~/python/2.4
mkdir python-build
cd python-build
wget http://python.org/ftp/python/2.4.6/Python-2.4.6.tar.bz2 http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c9.tar.gz http://effbot.org/media/downloads/Imaging-1.1.6.tar.gz
sudo apt-get install build-essential zlib1g-dev libjpeg62-dev libfreetype6-dev
tar -xvf Python-2.4.6.tar.bz2
cd Python-2.4.6/
./configure --prefix=$HOME/python/2.4
make
make install
python -V
cd ..
tar -xvf setuptools-0.6c9.tar.gz
cd setuptools-0.6c9/
python setup.py build
python setup.py install
cd ..
tar -xvf Imaging-1.1.6.tar.gz
cd Imaging-1.1.6
python setup.py build
python setup.py install
python -c "import PIL"
sudo apt-get install libldap2-dev libsasl2-dev libssl-dev

Notes:

Remove all old eggs.

Add the python install path to ~/.bashrc or the current shell.

install python, easy_install and PIL.

PIL needed to be install manually with `python setup.py install` as easy_install was not installing the PIL.pth files correctly.

Install python-ldap requirements if needed in plone are the last apt-get line.

Rerun bootstrap and buildouts in plone.

When creating the certificate request enter for eg *.example.com in the common name field and you can then use the one certificate on all servers that fall in that domain.

BIG WARNING! This is very dangerous if the keys are not secured properly. As if somebody compromises one key, they have compromised all of the services on all the machines that use the same key! So make sure you really want to do this, have the file permissions set correctly and are prepared for the consequences.

The openldap-server guide does try to take you from wo to go fairly well and the following notes are just extra things that would have been handy to know or were needed to continue.

Configuration

The first ldapmodify commands are add extra DbIndex’s. These are handy to know when you receive errors in syslog such as:

• Apr  3 11:01:18 virt01 slapd[11465]: <= bdb_equality_candidates: (objectClass) not indexed

You can view the current DbIndex values:

• ~$ ldapsearch -xLLL -b cn=config -D cn=admin,cn=config -W olcDatabase={1}hdb olcDbIndex
  Enter LDAP Password:
  dn: olcDatabase={1}hdb,cn=config
  olcDbIndex: objectClass eq
  olcDbIndex: cn eq,pres,sub
  olcDbIndex: uid eq,pres,sub
  olcDbIndex: entryUUID eq
  olcDbIndex: memberUid eq
  olcDbIndex: uniqueMember eq

And index’s can be added using the ldapmodify examples. If you need to delete the entire list and start again these ldapmodify commands will help you:

• dn: olcDatabase={1}hdb,cn=config
  delete: olcDbIndex

The sample of output from my ldap setup above will make an okay base to work with.

I followed the schemas section ok with the exception of a correction of the ldapadd command missing a ‘-W’ arg (prompt for password).

• ldapadd -x -D cn=admin,cn=config -W -f /tmp/ldif_output/cn\=config/cn\=schema/cn\=\{8\}misc.ldif

Populating LDAP

You will now notice with the ldapadd command the reference a second admin principal (cn=admin,dc=example,dc=com). This principal is created with the same password as the first admin (cn=admin,cn=config).

LDAP replication

LDAP replication without TLS was setup within minutes following the guide.

Just be very careful to make sure all modifications are complete and correct before adding the ldif files.

The cn=config replication needs to be applied on both servers, but the dc=example,dc=com replication ldif only needs to be applied on one as once cn=config replication is setup it will replicate the dc=example,dc=com settings to the other server.

The second server also does not need to be setup with all the same changes as the first. Just setup the same domain and password when installing slapd then apply the replication ldif. The additional changes made on ldap01 will be replicated to ldap02 automatically. Once both servers have the ldif applied, then restart slapd on each.

It took a fair while to get replication working again using TLS. Most of my problems were certificate related and couple of missed changes in the TLS replication ldif that would not change back.

TLS and SSL

I created my certificates with CA.pl from an existing CA I have setup for other servers. When creating the certificates make sure that the commonName matches the host name of the server as it will be addressed. Ie:

• Server name: ldap01.example.com
• commonName: ldap01.example.com

TLS verification will fail if you do a ldap search against ldap://ldap01/ (as opposed to ldap://ldap01.example.com/).

I have not yet tested wildcard certificates either. Also the ca-certificate can be referred to either by file (olcTLSCACertificateFile) or the dir containing the cert (olcTLSCACertificatePath). But only referring to the cert by file name worked in my testing even though I added to cert to Ubuntu’s ca-certificate management scheme. Ie:

• sudo cp cacert.pem /usr/share/ca-certificates/Example.Com-CA.crt

• echo 'Example.Com-CA.crt' | sudo tee -a /etc/ca-certificates.conf

• sudo update-ca-certificates

I had to set the SLAPD_SERVICES line to: SLAPD_SERVICES=”ldap:/// ldaps:/// ldapi:///” To allow TLS ldap:// connections from other hosts.

The TLS connections are over ldap:// and SSL is over ldaps://. TLS is the newer and preferred standard, but the ldap:// protocol also allows cleartext communication as a fallback.

To test a TLS connection use:

• ldapsearch -xLLL -b cn=config -D cn=admin,cn=config -W -H ldap://ldap01.example.com/ -ZZ

• -H ldap://ldap01.example.com/ – specifies the hostname. Make sure it is the same as commonName in the certificate used.
• -ZZ – Issues a StartTLS command and requires it succed. Using just -Z issues StartTLS but still allows a cleartext connection if the StartTLS directive fails.

To debug a connection append -d5 to the command. It will spew a lot of output, but in there will most likely be the reason for the cause of the error.

The notes at the end of the TLS replication section are also handy for getting a TLS client connection working. I needed the following line in /etc/ldap/ldap.conf on all hosts:

• TLS_CACERT /etc/ssl/certs/Example.Com-CA.crt

Or if you do not wish to check the validity of certificates then you could use:

• TLS_REQCERT allow

TLS Replication

This should only be attempted once both hosts are confirmed working over TLS. Also make sure you test establishing a TLS connection from one host to the other (ie, running the ldapsearch from ldap01 using -ZZ -H ldap://ldap02.example.com/).

After adding the cn=config TLS change to one host (remember it will be replicated to the other host automatically), then restart both slapd backends (sudo /etc/init.d/slapd restart).

If that is working without any errors in syslog then proceed to the dc=example,dc=com tls replication and restart both backends again.

LDAP Authentication

My next headache was with the auth-client-config command. The correct command to use is:

• sudo auth-client-config -t nss -p lac_ldap

Some sites also note another command that is actually unnecessary:

• sudo pam-auth-update ldap

pam-auth-update allows you to pick which auth methods will be used in your system, but ldap is automatically added when you install libnss-ldap.

Other changes I made to /etc/ldap.conf was uncommenting two lines:

• pam_filter objectclass=account

• ssl start_tls

This requires a user have the objectclass=account attribute to be able to login. And the second requires a TLS connection to the ldap server.

To use TLS for pam auth you will need to make sure the cert is specified in /etc/ldap/ldap.conf just as on the server. This can be tested by running the same ldapsearch command on the client as on the server.

After managing to get that working on one client the rest were managed easily using puppet.

User and Group Management

The command to put the ‘secret’ in /etc/ldapscripts/ldapscripts.passwd will not work as expected if you have a dollar symbol in the password. Ie:

• sudo sh -c "echo -n '$ecret' > /etc/ldapscripts/ldapscripts.passwd"

Will result in an empty passwd file. The correct way to echo the password is to escape the $. Ie: ‘\$ecret’. This is not a problem with the server guide, just an oversight that caught me out. It can be seen in the log file as:

• >> 04/01/09 - 17:23 : Command : /usr/sbin/ldapadduser george example
  ldap_bind: Server is unwilling to perform (53)
          additional info: unauthenticated bind (DN with no password) disallowed
  ldap_bind: Server is unwilling to perform (53)
          additional info: unauthenticated bind (DN with no password) disallowed
  Error adding user george to LDAP

The ldapscripts will also often fail if replication is not working either. Check the output of /var/log/ldapscripts.log.

• >> 04/02/09 - 13:49 : Command : /usr/sbin/ldapadduser george example
  Additional information: value does not conform to assertion syntax
  ldap_modify: Server is unwilling to perform (53)
          additional info: shadow context; no update referral
  Error adding user george to LDAP

That is the ldapscripts log output I get when replication was not working.

Finale

After that was all sorted out I was happy as Larry. Next thing to try is SASL authentication i guess. If anybody wants to point me in the right direction to assist with the Jaunty server guide it would be much appreciated. As well as any more notes and corrections to my post.

I purchased one from Dick Smiths last night plugged it into my (Ubuntu-8.10 powered) laptop and was up and running within seconds.

Signal strength and speed are fairly industry standard. Cost AU$50. I probably could have found it cheaper elseware but DSE was conveniently on my way home last night.

D-Link DWA-110

Now just to find an access point with decent signal strength.