Elvar started with all the correct elements to setup a functioning multrouting gateway. Two connections from two different providers, eth1, eth2. Running on an Ubuntu box with eth0 as internal. Both internet connections working on their own.

But alas. Whenever both eth1 and eth2 were active on the host, outgoing packets just would not go out. I don’t know if incoming packets were being replied to as we were unable to check that.

If just eth1 or eth2 was active than everything traversed ok. But we wanted it to work with both connections.

After a LONG time of diagnosing we noticed there were two default routes in iproute2 (ip’s fudged, but you get the idea):

firewall# ip route show
...
default via 1.1.2.1 dev eth2  metric 100
default via 1.1.3.1 dev eth1  metric 100

My firewall often has two default routes listed on the main table (ppp0, and ppp1) until the cleanup script fixes it. Without any negative side effect. I may just be lucky though.

Upon removing the eth2 line from table main, everything started working correctly. Incoming, outgoing, forwarding, balancing.

I also noticed the output of ifconfig eth2 looked a bit screwed too, but there was not much we could do about that as it was assigned by dhcp.

eth2      Link encap:Ethernet  HWaddr 00:11:22:33:44:55
    inet addr:1.1.2.1  Bcast:255.255.255.255  Mask:255.255.255.0

See it? No not the MAC address. The broadcast address. A quick ipcalc 1.1.2.1/24 gives me a broadcast of 1.1.2.255. But once we removed the eth2 default route line, it all started working again and didn’t get to dig into it to see if the broadcast actually affected it.

So that’s all. Just remember there are a lot more things to go wrong in a multigateway setup, including things outside of iptables.

I cannot remember what site I used as a reference when creating the diagram. The original was an ascii chart though. This is created in dia and exported to pdf too. Both attached for convenience. CC licensed as long as the original chart I copied allows it.

Packet flow examples:

All packets to or from localhost travel down the left side of the chart.

From external destination localhost: PREROUTING -> INPUT -> [Local Process] -> OUTPUT -> POSTROUTING.

From localhost destination external: OUTPUT -> POSTROUTING -> [External Host] -> PREROUTING -> INPUT.

All forwarded packets travel the right side of the chart and travel all three tables coming in and then again going out.

From external dest internal: PREROUTING -> FORWARD -> POSTROUTING.

Internal response to external: PREROUTING -> FORWARD -> POSTROUTING.

and so forth.

This is what caught me for a while. Forwarded packet travel the right side route and get out of (or in to) the network. The response then gets generated and does not start from FORWARD or POSTROUTING, but from PREROUTING again. So all NEW forwarded packets need to be marked in PREROUTING and the mark saved. Not new packets need that mark restored in PREROUTING, and all packets need that mark restored in POSTROUTING. (As demonstrated by my previous post).

Locally generated traffic only sees OUTPUT and POSTROUTING before hitting the network and needs to be marked before hitting POSTROUTING, hence the OUTPUT chain rules in my previous post.

I will accept changes to the chart too if anybody wants modifications made.

iptables routing.dia

iptables routing.pdf

This only balances outgoing traffic as incoming traffic balanced via DNS RR and the firewall just returns the traffic on the interface it arrived on as per the previous post.

On the whole, I prefer the iptables solution. It seems to balance the traffic better. ip route balances outgoing connections based on nexthop of the route to that host is not already in it’s routing cache. While iptables balances traffic by alternate outgoing connections. The only downside I have seen is occasional connection drops to the BlackBerry servers.

After 24 hours of iptables balancing:

ppp0      Link encap:Point-to-Point Protocol
          RX bytes:1186783900 (1.1 GB)  TX bytes:1290603327 (1.2 GB)
ppp1      Link encap:Point-to-Point Protocol
          RX bytes:1109227490 (1.1 GB)  TX bytes:1140565429 (1.1 GB)

This is using inclusion rules for determining balanced traffic. These are the rules that ended up on the production server:

# Load balancing rules (Split 50/50 between fwmark 1/2)
iptables -t mangle -A balance1 -d 192.168.0.0/16      -j RETURN
iptables -t mangle -A balance1 -d 10.0.0.0/8          -j RETURN
iptables -t mangle -A balance1 -m connmark ! --mark 0 -j RETURN
iptables -t mangle -A balance1 -m state --state ESTABLISHED,RELATED -j RETURN
iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 0 -j CONNMARK --set-mark 1
iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 1 -j CONNMARK --set-mark 2

# Check to see if we have already marked a packet
iptables -t mangle -A PREROUTING  -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT      -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark

# Mark incoming connections to return on the interface they came in on
iptables -t mangle -A PREROUTING          -i ppp0                     -m state --state NEW  -j CONNMARK --set-mark 1
iptables -t mangle -A PREROUTING          -i ppp1                     -m state --state NEW  -j CONNMARK --set-mark 2

# New outgoing packets
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  22 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  25 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  80 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport 443 -m state --state NEW  -j balance1
iptables -t mangle -A OUTPUT                       -p tcp --dport  80 -m state --state NEW  -j balance1

# Choose our route and save the mark
iptables -t mangle -A PREROUTING  -m connmark --mark 1 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING  -m connmark --mark 2 -j MARK --set-mark 2
iptables -t mangle -A PREROUTING  -m state --state NEW -m connmark ! --mark 0  -j CONNMARK --save-mark

That’s all of the CONNMARK and MARK related rules I use.

The new outgoing packets section is where I choose what packets should be balanced and accounts for about 95% of our outgoing traffic.

The balance1 chain just has some checks at the beginning to catch further traffic that should not be balanced in case some rule gets messed up.

Of the new outgoing packets rules, the PREROUTE lines are for forwarded traffic and the OUTPUT rule is for traffic generated on that host by a transparent squid proxy.

Hope somebody finds that useful one day.

Running on Ubuntu-9.10 with two internet connections ppp0 and ppp1 both with static IP’s from two different internet providers in Australia (iiNet and Internode).

Preperation:

Extra ip route tables per gateway.

Add tables to /etc/iproute2/rt_tables. Table names and numbers can be anything as long as they are consistent later on.

echo -e "101 connection1\n102 connection2" | sudo tee -a /etc/iproute2/rt_tables

Add routes to the extra rule tables. Copy the local routes from the main table then add the default gateway specific to this connection. Replace the vars at the beginning with your relevant settings.

#!/bin/sh
DEV1=ppp0
IP1=100.0.1.1
GW1=100.0.1.254
TABLE2=connection2
DEV2=ppp1
IP2=100.0.2.1
GW2=100.0.2.254
ip route flush table $TABLE1
ip route flush table $TABLE2
ip route show table main | grep -Ev '(^default|ppp)' | while read ROUTE ; do
    ip route add table $TABLE1 $ROUTE
    ip route add table $TABLE2 $ROUTE
done
ip route add table $TABLE1 $GW1 dev $DEV1 src $IP1
ip route add table $TABLE2 $GW2 dev $DEV2 src $IP2
ip route add table $TABLE1 default via $GW1
ip route add table $TABLE2 default via $GW2

ip route output:

~# ip route show
100.0.1.254 dev ppp0  proto kernel  scope link  src 100.0.1.1
100.0.2.254 dev ppp1  proto kernel  scope link  src 100.0.2.1
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.1
default via 100.0.1.254 dev ppp0

~# ip route show table connection1
100.0.1.254 dev ppp0  proto kernel  scope link  src 100.0.1.1
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.1
default via 100.0.1.254 dev ppp0

~# ip route show table connection2
100.0.2.254 dev ppp1  proto kernel  scope link  src 100.0.2.1
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.1
default via 100.0.2.254 dev ppp1

Add the ip rules:

ip rule add from 100.0.1.1 lookup connection1
ip rule add from 100.0.2.1 lookup connection2
ip rule add fwmark 1 lookup connection1
ip rule add fwmark 2 lookup connection2

Add the iptables rules for SNAT:

iptables -A POSTROUTING -o ppp0 -j SNAT --to-source 100.0.1.1
iptables -A POSTROUTING -o ppp1 -j SNAT --to-source 100.0.2.1

And finally add the rules for marking the connection they should be going out on. The first PREROUTING rule is for packets we forward to be returned via the interface they were received on. The OUTPUT rule is for packets handled on this PC to be returned on the correct interface too. We only want to mark new packets and restore marks on established connections else the packets

-A PREROUTING          -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
-A OUTPUT              -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
-A PREROUTING -i ppp0  -m state --state NEW                 -j CONNMARK --set-mark 1
-A PREROUTING -i ppp1  -m state --state NEW                 -j CONNMARK --set-mark 2
-A PREROUTING -m connmark --mark 1                          -j MARK --set-mark 1
-A PREROUTING -m connmark --mark 2                          -j MARK --set-mark 2
-A PREROUTING -m state --state NEW -m connmark ! --mark 0   -j CONNMARK --save-mark

Selective routing:

To send all outgoing traffic on a specific table:

-A PREROUTING -i eth0 -m state --state NEW -p tcp --dport  80 -j CONNMARK --set-mark 2
-A PREROUTING -i eth0 -m state --state NEW -p tcp --dport 443 -j CONNMARK --set-mark 2

References:

http://www.clintoneast.com/articles/multihomed.php

http://linux-ip.net/html/adv-multi-internet.html

The only bugs I have been hit by was #446146. A Huawei E169 USB here modem only shows up as a memory card reader. Quick install of one of the kernel packages listed fixed this for me though.

A PC that after upgrade was trying to detect the harddrives as part of fakeraid setup and would fail to boot. Booted with the previous kernel, removed dmraid, and rebooted and she’s all good.

And gscan2pdf needed three packages from Jaunty to save as PDF again:

• libmagickcore1_6.4.5.4.dfsg1-1ubuntu3.1_amd64.deb
• libmagickwand1_6.4.5.4.dfsg1-1ubuntu3.1_amd64.deb
• perlmagick_6.4.5.4.dfsg1-1ubuntu3.1_amd64.deb

I am really impressed with the direction that to boot theming is taking, well done Scott and team. Nvidia black redraw issues have gone away too. Sound is working better for me. Easier to select the speakers I want to output to. Havn’t tried Ubuntu One yet as most of my family is already using dropbox. Will probably try it for backing up some app data though.

The nvidia-settings tool would not save my settings for two monitor to xorg.conf as it could not parse it. So I instead just replaced the Screen Section in xorg.conf with the following and the extra monitor is auto detected and expanded to now:

Section "Screen"
    Identifier    "Default Screen"
    DefaultDepth    24
    Option         "TwinView" "1"
    Option         "TwinViewXineramaInfoOrder" "DFP, CRT"
    Option         "metamodes" "DFP: nvidia-auto-select +0+0, CRT: nvidia-auto-select +1280+0"
     SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

I still have to work out how to slow down my mouse further with X as the slowest I can set it to in Gnome is still to fast at times.

And incase anybody like to compare notes, these are the packages I have installed/removed in the first couple of days (some from PPA’s, etc).

Installed:

cowbell
bash-completion
vim
vim-gnome
screen
mc
gwibber
inkscape
chromium
virtualbox-3.0
gvim
fontypython
nautilus-dropbox
conky
wine1.2
cups-pdf
gnome-do
shutter
libnss3-tools
gstm
gscan2pdf
nmap
thewidgetfactory
agave

Removed:

latex-xft-fonts
ttf-thai-tlwg
ttf-kacst
ttf-indic-fonts-core
ttf-lao
ttf-wqy-zenhei
ttf-vlgothic
ttf-unfonts-core
f-spot

I removed the list of fonts to see 1) what would happen, 2) my default font list was cluttered & 3) I can’t understand any of those languages anyway.

WinterSky ver-11

Theme now looks correct in IE7 and IE8.
IE6 is functional, only sub color schemes (to do with selectors in the color scheme css), and png images need work.

Added extra option in templates/init_custom.vm:
$custom_places = true ## will replace the communities/my places menu with a simplified view hiding any public community view from non community owners. (Beta feature added to simplify the my places menu for sites that only use private pages)

Head tag also employs google chrome frame if enabled for better rendering in IE6.

Update can be installed now via the software catalog directly in Liferay.

http://www.cyrius.com/debian/orion/d-link/dns-323/install.html

The install took quite a while, I selected the standard package list. Then because it gets quit hot here in Griffith I really wanted to have fan control back, so instruction here were also followed.

As I wanted to install to an 8GB Flash drive I had handy I first uploaded the netboot.img firmware, turned off the DNS-323, removed the hard drives and left only the flash drive inserted, then powered up and followed the installation instructions.

I like keeping root on the flash drive as the hard drives can be stopped all day when not in use to save temp and power.

Fans are also controlled by lm-sensors and the unit seems to run about 47degC most of the day.

I use the unit mostly for irc (screen/weechat), rsync backup target and samba media server. It handles all that pretty well but I do notice it bog a fair bit with only 64mb ram and hard drive transfer rates are not as good as I am used to, but on the whole a great little unit (now that it runs debian).

Life Portal 5.2 Systems Development

• Author: Jonas X. Yuan
• Pages: 552
• Publisher: Packt Publishing
• Rating: 9/10
• ISBN: 1847194702
• Summary: A Java developer’s guide to developing custom intranet systems, based on Liferay’s platform

As one of only a few books on advanced Liferay Portal development Jonas had a LOT of material to cover. My two weeks of experience with Liferay up to this point had only involved deployment, configuration and theme development. After been impressed with the results so far my next three aims were customising the WYSIWYG editor with styles more specific to the site, adding layout templates to support quick deployment of content and some custom portals for integrating our other information management systems. A week later with the book and I have the first two knocked over and am on my way with the portals.

I have no experience with Java programming, but luckily there was enough detail to get me started. Because of the lack of knowledge I started right from the beginning and read thoroughly all of the background information on Liferay Portal and the JSR-286 specs. This was quite a laborious experience. Not a lot of the JSR-286 specs made any sense until further on in portal development, but helped as a good reference on the portal boundaries later on.

A very progressive structure is taken in this guide. So much that I found it hard to dive into later chapters out of order. Even just the theme development chapter made more sense once I read the previous chapters. Once I had completed a few chapters I found myself using it a lot as a reference. Especially when developing Struts Portlets.

With the latter chapters on workflow, CMS, WYSIWYG editors, communities and templates and themes I was using code from the book directly in my deployment. The examples applied well to customising Liferay for an intranet CMS environment (luckly for me).

There are little quirks in the code examples where more proof reading would have helped, but these were easily ignored by downloading the example code archive.

This book is not for you if you only want to change the database backend or create a basic theme. Nor if you dislike getting your hands dirty in XML. If on the other hand though you are deploying a Liferay based intranet and want to tie in existing content services or want a unique and flexible public site, then the book is right up your alley.

Overall I am impressed with the depth of the book and will probably be spending a few more weeks working through it as I get time. And I will be sure to post updates about the portals if I start developing any applicable to use outside the company.

Requisite Screenshot

It has made a perfect intranet theme for our uses. Most appreciated is the simplified interface look.

I have finished most visual touchups. Things left on my to do list are submenu indicators, tools menu rearrangement and rethemeing the chat portal.

If anybody has ideas on integrating the chat portal visuals closer I’d be happy to listen.

DOWNLOAD and INSTALL: The theme should now be available from the theme installer in liferay-5.2.2 installs and the WinterSky theme page on Liferay is here.

Now to work on some new portals. Till next time.

There are some areas that Plone really does well such as the content type handling and content workflow. But lack of documentation in other areas has left me pulling my hair out at times, such as ldap user and group integration with openldap and active directory, debugging modules and some new gotcha each time something is updated in a buildout.

Liferay has also had a steep learning curve, but I found it no where as bad as Plone. No messing in the ZMI for setting up ldap auth, changing database backends or setting portal defaults. Just one config file. I have found the speed surprising compared to Plone on the same hardware too. With java webapps I have tried in the past I have been left with the impression of them being bloated, memory hungry and slow. But I am slowly coming around thanks to Alfresco and Liferay.

It took a long time to get the liferay-portal-5.2.2.war running in the ubuntu jaunty tomcat6 environment and it still had a couple of issues so I just went back to running the liferay-timcat6 bundle which does run well without issue. I guess that is the price I pay for avoiding java for so long.

To make it suitable for my needs the interface needed to be cleaned up and I decided to just create a fresh theme which can be seen in the screenshot below. The theme is called Winter Sky and is adapted from MMOZINE, and is available from the Liferay community plugin section.

Other additions over the classic liferay theme are multiple level navigation menus and print styling css targets. Which were just a couple of extras that made things much more usable for my victims…. oh.. target users.

There are still a few visual glitches to fix, but everything seems usable and I will post the link to the Liferay project page once it is approved.