Entries by Paul Dwerryhouse

I've been playing around with Android application development quite a bit, over the last few months. The one thing I've built that's actually quite usable has been the wake-on-lan client Etherwaker (because the world really needed another one of these, didn't it?)

I've just put the Mercurial repository for it up on Bitbucket and released it under the GPL-3, for people to peruse or fork at their leisure.

Five second guide to fetching the source: hg clone ssh://hg@bitbucket.org/pdwerryhouse/etherwaker

If you can't be bothered with all this, and just want to wake up your mythtv box from your bed, then it can be downloaded from the Android market.

I've recently been asked to build a redundant mailstore, using two server-class machines that are running Ubuntu. The caveat, however, is that no additional hardware will be purchased, so this rules out using any external filestorage, such as a SAN. I've been investigating the use of DRBD in a primary/primary configuration, to mirror a block device between the two servers, and then put GFS2 over the top of it, so that the filesystem can be mounted on both servers at once.

While a set-up like this is more complex and fragile than using ext4 and DRBD in primary/secondary mode and clustering scripts to ensure that the filesystem is only ever mounted on one server at a time, it's likely that there will be a requirement for GFS on the same two servers for another purpose, in the near future, so it makes sense to use the same method of clustering for both.

The following guide details how to get this going on Ubuntu 10.04 LTS (lucid). It won't work on any version older than this - the servers that this is destined for were originally running 9.04 (Jaunty), however, I've tested DRBD+GFS on that release, and there's a problem that prevents it from working. As far as I'm concerned, production servers should not be run on non-LTS Ubuntu releases, anyway, because the support lifecycle is far too short. This guide should also work fine for Debian 6.0 (squeeze), although I haven't tested it, yet.

One thing to keep in mind - the Ubuntu package for gfs2-tools claims that "The GFS2 kernel modules themselves are highly experimental and MUST NOT be used in a production environment yet". There's a problem with this, however - the gfs2 module is available in the kernel, in Ubuntu 10.04, but the original gfs isn't there (it wasn't ever there) and the redhat-cluster-source package which provides it, doesn't build. I'm inclined to say that the "experimental" warning is incorrect.

Firstly, install DRBD:

apt-get install drbd8-utils drbd8-source

We have to install the drbd8-source package in order to get the drbd kernel module. When drbd is started, it should automatically run dkms to build and install the module.

Now, the servers I'm using have their entire RAID already allocated to an LVM volume group named vg01, so I'm going to create a 60Gb logical volume within this volume group, to be used as the backing store for the DRBD block device on each. Obviously, this step isn't compulsory and the DRBD block devices, can be put on a plain disk partition instead.

lvcreate -L 60G -n mailmirror vg01

After this, configure /etc/drbd.conf on both servers:

global {
  usage-count yes;
}

common {
  protocol C;
}
resource r0 {
  net {
    allow-two-primaries;
    after-sb-0pri discard-zero-changes;
    after-sb-1pri discard-secondary;
    after-sb-2pri disconnect;
  }
  syncer {
    verify-alg sha1;
  }
  startup {
    become-primary-on both;
  }
  on mail01 {
    device    /dev/drbd0;
    disk      /dev/vg01/mailmirror;
    address   10.50.0.11:7789;
    meta-disk internal;
  }
  on mail02 {
    device    /dev/drbd0;
    disk      /dev/vg01/mailmirror;
    address   10.50.0.12:7789;
    meta-disk internal;
  }
}

With this done, we can now set up the DRBD mirror, by running these commands on each server:

drbdadm create-md r0
modprobe drbd
drbdadm attach r0
drbdadm syncer r0
drbdadm connect r0

...and to start the replication between the two block devices, run the following on only one server:

drbdadm -- --overwrite-data-of-peer primary r0

By looking at /proc/drbd, we'll be able to see the servers syncing. It's likely that this will take a long time to complete, but the drbd device can still be used, while that's happening. One last thing we need to do is move it from primary/secondary mode, into primary/primary mode, by running this on the other server:

drbdadm primary r0

So, now we want to create a GFS2 filesystem. There's a catch here, however: GFS2 cannot sit directly on a DRBD block device. Instead, we need to put an LVM physical volume on the DRBD device, and then create a volume group and logical volume within that. Furthermore, because this is going on a cluster, we need to use clustered LVM and associated clustering software:

apt-get install cman clvm gfs2-tools

And then configure the cluster manager on each server. Put the following in /etc/cluster/cluster.conf:

<?xml version="1.0" ?>
<cluster alias="mailcluster" config_version="6" name="mailcluster">
        <fence_daemon post_fail_delay="0" post_join_delay="3"/>
        <totem consensus="6000" token="3000"/>
        <clusternodes>
                <clusternode name="mail01" nodeid="1" votes="1">
                        <fence>
                                <method name="1">
                                        <device name="clusterfence" nodename="mail01"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="mail02" nodeid="2" votes="1">
                        <fence>
                                <method name="1">
                                        <device name="clusterfence" nodename="mail02"/>
                                </method>
                        </fence>
                </clusternode>
        </clusternodes>
        <cman expected_votes="1" two_node="1"/>
        <fencedevices>
                <fencedevice agent="fence_manual" name="clusterfence"/>
        </fencedevices>
        <rm>
                <failoverdomains/>
                <resources/>
        </rm>
</cluster>

In the above, I'm using manual fencing, because at the moment, I don't have any other method for fencing available to me. This should not be done in production; it needs a real fencing device, such as an out-of-band management card (eg, Dell DRAC, HP iLO) to kill power to the opposite node, if something is amiss. All that manual fencing does is write messages to syslog, saying that fencing is needed.

Without fencing, it's possible to encounter a situation where the DRBD device might have stopped mirroring, yet the mail spool is still mounted on each server, with the mail daemon on each one writing to its GFS filesystem independently, and that would be a very difficult mess to clean up.

One other thing: there's an Ubuntu-specific catch here - Ubuntu's installer has this irritating habit of putting a host entry in /etc/hosts for the hostname with an IP address of 127.0.1.1. This will break the clustering, so remove the entry from both servers, and either make sure your DNS is set up correctly for the name that you're using in your cluster interfaces, or add the correct addresses to the hosts file.

You can now start up clustering on both hosts:

/etc/init.d/cman start

Run cman_tool nodes, and if all is well, you'll see:

Node  Sts   Inc   Joined               Name
   1   M    120   2011-09-14 10:53:32  mail01
   2   M    120   2011-09-14 10:53:32  mail02

We'll need to make a couple of modifications to /etc/lvm/lvm.conf on both servers. Firstly, to make LVM use its built-in clustered locking:

locking_type = 3

...and secondly, to make it look for LVM signatures on the drbd device (in addition to local disks):

filter = ["a|sd.*|", "a|drbd.*|", "r|.*|"]

Now start up clvm:

/etc/init.d/clvm start

At this point, we can create the LVM physical volume on the drbd device. Because we now have a mirror running between the two servers, we only need to do this on one server:

pvcreate /dev/drbd0

Run pvscan on the other server, and we'll be able to see that we have a new PV there.

Now, again, on only one server, create the volume group:

vgcreate mailmirror /dev/drbd0

Run vgscan on the other server, to see that the VG also appears there.

Next, we'll create a logical volume for the GFS filesystem (I'm leaving 10Gb of space spare for a second GFS filesystem in the future):

lvcreate -L 50Gb -n spool mailmirror

And then lvscan on the other server should show the new LV.

The final step is to create the GFS2 filesystem:

mkfs.gfs2 -t mailcluster:mailspool -p lock_dlm -j 2 /dev/mailmirror/spool

mailcluster is the name of the cluster, as defined in /etc/cluster/cluster.conf, while mailspool is a unique name for this filesystem.

We can now to mount this filesystem on both servers, with:

mount -t gfs2 /dev/mailmirror/spool /var/mail

That's it! We now have have a redundant mailstore. Before starting your mail daemon, however, I'd suggest changing its configuration to use maildir instead of mbox format, because having multiple servers writing to an mbox file is bound to cause corruption at some point.

Other recommended changes would be to alter the servers' init scripts so that drbd is started before cman and clvm.


Paul Dwerryhouse is a freelance Open Source IT systems and software consultant, based in Australia. Follow him on twitter at http://twitter.com/pdwerryhouse/.

There has been quite a campaign to encourage people to put Senator Stephen Conroy last on the Victorian Senate ballot paper, in light of his never-ending attempts to filter the internet in Australia.

I can sympathise - several years ago, I was advising people to put Senator Richard Alston last on the same ballot paper, for similar reasons, and did so myself. I was wrong to do this.

By putting Senator Conroy last, you are effectively saying that his policies are worse than everyone else on the ballot paper. I am utterly against the filter, but, that said, there are plenty of issues just as serious, and there are some absolute nutcases standing for election for Victoria's senate seats. Let me provide a few examples:

Family First are a group of extreme religious social conservatives, and most of their members belong to strange pentecostal sects. They too want a mandatory filter, but beyond that, they want to ban internet pornography entirely (good luck with that), they're firmly against abortion and euthanasia, and they believe that "Small Business (are) the True Heroes of the Economy", whatever that means. Now, I'm not saying that Family First are a front for whack-job churches like Hillsong and the Assembly of God, but whenever Senator Steven Fielding opens his mouth, I'm pretty sure he's speaking in tongues. Their Queensland lead Senate candidate has, err, issues, and in the last election, the party demonstrated their lack of judgement by endorsing Pastor Danny Nalliah of Victoria's-bushfires-were-an-act-of-retribution-from-God fame. Stephen Conroy may be a devout Catholic, but he's not beyond ignoring stupid church doctrine and taking advantage of the NSW surrogacy laws, something which his own state doesn't allow. He's far better than the Family First nutters and should be put higher on the ballot paper than them.

The Citizens Electoral Council are a pack of Larouchite loons who should be put absolutely last on any sane human being's ballot paper. Conroy is far preferable to them.

We all know who One Nation are, and what they stand for. The only reason I put them above the Citizens Electoral Council is that One Nation couldn't organise a dinner in a room full of fish-and-chip shop owners. They've proved that they're too incompetent to be dangerous. Nevertheless, they're racist and extreme-right. Conroy is easily better than them.

The Liberal Party of Australia is a socially conservative party with an almost-dead small-l liberal faction. It is led by a man who, when health minister, pulled out all stops to keep RU486 banned in Australia. He believes that "climate change is crap" and is so creepy that he talks to the media about his daughters' virginity. One of the Liberal Party's Victorian candidates that is running for re-election is a former National Party member named Julian McGauran. The Age has an interesting article that refers to him. Definitely going below Conroy.

Obviously, there are plenty of good parties to put above Labor: the Greens, The Australian Sex Party and The Australian Democrats are all socially liberal parties. Stephen Mayne (of Crikey fame) is also running for the Senate, and while I disagree with a few things he's said in the past, he's shown himself to be honest and generally progressive.

But to put Senator Conroy last on your ballot paper is to say that he's worse than a herd of far-right, bigoted religious fundamentalists, who want to interfere with your life. Despite his ridiculous stance on the filter, I don't believe that he is as bad as them.

So, I've finished my mad dash from the north of Norway, to Stockholm, in order to vote in one of the only two locations in Scandinavia and the Baltics that Australia makes available (the other being Copenhagen). Australia typically only provides voting facilities in embassies, and as Norway, Finland, Estonia, Latvia and Lithuania only have honorary Australian consulates, there's no opportunity to vote in any of those countries (unless, of course, you have a permanent address there, and thus can get a postal vote).

The voting process was all very straightforward - a room had been set up on the ground floor of the building which houses the embassy, so there was no need to pass through any faux-security measures in order to get in, unlike when I voted in The Hague back in 2001.

No identification was required, as is typical for Australian elections - it was just a matter of completing what was probably a postal vote envelope, and then filling out the ballot papers. The electoral officer then explained how to vote on each paper - the instructions were accurate, though I felt she emphasised a little too strongly that the Senate ballot paper was big, which I suspect caused a couple of people who followed me to vote above the line. That said, she did point out that all the group ticket preference allocations were available for people to read, if they wanted. I always vote below the line, so I didn't have any need for this.

I was amazed, however, at a question from one of the other voters in the room: "This isn't for local elections, is it?". Seriously, I know I'm more attuned to politics than the average person, but a question like this is probably a good argument for compulsory civics lessons in schools. I find it somewhat unbelievable that state schools still brainwash children with religious education, but fail to teach them the basics of how our democracy works.

For the last two weeks, I've been drifting around northern Norway, spending a few days in the university town of Trondheim, before moving further north to Bodø and the Lofotens.



I was lucky enough to arrive in Trondheim during the St. Olav festival, a week-long smörgåsbord (ok, that's a Swedish term) of music and food, including a concert by one of Sweden's biggest bands, Kent who, surprisingly, have absolutely no profile in English-speaking countries whatsoever.



My visit to the Lofoten islands included a couple of nights in a small fishing village with the simple, easy to spell name of Å, after a three hour ferry ride from Bodø, which left me feeling decidedly nauseous, although I'm not entirely sure if that was from the rough seas, or just the smell from the other passengers who had thrown up. Either way, I was glad to get back onto land.

The Lofotens would be, I imagine, a hiker's ultimate dream. Huge dramatic peaks emerging from the sea, and unbelievable views from the top. I'm not anywhere close to being an experienced hiker or bushwalker, but I have been getting out and walking up quite a few of these mountains, and in one case, high enough that there was still some snow at the top. On a clear day, you can see for miles, and there's virtually no sound other than the wind, and in some cases, running water.

I've found Norway to be particularly easy to travel in; almost everyone speaks English to some degree - and furthermore, Norwegian is very similar to both Swedish, which I took a short-course in three years ago, and written Danish, which I've attempted to teach myself, in the past, thus reading signs, menus and travel websites isn't too much of a problem. Being a Germanic language, Norwegian also shares quite a bit of vocabulary with German and Dutch (both of which I've had quite a bit of exposure to), as well as English itself, or at least the parts of it that weren't bastardised by the Normans. Unfortunately, my attempts to try a bit of Norwegian don't usually work too well, and I usually have to fall back to English.

One thing that is really fantastic here is the extent of good broadband internet access; I've been in tiny little towns, often with populations of one hundred or less, and it's been clear from the wifi signals (and, admittedly, a little prodding of the open ones, on my part) that good broadband is available widely. There would be towns of similar size in Victoria who still have trouble getting a reliable dial-up connection. Mobile broadband also appears to be widespread, and not just from the former monopoly telco Telenor, but also a second carrier Netcom - and while the prices are, naturally, fairly expensive for an Australian, Netcom at least allows unlimited downloads for 20kr (AUD$3.6 / €2.50) per day, rather than capping or just pretending that it's unlimited and then charging for excess usage (ie, more than 50Mb per day) like a certain telco in the Netherlands did to me.

I'm now in Narvik, a port city and part-time ski-resort, waiting for a bus to take me to my northernmost destination, Tromsø. I had originally planned to go further north to Nordkapp, but unfortunately the Australian election has put paid to that, and I have to get to Stockholm before August 21st, to vote.



While the midnight sun has long passed, it still does not get completely dark at night; it's possible to wander around at midnight and not require any artificial lighting at all. Two evenings ago, I walked up Narvik's closest mountain, leaving at about 3.30pm and not reaching the summit until around 8pm - the sun was still high in the sky, and it was as bright as it had been in the middle of the day. It took me another two hours to walk back down again, and at 10pm, the sun was only just beginning to drop below the mountains to the west.

Wow. I really am inept at keeping this up-to-date.

Well, I'll make the last month brief: Toronto (a week recovering from my travel so far); London - UK (two weeks recovering from my week in Toronto); Edinburgh (not surprisingly, recovering from London - I see a pattern developing here).

Following Edinburgh, I signed up with Macbackpackers for a five day tour of Scotland's Highlands and Isle of Skye. I don't normally take tours, generally preferring to travel independently, but not wanting to drive, this tends to limit my options to cities and larger towns. I'd also had recommendations from friends about this company, so I decided that it would be a nice change.

And they certainly weren't wrong; the tour was the most fun I've had during my trip so far. Our guide, a native highlander was excellent. From the moment he entered the bus, he had the group (of around 21-22 people) laughing and kept it up for the entire trip. His knowledge of the area and its history was first-rate, and had an amazing gift for storytelling while keeping the bus on the road.

The tour is designed for people under 35, but they don't enforce this, unlike many of the "youth tour" operators in Europe (who won't let someone like me, two years older than the cutoff point, aboard); they'll welcome anyone onto the tour, as long as you're happy to keep up with the fairly vigorous program, such as walking up steep hills, swimming in the freezing Loch Ness and late, alcohol-fueled nights in pubs. And then 9am starts the next morning.



Accomodation is at the company's many hostels, which range from utterly excellent (Castle Rock, Edinburgh) to fairly cramped and lacking sufficient numbers of showers, but otherwise clean and friendly (Inverness); but you're not obligated to stay in these - you can book hotels or B&Bs seperately, if you prefer.

The first day took us north from Edinburgh, via Pitlochry, to Inverness, visiting Ruthven Barracks and the Culloden Moor Battlefield. Day two was onwards to Skye, with a stop in Ullapool for lunch, and a scenic drive south along the west coast.
Following this was a day doing a circuit of Skye, including a couple of walks through the highlands.

The fourth day was packed with a boat trip on Loch Ness, and yet more walking, this time through Glen Coe. The tour's evening grand finale was a night of Ceilidh Dancing in Oban, which is great fun; essentially it's barn-style dancing, sometimes with one partner, sometimes with multiple partners, to traditional Scottish music.

After that, we wound down with a tour of Oban's whisky distillery and a visit to the National Wallace Monument... and then a relaxing drive back to Edinburgh.



I didn't know any of the people I was travelling with prior to the trip, but within just a few hours we all got along really well. It's amazing how quickly people will bond, if you pack them into a bus, goad them to strip down to their underwear (or bathers, for those of us who are slightly more prepared), bribe them to get into a freezing lake, and follow it with a bottle of whiskey (allegedly to warm them up, but frankly I think there was an ulterior motive).

Anyway, I now find myself back where I started: it's taken me the best part of a week in Glasgow and Belfast to recover from this...



Standard disclaimer applies: I'm not affiliated with this company at all, but I really really enjoyed the tour, and highly recommend it.

I've spent the last week in Canada, firstly in Quebec City and then Montreal. Quebec City gave me a bit of a chance to improve what little French knowledge I have - this amounts to a four week summer school course that I took at Melbourne University back in 1997, most of which I've forgotten.

Montreal, on the other hand, turned out to have a large English-speaking population, which is something I didn't expect from the largest city in a province that is so rabidly francophone that even their stop signs are in French:



From what I remember of France, for all of their anti-English sentiment, they have "Stop" on their stop signs. Clearly, they're softies in the face of anglophonic cultural imperialism.

I'm now in Ottawa, and hoping very much that this city, which seems to have had a history not unlike that of Australia's capital (chosen for its location midway between two bickering groups), isn't anywhere near as boring as Canberra.

I'm currently in New York, and have had the chance to see a live amateurish terrorist attack in progress. Well, at least, thousands of people standing around, on the streets surrounding Times Square, while police yell "Keep moving! You can't stay here!" to little effect.

I haven't exactly been keeping this blog up-to-date with my travel progress; so far, since Vancouver, I've travelled to Seattle, Chicago, Boston, Washington DC, and now here, all by train (and one bus). So I've now officially travelled the US from west to east coast, entirely over land. I haven't even done that in Australia.

In the unlikely event that one of you actually wants to read more of my travels (and other mindless, off-the-cuff thoughts), I'm on Twitter as paul88888.