Filesystems

Thanks to all who responded to last week's column, rsync and the unsung command line. Once again, I am indebted to my readers for bundles of great tips, advice, and corrections. In particular, I'd like to thank those readers who turned me on to a cool utility called unison. I've now replaced rsync with unison for some of my scheduled tasks, since it does a two-way synchronization by default and does it well.

If you want to do a two-way synchronization with rsync, you'll need to run it twice. It isn't difficult to run rsync twice, and doing it that way allows you to automate different synchronization options for one direction than for the other, which can sometimes come in handy. Nevertheless unison is so much easier for straightforward synchronization tasks that I've switched from rsync to unison when it comes to synchronizing my home directory across machines.

Here's the scoop on how to use unison. It is most powerful when you use it interactively, because this gives you a chance to decide how to resolve conflicts when both the source and destination files have changed. However, you don't have to use it interactively for it to be useful. In my case, I have a personal version of the following shell script in my /etc/cron.daily directory on each of my servers. This script synchronizes the home directory from client to the servers.

First, I had to set up my client to run unison as a server. I simply added the following line to one of my boot scripts:

unison -socket 755 &

Note that I'm using port 755 in this example. This is an entirely arbitrary choice. The unison program has no default port. You simply have to avoid using a port that some other network service has taken on your system. I recommend that you choose a privileged port below 1024 because these ports are more likely to be blocked at your firewall. If not, make sure that you block the port you choose, or wrap unison in some other transport such as ssh. Otherwise you'll be opening a huge security hole, because unison is not a very safe utility when left to its own devices.

I avoided the whole issue of port blocking by running the unison server at my client instead of on my servers. My client accesses the Internet using Network Address Translation (NAT), which means the client is inaccessible from the Internet unless I specifically enable port forwarding for any given service.

Here's the script I put in the /etc/cron.daily directory on the server machines.

#!/bin/bash
PATH=/usr/bin:/usr/sbin:/bin:/sbin
echo Synchronizing home
unison socket://myserver:755//home/me /home/me -owner -group -batch \
-prefer socket://myserver:755//home/me

The synchronization will take quite a while the first time you run it, because unison needs to create initial archive files on the source and destination machines, which it will later use to make subsequent synchronization sessions faster. It also creates lock files to help manage transfers.

This actually posed a problem at one point. I have no idea how it happened, but the archive files were somehow corrupted. The corruption of the archive wasn't a big deal - it was the fact that unison didn't want to resolve the problem without my intervention. It's possible I was simply missing a command-line switch somewhere that would automate the resolution to this problem, but given what I knew, all I could do was find out where unison put the archive files and delete them manually, along with the leftover lock files. This could get to be annoying if it happens frequently, because the whole point of using unison in my case is to have it synchronize the directories automatically every night.

We don't need Exchange? So, what's new?

Several other readers corrected my presumption that you need Microsoft Exchange server in order to use the Windows Offline Folders feature that lets you synchronize files in Windows the way rsync or unison synchronizes files on UNIX. I'm not surprised that I got this one wrong, since I don't use Windows for anything but Windows-only games. In case you're wondering where I got the idea that you needed Exchange, it was from Microsoft. I simply searched for the term "Offline Folders" at the Microsoft site and followed the first few links that turned up. You can perform the same search yourself, or check the resources section for a couple of the links I found, but here's the relevant portion of the text from two of the top hits:

"Working with Offline Folders: Offline folders are a feature of Microsoft Exchange Server that makes public folders and mailbox messages available to users when they work offline."

"OL97: What Are Offline Folders and How Do You Use Them? (Q161725) When you use Microsoft Outlook 97 with the Microsoft Exchange Server Information Service, you have the ability to work with folders "offline." That is, you can use the contents of a folder without a network connection, which is beneficial for remote users. This article discusses how you can create, use, and synchronize offline folders."

I'm inclined to believe my readers when they say Microsoft Exchange is now superfluous (as if we Linux fans needed convincing), and no doubt there is a web page somewhere on the Microsoft site that says so. It's a shame Microsoft fails to mention that little detail in the pages that rise to the top of the hit list when you search for the term "Offline Folders".

Duplication of effort

Still other readers pointed out that I was mistaken about having to be present to type in a password if you use Secure Shell (ssh) as your transport when you use rsync to synchronize files. They offered two different methods to get around this requirement - host key authentication and a script automation program called expect. I haven't tried either of these methods, but I can already say that I would prefer the former. It is better to authenticate against a public key than to embed a password into a script. Unless I'm misunderstanding the process, that's what you'd end up with if you use expect.

For those of you who also want to explore the host authentication route, I suggest you follow the link in the resources section to an IBM Developerworks article by Daniel Robbins. The article wasn't terribly useful to me but perhaps it will be for you. If you feel as confused as I am after reading it, drop me a note and I'll try this technique myself in order to write about it in a future column.

Still others mentioned that you could use the InterMezzo filesystem that is now available in the latest 2.4 and 2.5 kernels to perform automatic synchronization. As of this writing, I don't think anyone has mentioned the Coda filesystem, which has been available in the Linux kernel for ages and is similar to InterMezzo in concept. Both are intended for users who need to access files normally located on the network, even when they are disconnected from that network. Both implemented as filesystems. However, I have a feeling that's where the similarity ends. InterMezzo seems much easier to set up and use.

Working on network files while disconnected isn't something I do often, so I'm not compelled to try InterMezzo or the latest version of Coda to see if I'm right. Fortunately, Moshe Bar was clear about his experiences with InterMezzo in a Byte.com article (see resources for link), so if you're interested in this type of synchronization technique, check out his write-up.

Finally, none of the e-mails I've read so far mentioned CVS. CVS is overkill for typical file synchronization needs. It is entirely possible that those of you who know and love CVS will actually use it this way. If either one of you happens to read this column, let me know.