Here’s the short form in geek-speak: Apparently, you can use rsync to keep two computers running Lightroom in harmony. The long version is well, long, and digresses into Deep Vein Thrombosis and Olympic Table-Tennis qualification.
Recently, on a flight back from Chicago, I sat next to a personable gentleman and we conversed at length. He is a survivor of flying-related Deep vein thrombosis, something that everyone middle-aged and up who takes long flights should worry about. Let me put it this way: If you’re on a long flight, either swing an upgrade so you can lift your legs, or get up and walk around every two hours. Just do it. Otherwise you might die.
The reason my interlocutor, whose name I never got, was coming to Vancouver was to chair the 2008 Capilano Trials, a table-tennis tournament via which some Americans and Canadians will qualify for a trip to the Beijing Olympics.
The idea of going to a ping-pong tourney struck my fancy, and I sold it to the family. So this afternoon we trundled off and spent a couple of hours watching elite athletes try for a ticket to China.
I took the K20D and came home with well over a hundred pictures. These average just under 20M apiece, and the notion of stepping through them on my little MacBook to select the handful of keepers is just unthinkable. So I sucked them into the new Mac Pro and holy cow, can that suckah ever run Lightroom. Pretty soon, I had the eight I wanted to keep. Here are two.
· · ·
The woman facing you in the top photo is Zhang Mo and she earned herself a ticket to Beijing today. The fellow facing you below is Pierre-Luc Hinse and he didn’t (even though he won the match in the picture), but he’ll have another chance on Sunday.
The Problem · I want all my pictures on both the MacBook (from which I blog) and the Pro (on which I plan to edit). Since I ruthlessly throw out everything that isn’t either publishable here or of sentimental value, my entire photo collection is something under 40G in size, so why shouldn’t I always have it with me?
I poked around the Internet and found remarkably little wisdom on how to solve this problem. So, via Twitter, I asked ÜberPhotoGeek James Duncan Davidson what he’d do and he responded “My advice for LR or AP on 2 machines: designate 1 your master. The other is for intermediate work. Migrate in 1 direction only.”
I got depressed, because I want to edit mostly on the Pro but sometimes on the laptop, and I don’t see why this should be hard.
Unix Rules OK ·
So I poked around to see where Lightroom kept things, and discovered that
on the Mac, it’s all under your Pictures folder. And I note that
OS X is after all Unix, and on Unix everything is a file, and a Lightroom
catalog is an
SQLite database, but dammit, that’s
just files too. And on Unix, there’s this wonderful marvelous
thing called rsync whereby
you can say “Please make this directory on this machine the same as that
directory on that machine and do it really damn efficiently”.
Because I’d set up the Pro with
Migration
Assistant, its Lightroom universe and my MacBook’s started out about the
same; after which I’d edited all the ping-pong pictures. So I poked around the
rsync man page and typed this into a Terminal on the laptop:
~/ 513> rsync -avz xenophon.local:Pictures/ Pictures(Xenophon is the new Mac Pro.)
It ran through those thirty-odd gigabytes in just a few seconds. I’m in love. Well, there were a couple of warnings about this or that, but I decided to ignore them.
Important: For this to work, Lightroom has to be turned off on both machines!
Yak Shaving · First of all, I had to arrange for the computers to have different names. You need “Sharing” under System Preferences. And while you’re there, you need to enable what they call “Remote Login” and I call “ssh”.
Which is pretty well just a yaklet.
Futures ·
So far, this seems to Just Work; the pictures above (edited in LR on the
Pro, exported from LR on
the laptop) are evidence.
Experts will note that the rsync command above is directional;
my rsync-fu is a little stale
but I bet that there’s a way to tell it to just look at both and copy things
both ways to achieve the desired effect, because after all
Tridge is God.
I’m probably going to wake up in a cold sweat in the wee hours tonight when the reason that This Can’t Possibly Work dawns on me. But for now, it all seems to fit together unreasonably well.
Comment feed for ongoing:
From: Malcolm Tredinnick (Apr 06 2008, at 01:43)
There's no "pull both ways" magic option to rsync. Partly, I guess, because it's a little ill-specified as to how that would exactly work with deleted files, for example. However, the -u option uses timestamps to only update older files, so running two consecutive rsyncs with -u to push and pull keeps things in sync.
The only small problem to overcome then is handling deleting of files. Since you don't have a record of when a file was deleted, using --delete and -u together is a little dangerous (I can't remember how they interact).
Finally (and I am not a Mac user), I thought there were problems with rsync and resource forks on HFS filesystems. Is that no longer an issue?
[link]
From: Bruno Rodrigues (Apr 06 2008, at 02:05)
We did it differently over here.
We have a Mac mini Intel with two 320GB external harddisks. One holds the 117GB photo library, and the other holds other kind of files. Both are manually rsync'ed with each other. (/Volumes/Disk1/* -> /Volumes/Disk2/.backup/* and vice-versa)
Then Lightroom on my MacbookPro accesses the macmini remotely, via AFP or samba. Locally I only have the latest three months (which are also rsynced to the Mac mini, for backup purposes).
This way I have the whole library on my MacBook, accessible for slideshows (but not edit), whilst by coming home and connecting to the mac mini I can then access the whole library.
The data sizes:
Mac Mini: Photos Library: 117GB (~2000 up to three month ago)
MacbookPro: Photos: 9.3GB (last three months)
MacbookPro: Lightroom Catalog (just the .lrcat file): 728MB
MacbookPro: Lightroom Previews: 13GB
Photos: almost 40K
Photo types: old JPEGS, 300D JPG, 300D RAW, 400D RAW, IXUS JPEG
Hope it helps you
[link]
From: Graham Hughes (Apr 06 2008, at 02:49)
The main problem with doing this is that both copies of Lightroom should be closed at the time; SQLite does not handle changes to its backend database gracefully and may (due to journalling) have all kinds of crap strewn about it basically at random. Indeed, all of the data from the Lightroom instance may not be there at the time. That said it should work just fine for the pictures itself, as Lightroom uses nondestructive edits.
[link]
From: Constantin Gonzalez (Apr 06 2008, at 02:54)
Hi Tim,
great post! If you have some time, check out unison: http://www.cis.upenn.edu/~bcpierce/unison/ This is something like rsync on steroids. Multi-Platform, efficient, configurable, etc.
I use it to sync my wife's MacBook and my own PowerBook with my ZFS home server and my Solaris Laptop with my SWAN homedirectory. Can both do two-way (for real syncing) and one-way (for backups) and asks you when something doesn't fit (so you can resolve the conflict) or be 100% automatic.
Cheers,
Constantin
[link]
From: M. David Peterson (Apr 06 2008, at 03:57)
From your MacBook run,
rsync --ignore-existing --exclude='.*' --recursive --perms --times --size-only --whole-file --progress --rsh=ssh xenophon.local:Pictures/ Pictures
.. and then ...
rsync --ignore-existing --exclude='.*' --recursive --perms --times --size-only --whole-file --progress --rsh=ssh Pictures/ xenophon.local:Pictures
[link]
From: Rob... (Apr 06 2008, at 05:54)
Hi Tim,
Does this rely on using sidecar files for all the edits?
Also, if you add files to the catalogue at both ends and then rsync, does one override the other?
Regards,
Rob...
[link]
From: Norman Walsh (Apr 06 2008, at 07:40)
I think you'll be ok as long as you don't edit on both machines between syncs. Like you say, they're all just files.
I use rsync to keep my various photo backups in sync. I rsync from my laptop to the notional master backup then rsync from that backup to the other backups. Every now and then I rsync in the other direction (usually with -n so I can see what's going to happen) just to catch random edits I might have made on a backup.
I fear that if you do some Lightroom edits on both Macs, you'll be screwed. If you think that's going to happen, I'd run a test where I did that, then dump the two databases and see if a merge looks like a practical operation. Then again, maybe there's a straightforward SQL way to do the merge, I usually resort to Perl over the text dump.
[link]
From: Joe Cheng [MSFT] (Apr 06 2008, at 08:43)
For Windows, SyncToy:
http://www.microsoft.com/windowsxp/using/digitalphotography/prophoto/synctoy.mspx
Although I'm pretty sure you will run into conflicts (whether using rsync or SyncToy) if you make edits on both computers between syncs. I've seen this happen on my two PCs with LR, now I just sync one-way.
[link]
From: John Cowan (Apr 06 2008, at 10:29)
I second the recommendation for unison; even though it's hairier than rsync, its inherently two-way nature makes it superior for this sort of job.
[link]
From: Tim (Apr 06 2008, at 11:31)
MDP: Care to expand on the finer points of your lengthy rsync incantations and why it should be done that way? In fact, why don't you do it at length on your blog and I'll link to it.
[link]
From: James Duncan Davidson (Apr 06 2008, at 12:12)
Indeed, this will work as long as you are disciplined enough to keep track of which machine you currently have updated images last (and this means any kind of updates, either adding images or editing). If you are disciplined enough to do this, then you can make it work. The problem comes when you either accidentally mess up and forget to do the rsync before working on the other computer.
I actually did things this way at one time way back when my library would fit on a laptop drive, but there were a few times when I had to do edits on both collections (for various silly reasons dealing with having to work with other people's deadlines) and working the merges back was a royal pain in the ass even when I knew what needed to be merged. And then there was the time(s) that I just plain boffed it and forgot and merged killing some amount of work in the process.
As long as you're really disciplined, tho... :) Even if you slip, the good news is that unless you use the --delete-files option with rsync, you won't loose RAW data. Just metadata and edits.
Kudos on still having a library that can fit on small laptop drives. I vaguely remember that time...
[link]
From: Christian (Apr 06 2008, at 12:39)
when using rsync on Macs, you should use the -E option for resource forks and ACLs
[link]
From: Michael Norrish (Apr 06 2008, at 17:05)
Third vote for Unison; this scenario is exactly what it's designed for. (It uses rsync underneath.)
[link]
From: Aristotle Pagaltzis (Apr 06 2008, at 20:41)
I wonder how terrible a mess could possibly result from using git to attack this problem…
[link]
From: M. David Peterson (Apr 07 2008, at 19:52)
@Tim,
Just checking back on the follow-up comments now. I'll write that up this evening and point you to it when complete.
[link]
From: Eric Scouten (Apr 08 2008, at 10:46)
I'm not an rsync guy, but I do use Perforce to move my catalog between desktop and laptop. Latest version of P4 has a nice "save the last <n> revisions only" feature, which prevents my server from getting overloaded with too many copies of my catalog file.
[link]
From: Justin Sheehy (Apr 11 2008, at 08:55)
rsync will not work well in a two-way situation.
To add more weight to the unison suggestions: I currently use it for almost exactly the purpose you want. (not Lightroom, but same workflow)
It takes a little more setup up front, but then It Just Works.
[link]
From: Shawn Ferry (Apr 14 2008, at 22:51)
James Duncan Davidson
http://www.tbray.org/ongoing/When/200x/2008/04/05/Lightroom-sync-with-rsync#c1207509144.804633
Already addressed my major concern, that of losing work to a sync between two edited locations.
I can't fit all my images on my laptop but I can keep the aperture library that I use and referenced masters and library backups in redundant locations.
One suggestion would be to try using your Mac Book as an external FW disk. Boot it into FW target disk mode hold "T" at boot. You may then be able to use the LR library in a different location.
You could still use unidirectional rsync from the Mac Book to the Mac Pro (or over FW in target disk mode) to make a local/backup copy.
You would be penalized in your use of the Mac Book drive in terms of disk performance, but it could/would still be the master.
[link]
From: muro (Apr 15 2008, at 04:50)
just a remark regarding unison: it does not support unicode filenames. This might not be a problem for you in the US, but it certainly makes it unusable to many people (e.g. me).
[link]