As a side-effect of joining Google, I have a new 13" MacBook Pro with a solid-state disk (good!) whose capacity is only 113G. This has pretty well screwed up my previous picture-management habits so I had to invent a new workflow, which is working out pretty well. I’m not claiming it’s right for anyone else, but others might find it a useful read. Plus since it’s photo-related I’ve dropped in three miscellaneous pix that I think are eye-pleasers.
The Old Setup · I had always managed to keep my whole life, excluding music, on one computer’s disk. I achieved this, even though I take a lot of pictures and shoot RAW, by being ruthless; I only keep the ones that I think good enough to publish or which have sentimental value. Other photogs blanch in horror when I speak of discarding digital negatives: “How can you know what might be important later?!” I can’t; but I can accurately forecast the immense amount of pixel-pumping drudgery that a strict retentionist policy would entail, and I ain’t going there.
The pictures are all stored in directories of the form YYYY/YYYY-MM, so
today’s would be in
Home ⇒ Pictures ⇒ 2010 ⇒ 2010-04.
I find that this meets my needs as well as anything else I’ve tried; among other things, I like to do slideshows of recent pictures, and this works great for that. Unfortunately, the pictures going back to 1998 added up to more gigabytes than I now have, so change was required.
Infrastructure Issues · Outboard disks, now apparently all USB-2 (what happened to FireWire?) are plentiful and cheap and good. But I don’t want to be fiddling with them when I’m on the road, and their performance doesn’t feel as good to me as a real actual internal disk wired up however it may be. Especially when the internal disk is an SSD.
It’s intangible, but it isn’t subtle. Everything that runs off the SSD, in particular everything that involves reading a lot of data, feels quicker. So for pix that I’m actually working on and editing and publishing, I’d like them local.
Archives · I still have the per-year directories in the Pictures folder, only now they’re symbolic links to the versions on an outboard 500G disk made by, uh, I forget who, that I bought at the Best Buy, uh, I forget where. Man, disks must be a tough market to make a living in.
I still keep the archive directories on the outboard USB and the big honkin’ Mac Pro consistent via rsync, which despite warnings that This Could Go Off The Rails, has so far worked smoothly for me.
Current · That’s the name of the directory which constitutes the key change. It lives under the Pictures directory and is the sandbox for the current month’s works. That’s where they get downloaded to from the SD-card slot in the side of the Mac (damn, that’s a nice feature, thank you Apple). That’s where they sit while I pretty them up in Lightroom. And that’s where they get exported from for family use or into this blog, directly or via Dailyshoot.
The key non-change is that even though most of my pictures are living off in USB-disk-land, the Pictures ⇒ Lightroom directory is still right there on the SSD. This setup, where most of the I/O is on a local SSD, seems to make Lightroom happy; it’s still not as fast as in the halcyon days of Lightroom 1.0, but it’s faster than on the previous MacBook.
One nice thing about Lightroom is that it doesn’t care if most of the actual raw files are living off on a USB disk that might be in a different timezone.
At Month-End · I make a YYYY-MM directory on the USB disk (and a symbolic link on the Mac) and transfer the raw files over there from the Current directory and [important] I do this using Lightroom. I had quite a bit of trouble getting this to work; I could select a ton of photos, but I couldn’t move them, until I realized that you have to click-and-hold on the actual image in the photogrid, not the grey area around it.
Anyhow, this might not work for you, but right now I’m reasonably satisfied with my computer’s workings in general and my photo-processing setup in particular.
Comment feed for ongoing:
From: Will Emerson (Apr 17 2010, at 12:08)
About firewire external drives. Other World Computing(http://eshop.macsales.com/shop/firewire/) remains the champ for selection and value for firewire/combo drives for the Mac.
[link]
From: Mark Pritchard (Apr 17 2010, at 20:16)
You might light to check out this neat trick with rsync: http://blog.interlinked.org/tutorials/rsync_time_machine.html
It avoids the problem of an upstream source for rsync corrupting your backup by keeping a time machine differential backup using hardlinks.
I backup ~160GB of photos etc and my per snapshot overhead is only ~5mg.
[link]
From: Karl Voit (Apr 18 2010, at 01:51)
at Mark:
What's the difference between your script and http://www.dirvish.org/ ?
AFAIK you're trying to invent the dirvish-wheel again.
Let others maintain your scripts :-)
PS: http://karl-voit.at/howtos/en/digicam_and_linux.shtml
[link]
From: Miko (Apr 18 2010, at 02:47)
"What happened to Firewire" summarizes my backup system: Keep everything on your boot disk of your main desktop box, which means your boot disk will have to be an external to be big enough (which turns out to be convenient when your computer breaks down and you don't want your confidential stuff in the hands of some repair shop loser).
By keeping everything on one disk and making that one disk your startup and backing that up with automated bootable clones (as with SuperDuper), you guarantee your stuff will always be on media that is still readable. No backups on SCSI or Firewire or old CD-ROMs you never kept fresh. When you get a new computer you put your stuff on a hard disk or Flash memory or whatever is current. So your "backup" never gets much out of date.
[link]
From: Michael Kozakewich (Apr 26 2010, at 23:10)
I always dread that point where one technological configuration wins out, and all other die. I'm guessing FireWire is getting scarce. Like Netbook SSDs, and 8.9" screens.
Is there a Macbook model which can take two drives at once? If so, a multi-drive setup with a small SSD and a large HDD would be optimal.
[link]
From: Craig Bowers (May 02 2010, at 21:58)
Hmm. If it were I, I'd be somewhat uneasy with the lack of insight into knowing when/if things went off the rails (logging, CRC checks of files).
I'm queasy putting important things like photos on a USB drive that won't signal/Log health issues, or let S.M.A.R.T monitoring tools read that information (which most firewire and USB drives don't pass). I worry with various script replication "backup" approaches that for old photos that aren't touched in a long time may be found to no longer be readable when you go to restore (having lost the source) but you don't know because you aren't reading the file on each backup, you're reading the FAT/MFT/(insert file table of choice) table.
For me, I use a managed library in Aperture so that images don't as easily fall through the cracks. Images I'm working on can be in a local library, but the main library I don't trust to a single spindle (even though it's backed up to an external firewire drive via timemachine), but also like you I don't have enough space on the boot volume. So the main library is on a Drobo(there are faster and less expensive options, but like the rest of the workflow, I value simple maintenance and expansion). That important stuff on the Drobo gets synced to a second partition on the external Firewire drive. But the main backup tool is Crashplan (.com) At the basic level it's zero cost, and runs on Mac/Windows/*nix I can backup between my machines, between friends machines (or external drives connected to friends machines), and to my own attached storage. I have other machines, and my laptop backup to my Drobo via Crashplan. Then I have Crashplan backup parts of my user profile, and parts of my Drobo to an external USB drive that I bring home from work weekly to sync, then take back to the office. Plus it backs up to the Crashplan hosted servers for $5 per month (unlimited, for all my home machines). The main bonuses for me are:
1. The backups are de-duplicated and compressed (so I can have the same photos in iPhoto and Aperture, yet they aren't duplicated in the backup).
2. The backups are encrypted, so it matters not if someone takes my backup drive from work or in transit, or if the data is backed up to another Crashplan enabled client machine, it's visible only to me.
3. The files are integrity checked at an interval of my choosing (default is 2 days). So the backup medium (external USB) can be simple and dumb. It handles issues silently in the background. If a file fails a CRC check in one of the multiple backup destinations it's fixed automatically.
4. I have logging to let me know that it's doing it's job, and emailed status reports to confirm backups are happening trouble free, or when a remote client (like my wife's laptop) hasn't checked in and backed up for x period of time.
5. The software is dead simple to use.
I can't say that for the various scripts and sync tools I've used in the past.
[link]
From: Craig Bowers (May 02 2010, at 22:13)
"Is there a Macbook model which can take two drives at once?"
Not out of the box, but any current Macbook can sacrifice it's optical drive for a second drive.
Here's one example http://www.mcetech.com/optibay/
They also include an external USB enclosure for your optical drive once it's yanked.
[link]