Mac Migration Pain

What happened was, my backpack got stolen with my work and personal Macs inside. The work machine migrated effortlessly but I just finished up multiple days of anguish getting the personal machine going and thought I’d blog it while it was fresh in my mind. This might be helpful to others, either now or maybe later at the end of a Web search. But hey, I’m typing this ongoing fragment on it, so the story had a happy ending. And to compensate for its length and sadness, I’ll give it a happy beginning; two, in fact.

Happy corporate · The IT guys were excellent, they’re in quarantine mode too but somehow arranged on two hours’ notice that I could drop by the spookily-empty office and there it was on a shelf outside the IT room. I’d backed everything that mattered up straight to S3, it restored fast and painlessly, and all I had to do was re-install a couple of apps.

As I’ve said before, Arq worked well. But as you will soon learn, there are pointy corners and boy did I ever get bruises.

Happy Ukelele · When I’m writing text for humans to read, I like to use typographical quotes (“foo” rather than "foo"), apostrophes (it’s not it's), and ellipses (… is one character not three). These are included in every font you’re likely to have, so why not use them? The default Mac keystrokes to get them are awkward and counter-intuitive.

Since (blush) I hadn’t backed up the keyboard layout I’d been running for years, I went and got Ukelele, a nifty little software package that makes it absurdly easy to modify your Mac keyboard layout. Here’s a screenshot.

I produced a layout called “Typographic Quotes” which remaps four keys:

1. Opt-L is “

2. Opt-; is ”

3. Opt-' is ’

4. Opt-. is …

Anyhow, here it is (with a nice icon even), download and expand it and go looking on the Web for instructions how to add it on your version of MacOS, it’s not rocket science.

Another nice thing about Ukelele: It’s ships with maybe the best How-To tutorial I’ve ever encountered in my lifetime of reading software how-tos.

Who am I? · When I set up the new machine, it asked me if I wanted to transfer from an older one and I said no; so it unilaterally decided that my Mac username (and home directory) would be “/Users/timothybray” as opposed to the previous “twbray”. Sort of dumb and clumsy but you wouldn’t think it’d hurt that much. Hah.

Arq hell · My backup, made with a copy of Arq 5 that I paid for, is on our Synology NAS. I had to fumble around a bit to remember how to turn it into an AFP mount, then I went to arqbackup.com and hit the first Download button I saw. Once I fired up Arq I started, at a moderate pace and step by step, to be driven mad. Because Arq flatly refused to see the network filesystem. I tried endless tricks and helpful-looking suggestions I found on the Web and got nowhere.

Here’s a clue: I’d typed in my license code and Arq had said “Not an Arq 6 license code”, which baffled me, so I went ahead in free-trial mode to get the restore running. Wait a minute… all the backups were with Arq 5. OK then. So I went and dug up an Arq 5 download.

It could see the NAS mounts just fine but things didn’t get better. The backup directory looked like /Volumes/home/tim/Backup and there were two different save-sets there, a special-purpose one-off and my “real” backup, charmingly named “C1E0FFBA-D68B-4B51-870A-F817FC0DF092”. So I pointed Arq at C1E0… and hit the “Restore” button and it… did nothing. Spun a pinwheel for like a half-second and stopped and left nothing clickable on the screen. Nor was there any useful information in any log I could find.

Rumor has it that I impressed my children with the depth and expressiveness of my self-pitying moans.

So I went poking around to look at the backup save-set that Arq was refusing to read, and noticed that its modification date was today. Suddenly it seemed perfectly likely that I’d somehow corrupted it and blackness was beginning to close in at edges of my vision.

[Yes, dear reader, the Synology in turn backs itself up to S3 but I’d never actually tried to use that so my faith that it’d even work was only modest.]

I finally got it to go. Protip: Don’t point Arq at your saveset, point it at the directory that contains them. Why, of course!

So I launched the restore and, being a nervous kinda guy, kept an eye on the progress window. Which started to emit a series of messages about “File already exists”. Huh? Somehow I’d fat-fingeredly started two Arq restores in parallel and they were racing to put everything back. So I stopped one but it still kept hitting collisions for a long, long time. Anyhow it seemed to work.

Now, it turns out that when Arq started up it pointed out it was going to restore to /Users/twbray where they’d come from, so I pointed it at “timothybray” instead.

Lightroom hell · I really care about my pictures a lot. So I fired up Lightroom Classic, it grunted something about problems wth my catalog, and limped up in pile-of-rubble condition. I have a directory called “Current” that all the pictures I’m working on live in before they get filed away into the long-term date hierarchy, and it seemed to contain hundreds of pictures from years back, and my date tree didn’t have anything after September 2017, and all the pictures in Current that I could see were from my phone, none from any of the actual camera cameras.

I tried restoring one of the catalog backups that Lightroom helpfuly generates but that wasn’t helpful.

I may once again have made more loud self-pitying noises than strictly necessary. I poked around, trying to understand what was where and absolutely couldn’t make any sense of things. Scrolling back and forth suggested the Fuji camera photos were there, but Lightroom couldn’t locate them.

Eventually it dawned on me that Lightroom expects you to have your photos splashed around across lots of disks and so in its catalog it puts absolute file pathnames for everything, nothing relative. So the photos were all under “timothybray” and the catalog pointers were all into “twbray”. So how the freaking hell did Lightroom find all the Pixel photos!?!?.

Since I’d made probably-destructive changes while thrashing around, I just Arq-restored everything to /Users/twbray and left it there and now Lightroom’s happy.

Was my trail of tears over? If only.

Blogging hell · The software that generates the deathless HTML you are now reading was written in 2002, a much simpler and more innocent time, in Perl; in fact it comprises one file named ong.pl which has 2,796 lines of pure software beauty, let me tell ya.

It uses Mysql to keep track of what’s where, and Perl connects to databases via “DBD” packages, in this case DBD::mysql. Which comes all ready to install courtesy of CPAN, Perl’s nice package manager.

You get all this built right into MacOS. Isn’t that great? No, it isn’t, because it plays by Apple rules which means that relies on something called @rpath. Which can go burn in hell, I say. The effect of it being there is that things that load dynamically refuse to load from anywhere but /usr/lib, and your modern MacOS versions make it completely physically impossible to update anything on that filesystem. So in effect it’s impossible to install DBD::mysql using the system-provided standard CPAN. There’s this thing called install_name_tool that looks like it should in theory help, but the instructions include poking around inside object libraries and life’s too short.

SQL hell · You might well ask “Why on earth are you using mysql anyhow in 2020? Especially when it’s a dinky blogging system with a few thousand records?” My only answer is “It was 2002” and it’s not a good one. So I thought maybe I could wire in another engine, and the obvious candidate was SQLite, software that I’ve had nothing but good experience with: Fast, stripped-down, robust, and simple.

DBD::SQLite installed with no trouble, so I handcrafted my database init script to fix some trifles SQLite complained about, and fired up the blogging system. Boom crash splat fart bleed. It turns out that the language mysql calls “SQL” is only very loosely related to the dialect spoken by SQLite. Which one’s right? I don’t fucking care, OK? But it would have been a major system rewrite.

Docker hell · So I took pathetically to Twitter and whined at the world for advice. The most common suggestion was Docker; just make an image with mysql and Perl and the other pieces and let ’er rip. Now I’ve only ever fooled around unseriously with Docker and quickly learned that I was just not up to this challenge. Yep, I could make an image with Perl and DBD::mysql, and I could make one with mysql, but could I get one to talk to each other? Could I even connect to the Dockerized mysql server? I could not, and all the helpful notes around the Net assume you already have an M.Sc. in Dockerology.

So this particular hell was really my fault as much as any piece of software’s.

Deus ex machina! · Among the empathetic chatter I got back on Twitter, exactly one person said “I have DBD::mysql working on Catalina.” That was John Siracusa. I instantly wrote back “how?” and the answer was simple: Don’t use the Apple flavors of perl, build your own from scratch and use that. All of a sudden everything got very easy.

1. Download the source from Perl.org. Since I’d filled up /usr/local with Homebrew stuff, I worked out of /usr/local/tim.

2. ./Configure -des -Dprefix=/usr/local/tim

3. make

4. make test

5. make install

6. PATH=/usr/local/tim/bin:$PATH

7. cpan install DBI

8. cpan install DBD::mysql

Then everything just worked.

Now, I have to say that watching those Configure and make commands in action is a humbling experience. In 2020 it’s fashionable to diss Perl, but watching endless screens-full of i-dotting and t-crossing and meticulous testing flow past reminded me that those of us doing software engineering and craftsmanship need a healthy respect for the past.

Was I done? · Of course not. A whole bunch of other bits and pieces needed to be installed from Homebrew and CPAN and Ruby Gems and so on and so forth, but that’s not a thing that leaves you frustrated or bad.

Boy, next time I do this, it’ll go better. Rght?

---

---

Contributions

---