Last year I built (and of course blogged) this nifty little Android app called LifeSaver, which would copy your telephone-call and SMS logs onto an SD card, so you could move the SD card to another phone, run LifeSaver again, and get ’em all back. Calls and texts aren’t migrated by the excellent Android backup system.
A few thousand people used it (I sure did, since I change phones all the time), and reviews are good. But it’s becoming less useful, because lots of phones these days don’t have SD cards.
A smart Googler suggested I put an App Engine back-end on it, so it wouldn’t need anything but a network connection. I did, and it works. But I don’t think I can publish it.
Quiz · Can you see why this idea turned out to be flawed? I’ll tell all in a few paragraphs, extra points if you’ve already said “That’s a horrible idea!”
Mechanics · With one exception, the change was almost stupidly easy to make. The app already knew how to translate the Android log data back and forth into JSON, just fine for pumping back and forth over the Web.
Then I had to write the App Engine back-end; I’d never done one of those before. The main options for that are Python, Java, and Go. I do Java all the time and don’t particularly like it. I already learned Python once, partly. Learning Go would be a Personal Growth Experience, and what with App Engine and Android Authentication, this project already had two of those.
So I didn’t do any of those things. I used Sinatra on Jruby on Java on App Engine. Because I miss doing Ruby at work.
That stack isn’t an official day-job project for anyone, and the pieces don’t always fit together 100%; you’ll need to use rvm, and then I had to downgrade gem to get Bundler running. But once things were in place, it all Just Worked.
App Engine · I’m no App Engine expert so I don’t have a useful opinion on what kinds of apps you should and shouldn’t use it for. But I can report that for this project, it was just the ticket. I want to do basic CRUD operations on tuples, my latency requirements aren’t very demanding (the saves and restores run in background on the phone), and I sure don’t want to invest any time or money in sysadmin-ing a server.
App Engine’s DataStore thingie met my needs exactly; the semantic gap between a JSON object and its notion of an Entity is pretty well zero. I have to say, the whole thing was a treat to write. Here’s the core code that handles the call log being uploaded from the phone:
post '/calls/' do
begin
if !check_user(request)
redirect @auth
else
request.body.rewind # in case someone already read it
data = JSON request.body.read
count = 0
data['calls'].each do |row|
count += Call.incoming(row, @auth).store
end
"Stored #{count} calls"
end
rescue => e
croaked e.message + e.backtrace.inspect
end
end
The Java code on the phone wasn’t nearly as pretty, but that’s just the way Java code isn’t. There was nothing taxing about the data fetching or pushing or pulling of course. So, slam-dunk, right?
Except For... · Identity and authentication. My thinking was, on an Android device there’s going to be a Google Account for sure, right? And those work with App Engine, right? So connect A to B and you’re done.
I was half right. On the App Engine side, it really is just that easy. In
that little code snippet above, see that if !check_user(request)
thingie?
def check_user(request)
if AppEngine::Users.logged_in?
@auth = AppEngine::Users.current_user
true
else
@auth = AppEngine::Users.create_login_url(request.url)
false
end
end
This Just Works and does the right thing, whether you’re coming in from a phone app or an actual browser with a human behind it.
On the Android-client side, I found the authentication libraries to be a little on the twitchy and subtle side, and the specifics of using them to get at App Engine were thinly documented.
But I figured that out and decided it was worth encapsulating the solution in a little library and writing about it, so stand by for that.
Free Web Sites! · Then it occurred to me that since I had this data there on App Engine, all I had to do was make an HTML version of it and I’d have a site for each user containing the history of their calls and texts.
Since the data was just a bunch of tuples and I could code it up in Ruby, it took approximately 15 minutes to get it on the air; then I invested extra time in making it look like the LifeSaver screens in the Android app. Here’s a screenshot; phone logs in the cloud!
By the way, have you figured out the Big Problem yet?
Know Fear · I got a few friends to dogfood the app, and as soon as the first one said “Hey, it works. That’s... weird”, I saw it.
If I put this sucker on the air, I would become the proprietor of a shared database containing not just “Personally Identifying Information”, but people’s whole bloody lives. A programming error could ruin those lives, because there are quite a few people who’ve made phone calls and sent texts that they’d really rather not share with the world.
And I suppose it’d be only a matter of time before I got a subpoena from some law-enforcement official tracking down Anonymous hacktivists or maybe even real criminals.
Having said that, I showed a draft of this to a smart Googler and she said “I wonder how bad/how much worse it is given that law enforcement can pretty much go to carriers today to get all the info anyway.” Maybe I’m overthinking this.
What Next? · I’m not sure, actually. If all else fails, I’ll open-source the code and then if someone else wants to run such a service it’s their funeral.
One possibility would be to encrypt the data using some info from the Google Account which would only be available to someone who’s actually logged in. Then the information would still be in the cloud, but not accessible to the database operator.
Another possibility would be to use the cloud just as a temporary staging service; you can upload your history but it only persists for 60 minutes or until you download it, whichever comes first.
Which means you wouldn’t have the nifty online website of your history; but I’m increasingly conflicted as to whether that’s a feature or a bug.
What do you think?
Comment feed for ongoing:
From: Phil Windley (Sep 26 2011, at 09:45)
One solution is to allow people to push it to Dropbox or S3. Private credentials, private storage. You don't ever have or control the data.
Still possible for a programming error to expose data, but much less likely.
[link]
From: Dirkjan Ochtman (Sep 26 2011, at 09:59)
As a heavy user of LifeSaver (that is, everytime I install a new ROM, which happens several times per year), this sounds like a great feature. But I'm sure time-limited storage would be Good Enough (TM). Perhaps 60m is on the low side, make it more like three hours?
[link]
From: Anton McConville (Sep 26 2011, at 10:15)
I really like your experiments and your writing.
I'm also a big fan of App Engine, and pushed myself to make the transition from Java to Python, when Python was the only option for it. In doing so, it really made Java feel bloated to me, and has forever changed my outlook.
A couple of thoughts. ( Your service looks great ).
One thought is to go with encryption, authentication and a carefully written legal agreement. The user then takes on the risk.
No matter what, you'd need some kind of legal agreement it seems.
Another thought is to somehow relate the App Engine service to a phone, or set devices - so the only way that anyone could access the data would be by some unique signature from a device or phone account, and a combination of authentication and encryption.
It would mean that the data could be looked at from a mobile app on the device, or restored to the logs on it. App engine would be just an 'invisible' staging area that only the user's devices could access.
Just thinking out loud.
I don't know enough about device/account uniqueness to know if that's even viable.
[link]
From: Dave Pawson (Sep 26 2011, at 10:25)
How about emailing it Tim?
Less (0bviously) open to abuse, if it is, then it's down to the sender?
HTH
[link]
From: Noah (Sep 26 2011, at 10:27)
If you forego the public profile bit, surely you can write some end-to-end encryption?
[link]
From: Baz (Sep 26 2011, at 10:54)
Firefox sync? Its open & encrypted end to end -unlike dropbox who have the keys to your data. Its a web service, the server is free to use, or you can run your own. You dont need firefox to use it.
https://wiki.mozilla.org/Services/Sync
[link]
From: Tom Jones (Sep 26 2011, at 10:56)
yeah, just encrypt it on the client and forget about the website.
use the cloud as a dumb blob storage..
[link]
From: D. J. (Sep 26 2011, at 11:05)
I'm sure other folks with chime in, but I think this is covered really well in Peter Wayner's excellent "Translucent Databases" book.
You would need a) your users to enter a good password to use as the basis for a crypto key, b) encrypt the JSON or specific fields in the JSON on the device (probably base64-encoding it for easy transit & storage), c) probably forego the web viewer, and d) do the corresponding decrypt on the new device.
If you'd really like to keep the web viewer idea, you can do in-browser decoding of the base64 and decrypting of the AES content. That way the "key" never hits your server, thus a subpoena would be nearly meaningless. There are a couple apps and some free Javascript libraries out there that can do the crypto/encoding work, e.g., http://www.movable-type.co.uk/scripts/aes.html
The important part is the "zero-knowledge" of the user's key, giving you plausible deniability and giving the user "assured delete" once they walk away with the key. There's a good discussion of this on the SpiderOak site (https://spideroak.com/engineering_matters#true_privacy) and why it's important that only the end user has the key... unlike the recent blowup about Dropbox.
There are also some interesting papers by Radia Perlman on "assured delete" and ephemeral key management... She gave a good presentation on the subject at a Sun CEC conference a few years ago. But I don't think your system needs to be that complex.
[link]
From: Boris Mann (Sep 26 2011, at 11:33)
As others have suggested, storing it in user owned storage is a good thing.
But, really -- the mass market will love it, and won't think about the security aspect, and won't have those other services.
It's just darn useful. And only if there is FUD flung about will the mass market even think about it.
For that matter, the people who sell cellphones and help people migrate will also love offering this to their customers. We can talk about this with my day job, if you like.
In Chrome, I have the sync service setup and use a pass phrase. I think that's an acceptable piece of encryption by the user.
[link]
From: Jason Rogers (Sep 26 2011, at 11:46)
I like the idea of encryption via some Google-esque attribute or set of attributes. I have not had an SD card for my phone for two years or more, so having this online would be great for me.
Of course, open-sourcing it and giving instructions to users for creating and linking to their own server would be fine as well. A new twist on viral: personalized app cloning instead of app usage.
Ooooo -- and if you could host a site with a link that would automatically set this up for an end-user, that would be ... never mind, that's overkill I guess.
[link]
From: Dirkjan Ochtman (Sep 26 2011, at 12:00)
Ah, reading the comment about Mozilla Sync made me remember that I thought about building something that syncs my call log/SMS messages to Mozilla Sync, too. It's an obvious good thing, managed by a non-profit and with full-on client-side encryption.
[link]
From: Kevin H (Sep 26 2011, at 12:07)
You mention in the lede that Android already has an excellent backup system. So I apologize if I'm missing something obvious, but why wouldn't you just implement android.app.backup.BackupAgent and let the operating system decide where that gets backed up to or restored from?
[link]
From: Hossein (Sep 26 2011, at 12:15)
I think about this question every day. I wonder if my pet project would ever be useful outside of the lab where we use it to understand smartphone usage. To see live data see the demo section: <http://systemsens.cens.ucla.edu/service/viz/login/#demo>
and here is a demo:
<http://systemsens.cens.ucla.edu/demos/demo2.html>
I have open sourced it already, both the client and server source are available but only researchers have shown interest.
[link]
From: Davanum Srinivas (Sep 26 2011, at 12:52)
Tim, Can you let users upload their data to their google docs? http://code.google.com/appengine/articles/python/retrieving_gdata_feeds.html
[link]
From: Justin White (Sep 26 2011, at 13:32)
Although many new phones (such as the Nexus S 4G I use) don't have removeable SD cards, the internal storage partitioned as the "SD Card" doesn't get erased under normal circumstances, including flashing a new ROM or restoring backed-up ROMs. I've gone from a Gingerbread to a broken CM7 (my fault), back to GB, to good CM7.1, and back to GB (for 2.3.7 to try Wallet) and not had anything happen to the "SD Card" storage area. It's also pretty easy to mount the USB storage and pull a copy of the Lifesaver-F folder just in case something does happen to "SD Card" storage, which is what I do with TitaniumBackup and some other app data, now including LifeSaver.
[link]
From: Jens Geiregat (Sep 26 2011, at 13:51)
If both phones can be booted during the transfer of the data, you could do this:
- Generate a secret encryption key on the sender side, save it on the phone.
- Send the encrypted data to the server
- When the time comes to transfer the data to the receiver phone, the sender phone displays the encryption key as a QR-code.
- The receiver phone scans the encryption key, retrieves the data from the server, decrypts it and stores it all to disk.
Upside: privacy
Downside: no website... But you could make the security an option.
[link]
From: Watching (Sep 26 2011, at 15:59)
Sell it to google!
[link]
From: Ross Reedstrom (Sep 26 2011, at 16:14)
Others have covered the technologically cool 'translucent database' option. From the 'end user utility' perspective, how about make it a plugin/app available to Google Sites - so the user has an authenticated way to dump there phone logs to their own personal site.
[link]
From: Murray Barton (Sep 26 2011, at 17:23)
I don't change phones very often but having my call log web or file system accessible would be useful. I like the Dropbox option someone else suggested but if your employer were to pick it up (GDrive? :) I'd still use it- I figure if Google want to know about me and who I call/text they already do.
[link]
From: Janne (Sep 26 2011, at 17:45)
Client-side encryption, definitely. And a second option to email the blob to yourself instead of a separate server. A third option to store in a specific folder (so you could drop it into Dropbox or Ubuntu One if you wanted) would be nicer still.
I can do without the nifty online view of my history. After all, that history is right there on my phone whenever I want to look through it — which is about, oh, never once in my life so far.
[link]
From: Rossi Kwan (Sep 26 2011, at 22:43)
Why don't put the data to client users' Google Doc, Gmail with specific label.
[link]
From: Ciaran (Sep 27 2011, at 06:57)
"My thinking was, on an Android device there’s going to be a Google Account for sure, right?"
Nearly always, of course. Not on any of mine though, just as a matter of interest.
[link]
From: Raymond Chenon (Sep 27 2011, at 16:05)
For the privacy problem , one solution is to store on one's own gmail account with a specific label marked as unread.
That's actually what the app SMS Backup+ does. And the code is open source. You can get them
http://code.google.com/p/android-sms/ or
https://github.com/jberkel/sms-backup-plus
[link]
From: Karl Voit (Oct 01 2011, at 02:00)
The simplest solution would be to email the (compressed?) databases to myself.
Also worth thinking about: after collecting the data, you provide an URL on the display showing http(s)://<myip>/<shortid> where I can download the package by myself from within the same network.
Keep the ID short for easy typing.
Additional: provide the URL also as QR-Code for my other phone if I can run them in parallel.
[link]
From: Jonas B. (Oct 07 2011, at 05:53)
I second the recommendation on SMS Backup Plus for this. It syncs call logs and SMS to an IMAP server (which could be GMail).
Here's a thought: If your phone was stolen, wouldn't the thief get your Google Account credentials as well? In that case that would not be a good place to store your backup.
Does anyone know how credentials are stored in Android? Is it a unique identifier just for the phone with limited rights or is your password in there somewhere? Can you, for example, change someone's password after stealing their (presumably unlocked) phone?
[link]
From: Steve Pomeroy (Oct 13 2011, at 13:13)
Encryption is the way to go. Create a key from the user's google account + a password that you tell them to NEVAR FORGET. You could even give them a randomly generated key or password that they're told to write down on a piece of paper or other non-phone storage medium. As long as that key doesn't automatically end up on any other cloud services, you're good to go.
That said, I wouldn't use it or trust it unless I could read the source.
[link]