I wrote a post in the Android-Developer blog today cautioning about the use of Content Providers that aren’t part of the published Android API. John Gruber pounced, deducing that, contrary to our stated policy, this constituted evidence of “private APIs”. Let me explain exactly what’s going on here.
The Policy · I think it was best expressed by Android platform engineer Dianne Hackborn in her Multitasking the Android Way post:
The available APIs must be sufficient for writing the built-in Google applications, as part of our "all applications are created equal" philosophy. This means background music playback, data syncing, GPS navigation, and application downloading must be implemented with the same APIs that are available to third party developers.
The SMS Application · It’s like this; Google built an SMS app that’s part of the basic application suite. It doesn’t use any private APIs. Anyone could write a replacement SMS application and deliver it via in the Android Market, and in fact this has happened, for example Handcent SMS. It’s perfectly possible that a handset provider might ship some other SMS app instead of Google’s.
The Android platform includes the SMSManager class for sending SMS messages, and you can set up a listener to receive them as well. Nothing private or secret.
The Google-provided SMS app has its own database that it uses to stash away the SMS history, and it sets up its own Content Provider for its own internal use. It turns out that if you read the source code, you can figure out how to reach in and access that Content Provider. Which is probably a bad idea, because it’s part of an application that might not even be there.
I personally think the benefits of an Open Source platform exceed this sort of cost — when someone uses the source to figure out how to do something that really isn’t very smart.
That’s the story. Does this constitute a “Private API”? Your call.
Contributions
Comment feed for ongoing:
From: B.J. May (May 06 2010, at 20:14)
Most certainly not a private API. It's quite the opposite. Android, in being open-source, has given developers enough leash to hang themselves with, and Tim has been kind enough to point this out as a warning. This is no more a private API than any other content provider in any other application. The difference is that the code of the core applications is observable, whereas the content providers in 3rd-party applications are not.
[link]
From: df (May 06 2010, at 20:14)
I'd say not. Seems perfectly reasonable to me. I quit reading Gruber a few weeks with all the ipad stupidity; haven't missed him.
I guess being a more public figure you don't have that luxury. Keep up the good work Tim! I'm looking forward to getting my Android phone this year and kicking my iphone to the curb.
[link]
From: Joe Wildish (May 06 2010, at 20:30)
Tim - I don't know why you feel the need to rise to anything Gruber has to say.
He is not qualified to comment on the matter. All he is is an Apple sycophant who is obsessed with self-aggrandisement. He is compelled to comment on anything and everything that he considers to encroach upon Apple (in this case, Android).
To make matters worse, he does so with zero humility and acts as though he is a sage regarding everything he makes comment on - see his recent "analysis" of the legalities regarding the 4g iPhone. What amazes me is that people still listen to him, despite his track record of being wrong, wrong, wrong. (Intel, AT&T, etc).
Who cares what Gruber thinks? The blinkered and the fellow-sycophants. Please don't encourage him by posting articles like this one.
[link]
From: Thomas (May 06 2010, at 20:37)
It's a private API the way that hardcoding a link to notepad.exe is a private API. That is to say, not at all. It's the equivalent of calling another program's library from your own. Labeling that a private API is ridiculous, especially since the SMS app in question only uses documented, public interfaces to create the content provider in the first place.
Gruber's just trolling as usual. I'm astonished you dignified it with a response.
[link]
From: Peter Keane (May 06 2010, at 21:17)
No of course not (a "Private API," at least in the bad sense). This conversation over what's "Open" and what's the "Web" has gotten a bit out of hand. That comment by Gruber is a real head scratcher. As if a program that uses an Open (probably well-documented) API is not allowed to use it's own encapsulated API somewhere lower down in its internals. That's basic good programming practices (and yes, you get extra credit for allowing someone to see what's happening for edification/ideas/"borrowing").
[link]
From: Al Sutton (May 06 2010, at 22:40)
Market is a better example of a Google non-core app which uses private APIs, and has done since Android 1.0.
Market can displays the requested permissions before downloading an app and then it can download & install the APK without the user confirming the permissions it needs.
If you try doing that using only the public APIs you can't, all you can do is download the APK and pass it to the package installer via an intent, at which point the user is forced to confirm the permissions.
If you look at the changes from API 3 to 4 (1.5 to 1.6) you'll see the public installation methods were actually removed which, in effect, took the only available API from public to private.
[link]
From: Peter Kirn (May 06 2010, at 22:54)
Tim, my vote is absolutely that this is not a "private API." I think it's fair what you're saying.
Now, at the same time - and I am biased, philosophically, toward Google's approach - I think there's no question that Apple is discouraging the use of private APIs for the same reason, at least in some cases. But there's a difference that Gruber and others are steamrolling over, which is that Apple isn't just discouraging the use of private APIs, they're actively blocking apps that use them. If Google started to do that on the Market, then the comparison would be fair.
Essentially, Google is saying we have the right to break our apps in this way, but please don't. ;) Or at least, be *aware* of where potential breakage could occur, and be smart developers.
Since there are numerous other ways to break your app on either platform, I think that's worth the cost. And indeed, because the platform is open source, it's possible for developers to submit patches and improve the platform with full knowledge of these APIs and the valuable Google apps that run on them.
Not to give Google a free pass, it then becomes Google's responsibility to document stuff where appropriate, and to incorporate useful patches into the resulting OS.
[link]
From: mtp (May 06 2010, at 22:55)
Tim,
As always, well-reasoned and -argued. Two-bit hacks and shills, beware! :-)
- M.
[link]
From: Gabriel K. (May 06 2010, at 23:18)
I use the ContentProvider of the SMS to retrieve messages and store them- as file that I upload on my computer. For the moment I do half of the work with a simple app and do the rest it by hand (I did not yet take the time to create an application that do the whole thing)
And yes of course I looked in the source code in order to do that. Reading that part of the code was one of the most interesting things I have done to learn Android...
Is it a private API? To me... No. If you publish a content provider... But this is a limit of the api of the content provider. You need to know what URI the content provider can parse, if you want to use the content provider. There is no "signature" such as with an interface or an ejb (which are, to me, other way some methods can be used by other applications). This is the difference between ws and Rest. But the Rest way is OK to me :) I prefer freedom and reading source code to constrained apis. I know there are some risks, but anyway....
[link]
From: zmarties (May 07 2010, at 00:43)
It's not only a question of whether the provided published APIs are sufficient to replace the provided applications that people care about, it's also whether the APIs provide the features that people expect.
As you found when writing Lifesaver, the ability to be able to backup and restore data off the phone is a serious omission from the APIs, and hence the absolute need to drop down to talking direct to the content providers to get at fundamental data.
[link]
From: maks (May 07 2010, at 01:42)
It hardly needed the clarification Tim, anyone who can actually read an english sentence and has the most basic understanding of programming would have understood that you were talking about the private data of apps instead of platform apis providing access to important hardware like sensors. Gruby Gruber is basically an apple apologist and troll and this isn't the first time he's quoted peole out of context.
[link]
From: Christopher Mahan (May 07 2010, at 01:57)
Can you say DLL Hell?
In any case, I think Content Providers should be versionable, so you could target a particular version. The problem then is that you might have to have more than one running if several installed applications are expecting different versions of the same Content Provider.
Like I said, DLL Hell.
I think maybe the Content Providers should attempt to be backward compatible, and also have an web uri where developers can check on latest goings-on.
Maybe such Content Providers should be released as their own app. Dunno what's best.one problem with that is that the user might look at this app and say: I don't know what this is... I'll uninstall it. BOOM, other apps don't work unexplainedly.
Now we're looking at program dependencies.
Is someone porting APT to Android?
[link]
From: Roger D. (May 07 2010, at 04:08)
Yes, it is a private API, but there's not anything wrong with that. A lot of APIs remain "private" until they are stabilized, and there is no better way to stabilize them than in real-world application use...
That said, why not open source the Google Android apps, since the bulk of the value is on the back-end servers?
[link]
From: J. King (May 07 2010, at 05:38)
You know, Tim, and I find your writings even more interesting now that you work for a corporation which has products I actually care about and find a little interesting. :P
[link]
From: Richard Sewell (May 07 2010, at 06:14)
To my mind, the SMS provider occupies an uncomfortable middle ground between private and public.
It's neither entirely secret nor entirely supported. Using it opens up some juicy niches for developers, so it has attracted apps, and some of them are pretty popular.
So far, the apps we've written that use it continue to work on all the devices we know about, and if the provider changes in future releases it will be worth our while to extend our code to work with it.
The big downside for us, as developers, is not really caused by the API being undocumented or potentially changeable (although both of those carry a cost).
The real downside is that there's no presumption in the API that there may be more than one app with a legitimate interest in the user's experience of SMS. For an example, look at the way that a Handcent user gets two notifications for every message until they go to the built-in app and turn off its notifications by hand. On a Samsung device, they need a completely different set of instructions to get there. If Handcent's experience is anything like ours, that is a major cause of support emails and user dissatisfaction.
So, one consequence of the semi-private nature of the SMS APIs is that we get apps which serve a legitimate-seeming purpose but which have to do their work in some ugly ways.
[link]
From: Marten Veldthuis (May 07 2010, at 06:29)
This sound more like "don't depend on libraries which may not be there, if you're not going to distribute them along with your application". That's no more an API for Android than some random header file you find on your Linux machine...
[link]
From: Eric Mill (May 07 2010, at 07:35)
Calling the database of an *app* on the phone a "private API" is an (understandable) misunderstanding of the Android platform. In the iPhone world, where the only APIs you could possibly call are from the system, anything Apple builds for itself is a "private API".
In the Android world, each app can make its own API into its data, that it can share amongst other apps. No app is under some sort of ethical obligation to make this data publicly specified - be it an app made by Google or a 3rd party developer. In this way, Google and developers *are* on equal ground.
[link]
From: Dave (May 07 2010, at 07:51)
The point is that there are no special platform APIs reserved only for the use of Google's apps, which would allow them to do things that others couldn't.
That shouldn't be confused with internal APIs of the platform (used only from within the platform, not any apps at all) or internal APIs of the Google apps themselves.
Total fail:
http://www.electronista.com/articles/10/05/06/android.not.much.better.than.iphone.for.apis/
[link]
From: Thomas (May 07 2010, at 09:04)
Interesting that Gruber has now linked to this, while cutting the bits indicating that it's not a private API at all. Also that he does not apologize for essentially accusing Andy Rubin of lying.
[link]
From: John (May 07 2010, at 09:23)
Public vs private APIs. Public vs private classes.
These are set up this way for a *reason*, having to do with reliability, supportability, and upgradability.
This comes down to much you do or don't care about impact to your customer. Do the private API thing all you want, but your customer going to be the one to suffer for your hubris.
[link]
From: Matt Kanninen (May 07 2010, at 09:33)
Responding to Joe: I've been an Android developer for a long time, and a mobile java developer for a long time before that. I'm pretty much a Google partisan, and it feels like Android was written just to make my life better. Almost against my will I did iPhone development for 1 day, and I've worked closely with a great many iPhone developers. I don't envy them one bit. When I write a line of code, I'm hoping it will be run a great many times, or it will be easily replaced by the next engineer, with little effort. I choose my platform, programming language, etc. with those goals in mind.
The iPhone is well optimized, but you have little hope of re using your code. Your code may be run many times in the next couple of years, but it has little hope of seeing most users. It is very hard to build an application that is not brittle, at the mercy of apples arbitrary whims, or that is simply to complex to be stable. I have seen how much work required just to keep an iPhone app running with a firmware update. I spend less effort delivering Android apps to whole new handsets!
I have high hopes that the Android applications I have already worked on will be able to run with little modification, for the fore see able future. Those apps could be run on many different kinds of hardware. The whole platform was clearly written for the future.
The iPhone OS is clearly a gimmick designed to help sell harshly limited hardware. Steve realized the importance of the application platform, and has invested heavily in it... but he will always sacrifice the developer experience in pursuit of hardware sales. He will gladly lock it down, close it out, and strictly limit what can possibly be done.
All this being said, I'm also a big fan of Daring Fireball, and even many of Grubers tweets. He is an Apple partisan, but he is also clearly a fan of Android. Reading him has helped me to appreciate what Apple has accomplished.
I don't agree with how they've accomplished it. They stopped making computers a long time ago, and started making toys. The iPhone wouldn't even fit the definition the word "smartphone" had before the iPhone came out! In my honest opinion it is still just a high end feature phone.
I think Gruber knows a lot of this. There is a reason he keeps watching Android closely. Heck there is a good chance he will re visit this comment page. It's the same reason Steve has grown frantic as all of his perceived enemies have started working together... on an open platform. It's the same way Apple was beat last time :)
[link]
From: jsd (May 07 2010, at 11:14)
@Matt Kanninnen:
I guess you should have stuck at the iPhone thing for more than one day. I've been doing it for over and year and I couldn't disagree more strongly with everything you wrote.
[link]
From: Brian Sniffen (May 07 2010, at 11:14)
Seems much like the instapaper and similar custom/private URL schemes popular on the iPhone.
[link]
From: Geoff (May 07 2010, at 12:36)
Peter Kirn notes: "Essentially, Google is saying we have the right to break our apps in this way, but please don't. ;) Or at least, be *aware* of where potential breakage could occur, and be smart developers.
Since there are numerous other ways to break your app on either platform, I think that's worth the cost. "
Perhaps to you. But it can help drag down the platform. Raymond Chen at Microsoft has devoted a whole series of posts on his blog, The Old New Thing, to examples of cases in which new versions of Windows had to be modified to accommodate popular apps which had used private APIs, hardcoded file paths or other programming foibles. Because, in the end, if you upgrade the OS and a piece of software doesn't work, it's the OS vendor that gets blamed, not the app vendor.
To its credit, Microsoft worked around a lot of improper behavior to make sure OS upgrades didn't break popular apps. This all has an obvious cost to development of the OS, though, and it's something that Apple is trying to avoid with its restrictions. Examples: http://bit.ly/9Vl5Ty http://bit.ly/9M3YSU
[link]
From: Rachel (May 07 2010, at 15:52)
Conceptually? Yes, it's a private API.
Certainly there are technical/implementation differences -- Apple's private APIs allow their onboard apps an advantage over third-party apps, which is unfortunate, where in theory any third-party app /can/ do anything an Android app does -- but in both cases it's part of something that 'comes with the system' and which developers want to get at.
People want to get at the SMS data. The 'ideal' way to do this without the aforementioned hassles of duplicated notifications and such is to deal with the SMS app's content provider. This is possible, but -- as was pointed out initially -- not recommended, because that API (and I use the term loosely here) is not guaranteed not to change. And the SMS app is viewed by many as 'part of the system,' rather than just another app.
On Apple, people wanted to get at the raw camera data and do video. The only way to do this was with private APIs. Now, whether or not we BELIEVE them, Apple has said that private APIs are private because they are not guaranteed to be supported or remain unchanged. (And we DID see a number of apps break in past updates of iPhone OS, when private APIs they had been using went poof.)
The situations are very similar conceptually, regardless of what you call it: in both cases, you're dealing with functionality which comes bundled with the OS and yet is not part of the implicit 'we will maintain backwards compatibility and try not to break your apps' promise of real public API, but which some developers will want to use nonetheless.
Now, in both cases, the companies have shown a willingness to take 'private' API that becomes in-demand and push it public in future revisions, committing to supporting that particular interface. Apple did so with the aforementioned camera APIs, Google addressed many complaints about contacts limiations by expanding the contacts API for Android 2.x, etc.
So far, there's a lot of similarities. Now we come to the big difference: the handling of those APIs before they get pushed public.
Google simply goes 'okay, well, that could break, but in the name of developer freedom you can use it; you're just on your own if it does.' Apple goes 'when those things using private APIs broke during past upgrades, there was a lot of outcry. So to prevent that, we're just going to ban anything that's using private API from our store, in the name of a more solid user experience.'
There's lots of nitpicky details you could get into, because there's certainly not a 100% mapping. But in the end, the real difference that /matters/ is that one right there.
[link]
From: Relwal (May 07 2010, at 18:51)
It's not just OS updates that can break private interfaces overnight.
Let me tell you a true story how a private protocol stopped working on the iPhone platform.
For about a year and a half iPhone apps could access the DCIM folder where photos where stored on the iPhone. Apple's rules, however, specify that iPhone apps may only read and write to their designated file storage area. This rule is mostly enforced through the iPhone file system. But the DCIM folder was not protected because the photo roll framework needed to have read-write access to the DCIM folder. So programmatic access to the DCIM folder was not locked by the file system. Likewise desktop software could access the contents of the DCIM photo folder on any iPhone over USB cable using the standard PTP file transfer protocol widely used by digital cameras.
Word got out that this "secret passage" existed and that iPhone apps that were desperately looking for ways to share data with each other and with the desktop could use it to get around their sandbox limitations. iPhone apps like Stanza, GoodReader, Noise.io synthesizer and Beatmaker all used the photo folder as a backdoor to transfer large files between apps and between the desktop and apps. Desktop-hosted utilities like iPhone Explorer and DiskAid employed the complementary piece of the functionality.
Nothing ever broke. No user's security was compromised. No viruses or Trojan horses were introduced into the system.
What happened? Apple found out.
Apple eventually caught on and started demanding that apps remove this functionality. As developers submitted updates to their apps over the course of several months the app updates were rejected or suspended until the developers removed the functionality from the iPhone apps. Likewise desktop utilities were also modified to remove the functionality motivated by methods unknown.
Users expecting to be able to share music samples between their music apps found they could no longer do so. Users expecting to be able quickly upload large ebooks and PDFs from their computer to their iPhone found instead that the capabilities they had grown used to were broken overnight.
The DCIM backdoor is still there. PTP still works to move files back and forth from the desktop to the iPhone. Apps can still read their photo files and old apps that were never updated can still read and write non-photo files from the photo folder.
Do any App Store-vended apps dare use that backdoor any more?
No.
That is how use of private interfaces can break the user experience on the iPhone platform.
[link]
From: Mike (May 07 2010, at 20:47)
The great thing about this post was that it was short, with a single, clear example. Even a borderline techie like me can understand it and get the point.
[link]
From: Giuliano (May 08 2010, at 04:59)
I have to say I find the comments on this thread a bit childish. This is just a clarification between two writers that in the eye of the readers become a sort of holy war between os. I follow both the weblogs and I am a partisan of nothing. Some day ago Gruber quoted Bray saying:
<quote>So, go ahead and look at the undocumented Content Providers; the code is full of good ideas to learn from. But don’t use them. And if you do, when bad things happen you’re pretty well on your own.</quote>.
Now, to me it seems a bit the description if not of a private API, of why is better don't use private APIs. Simply i think at google they are called with another name, and that the reason is pure marketing. Apple make is fortune on quality and security control of his platform. Google respond with openness. So always as quoted by Gruber Andy Rubin declare to the NYT:
<quote>A lot of guys have private APIs. We don’t.</quote>
A clear opposition to the stringent control Apple exercises on his platform and developers. Not a technical one, a marketing one.
Moreover Tim Bray responds not that the Content Provider isn't a private API, but why it is accessible. And you all start this iPhone OS vs. Android war.
What I see is that you can use private APIs on Android and iPhone OS, but with iPhone OS Apple will not publish your app if you do. With Android you can have your app on Android store also if you use these APIs. BUT Android developers are the first to say it is a stupid thing to do (using the API). In the end is more of a semantic and marketing difference than a difference in openness or a technical one.
To answer to Bray's question: I think that developers think this is a good trade-off (making their life easier) but from a user point of view Apple solution is better.
[link]
From: sandifop (May 08 2010, at 05:27)
I am not qualified to swim in these waters but, as a long time user and observer, I think the commenters are missing that Apple has had allowed use of, what is now called, private APIs for years. They have had the advantage of walking on both sides of the issue and find it advantages to label endangered APIs as private now rather than issue warnings.
As a user, I would rather not spend money for a piece of software that has taken ... what to say ... shortcuts(?) because I have seen these application suffer when an OS is updated.
[link]
From: Giuliano (May 08 2010, at 08:51)
Another thing to note (and who know both the stores also know it is true) is that quantity of crap present in android app store is in no way comparable with the quantity of crap in apple app store (both has but android store has way more). It seem shortsighted to me for developers to think all these freedom is totlly good because, in the long term, for users (the people that buy your applications) the results of apple control is a win: user has less risk to find and install crappy software and form in their mind a bad idea of the platform as a whole.
My main point remains that freedom as in "freedom as a selling point because our competitor selling point is quality-control " is not at all the Real Freedom. So I see no need for Holy Wars about that.
[link]
From: Mark Waks (May 10 2010, at 10:32)
Honestly, the "private"/"public" thing is a red herring -- the terminology is deliberately overheated, but there's a real issue underneath.
The reality is that they are effectively *unsupported* APIs. That is, they are effectively APIs -- it certainly sounds like people are treating them as such -- but Google is, for fair reasons, not getting into making any support promises. That's reasonable, but it's worth keeping in mind that it's essentially the same logic Microsoft has usually had. MS tends to be more draconian about it, and tends not to give a damn about the level-playing-field thing, but I suspect the real rationale is very similar.
So this is a tempest in a teapot, but it does bring up a very real point: that there *should* be movement towards a supported API for this. Seriously: if lots of developers are finding a need to use an unsupported API, that indicates an architectural flaw.
So my suggestion to Google is to ignore all the heat and red herrings, but try to move towards some community consensus about what these over-used Content Providers should look like in the long run, with an eye towards eventually turning them into a properly supported standard protocol.
(Yes, yes -- there's always the danger of the app not being present. Life is tough. But a community-agreed API would greatly improve the odds that, if you have a device with an alternate app for the purpose, it would support the same Content Provider format. Thinking about it as a protocol-standardization process is just plain more useful than all this private/public nonsense. And it conceptually maps nicely to the way standard protocols often do evolve, with proprietary ones eventually mutating into standards...)
[link]