Not a great launch. Wonder how many people with serious Web street cred are surprised? I’ll tell you: zero. But it’s amazing how many political commentators are suddenly overflowing with site-building chops.
In California, driving down the highway, I put the radio to scan and it pulled in some lively talk show, the man and woman were exchanging amazement over problems at Healthcare.gov.
She said “Mark Zuckerberg built Facebook basically overnight! What’s wrong with these cretins!” And I just about drove off the road; have been holding my breath waiting for someone who knows this biz to explain why it could never have worked, out of the gate. I haven’t seen that, so I’ll try.
Doing It Wrong · Not only Healthcare.gov; it’s just another example of what people in the biz call (shaking their heads, looking sad) “Enterprise Software”. Which every insider more or less knows, as I wrote in 2010, is just another term for Doing It Wrong.
Only, in this case, bigger and more expensively.
Unprecedented · So let’s see: A hundred million or so users, every dot and comma subject to eight layers of regulation, and nothing like it has ever been built before. The chances that the most elite squad imaginable of Googlers, Facebookers, NSA geeks, Government-of-China attack hackers, and Linus Torvalds, all laid end to end, could have made this work at startup? Zero.
The disciplines of Engineering are what keep you fed, clothed, warmed, and washed every day. They rank among humanity’s towering pinnacles of achievement. They prize (wait for it) repeatability, predictability, and hard lessons from experience. None of which this project has.
It’s like this: When you’re exploring unmapped territory, expect bumps and beasts and bandits.
Political problems · I don’t fully understand them, to be honest, although I’ve had a close view of a couple of colossal government-systems failures. Most experts agree that amid the general Doing-It-Wrong morass of Enterprise tech, the peaks of public-sector-software failure tower high, unequaled in their scale and awfulness.
It’s a combination of a culture that says all development must be outsourced not in-housed, and an ecosystem of government-RFP responders whose core competence is winning bids not building anything worthwhile. Since it’s in and of the government, by definition it’s a political problem.
Me, maybe I’m old-fashioned, but I think that large-scale information management ought to be a core competence of government, just like routing is for Fedex and searching is for Google; and they ought to hire (not contract, hire) the best people to build and run it, and pay them top dollar. They’d save boatloads of money over what they’re paying the bid-winning specialists.
Remember Obama For America? · Someone I read pointed out that Obama (from an IT point of view) was better at campaigning than governing. Well, the people behind the astoundingly successful and resilient Obama For America operation (now called Organizing For Action) — let’s call them the @Harper tribe — could surely have done a better job with Healthcare.gov. But on the other hand OFA didn’t have to worry about all those annoying regulations and (in particular) privacy rules, and in particular particular HIPAA; just saying that acronym makes strong men blanch and dive for cover.
But yeah; the problem is that the system as structured makes it impossible to get that tribe on your side, once you’ve actually won office.
Star Wars · The Strategic Defense Initiative, I mean. Call me mean-spirited, but I can’t help but remember how, thirty years ago, the foaming-at-the-mouth right wing, the same people palpitating in horror over Healthcare.gov, were cheerleading for a never-been-built-before, bugs-mean-millions-of-deaths, nuclear-war-promoting software system, to be built by the usual bid-winning experts.
Could Healthcare.gov work? · Sure. It probably will, eventually. And if the system had managed to break out of the outsourcing + closed-source + traditional-RFP trap, it might be already.
But I’m afraid that Mark Zuckerberg can’t help you.
Comment feed for ongoing:
From: Chris Swan (Oct 21 2013, at 23:15)
Many of the issues mentioned here are the US approach to government, which is sadly too broadly exported, rather than government per se. Israel and Estonia wouldn't have fouled this up, and not everything can be ascribed to scale.
The UK has recently been doing a great job of escaping from the bid winner oligopoly. Organic skill has been brought back in house, so the right things are being done, and they're being done right. Check out http://digital.cabinetoffice.gov.uk/
[link]
From: Dave Walker (Oct 22 2013, at 03:07)
Actually, while I don't have all the detail about what healthcare.gov was supposed to do, I suspect it isn't entirely without precedent; see the NHS SPINE project here in the UK, which was meant to do everything from patient healthcare record management to online GP and consultant appointment booking.
It didn't work, either (though it must be granted that a few point elements did), and was eventually cancelled after vast expenditure.
The NHS is now having another go at it, with "SPINE 2" - only instead of having a big SI marshalling a bunch of big vendors building big iron running big software, they've engaged a smallish consultancy / design / engineering firm who have chosen to build it all on top of Riak, instead of Oracle.
If you're interested in this area, it's worth looking up.
Naturally, one of the biggest problem was - and remains - lack of clarity around the security requirements. Probably the best piece of writing on the subject of what the NHS needs, security-wise, is the Caldicott2 report (from the head of its panel of authors, Dame Fiona Caldicott) - and even this doesn't cover the complete picture. Again, worth a look.
[link]
From: PeterL (Oct 22 2013, at 03:57)
You might enjoy Risks Digest:
http://catless.ncl.ac.uk/Risks/27.55.html#subj3
http://catless.ncl.ac.uk/Risks/27.55.html#subj4
and the articles they link to.
[link]
From: dr2chase (Oct 22 2013, at 05:28)
Facebook and HIPAA, there's an amusing juxtaposition.
[link]
From: Matěj Cepl (Oct 22 2013, at 07:00)
I am not a US citizen, and although I generally agree with your point (don't let me start on the Czech car registration system ... it takes couple of months to book for interview at DMV here now), would it be too unfair to cut some slack to the creators of the healthcare.gov? Couldn't we just admit that very slow gradual growth of (originally) really simple website like Twitter, Facebook, or Github, with no privacy, financial, or security considerations, with (originally) almost no users, is something very very different than this craziness which requires immediate perfection in many very complicated meanings of the word by sudden inflow of couple of million users at once?
[link]
From: Ray Paseur (Oct 22 2013, at 07:29)
And in current events, Verizon is brought in to "tech surge" the solution. I can't help but think of two important ideas -- Brooks Law, of course, and this gem from Edward Tufte: "Design recapitulates bureaucracy."
[link]
From: Ian Rae (Oct 22 2013, at 13:46)
>large-scale information management ought to be a core competence of government, ... and they ought to hire (not contract, hire) the best people to build and run it, and pay them top dollar.
Yes! I have worked at government departments and seen this up close. No one wants to touch any code because no one (that works there) understands how it works. The result is the opposite of DRY.
[link]
From: Mr. Roebuck (Oct 22 2013, at 22:44)
I used to work for a large American retailer and I can say that Obamacare looks like Facebook compared to their ecommerce backend. Our team spent about 6 months putting together a native iPhone and Android app and most of those months were spent reverse engineering our own company's API.
The API team was awful because they were in a different business unit and every moment they spent with us was a moment they weren't doing what their bosses wanted.
That would have been fine if the API was good, but the most important call had 45 parameters, most which we didn't know. And if you didn't put the call just right right it would drop you somewhere in the middle of the transaction. Fuck those guys.
[link]
From: len (Oct 23 2013, at 11:27)
"It’s a combination of a culture that says all development must be outsourced not in-housed, and an ecosystem of government-RFP responders whose core competence is winning bids not building anything worthwhile. Since it’s in and of the government, by definition it’s a political problem."
Amen.
The problem is not that it is crashing. Full-up tests at integration fail 9 times out of 10 with the exception being the Saturn V and that is because its designers had been together from Peenemunde to Huntsville and personally managed the fabrication of every piece for the first run. No the problem is a schedule that says if it isn't working three months after turning it on, it fails.
It's the talking head pundits who keep blaming a President who didn't have a chance that bother me. Sitting here writing a FOSI (yes, a FOSI) for a rather largish standardish set of enterprise XSDs that have the usual problems of over-design and under-specification (say bad stereo instructions), then looking at a web portal spec'd to integrate dozens of backend systems with different architectures, ages and support without load testing and ... managed by Canadians, I can only feel.... better. :)
I listened to the CNN experts truly ROTFLMAO. After all the finger pointing is past, it will be interesting to get a technical analysis of the architectural choices. Was it really "enterprisy" or did the identity system gum up processes? REST? SOAP? Spun out of navel? This one will be a classic because it attempted what has been promised to governments for decades now, a "frictionless system" and knowing why it is failing is well worth studying in detail.
[link]
From: Max Hadley (Oct 23 2013, at 12:42)
At a slight tangent, one of the best known and well liked upmarket retailers here in the UK is the John Lewis Partnership (http://www.johnlewis.com/) who run John Lewis department stores and Waitrose supermarkets. They are an employee-owned company. One thing that has always struck me is how nice-looking their point-of-sale system UI's are. I've never used one myself, but they look as though they are well thought out and pleasant to use. But of course the users (being the company owners) are the customers for this software and that always makes a difference. I don't know who developed this software, a subcontractor or in-house developers. It would be interesting to know.
[link]
From: Mike Perry (Oct 26 2013, at 19:59)
I Loved when I heard Rep. Anna Eshoo of California say "Amazon and eBay don't crash the week before Christmas"
Hey Anna, do some research before you show everyone your ignorance.
As recently as Christmas Eve 2012, Amazon AWS suffered an outage, bringing down Netflix for many of it's users.
Amazon built AWS based of years of experience with Amazon.com Christmas traffic, they didn't get their overnight.
[link]
From: J. King (Nov 02 2013, at 22:55)
I find myself wondering if, thirty years ago, hiccups in the system would have been news. I'm not American either (I've lived in Canada my whole life), so I'm not really familiar with the extent of the outages, but something always goes wrong with large systems, be they high-tech problems or something as mundane as a missing batch of pencils at a polling station.
The difference is that today we not only hear about such problems essentially instantly, anyone can -publicly complain- instantly, and other people can see these complaints, and gripe in turn and it all gets pretty visible pretty fast.
I expect soon enough it'll all be purring and in a few years hardly anyone will remember there was ever a problem.
[link]