This is astounding. Many software licenses, including some from Microsoft and Oracle—and in this biz, if those two are doing it everyone is—have language saying you can’t publish benchmarks or studies about the software without written permission from the vendor. Apparently, the US courts are giving this practice the treatment it deserves. I feel a severe flame coming on, but there’s some interesting background to look at first.
This came to my attention when I was told that a nice piece of performance analysis I’d read benchmarking some XML infrastructure couldn’t be published, because MSXML (the bottom of Microsoft’s XML stack) has this clause in its license.
I asked about this over on the xml-dev mailing list, and sure enough, here’s a quote from the MSXML EULA courtesy of Deepak Shetty:
c. Performance or Benchmark Testing.
You may not disclose the results of any benchmark test using the Product to any third party without Microsoft's prior written approval.
Spitzer to the Rescue · A bit of further poking around revealed that there’s been litigation on this. New York State Attorney General Eliot Spitzer, the same guy who’s been walking the Wall Street perps, took Network Associates (A.K.A. McAfee) to court over their EULA limitations and won, back in January. Network Associates said at the time they were going to appeal, but speculation was that they were more worried about the substantial penalties that might be due than over their right to suppress free speech. Mind you, the company kind of asked for it when they threatened Network World magazine with legal action based on the EULA when it was going to run a product review they didn’t like. (I’m beginning to think that Eliot Spitzer, he da man. Or at least da lawyer.)
I can’t find a status update on whether the company did appeal, and if so whether there’ve been any developments.
Why This is Bad · So, on the face of it, not a complicated story. It’s common for companies to want iron control over what’s written about them and their products, and they've been trying to slip this in under the radar in their licensing agreements, and they’re getting slapped down.
The reason this is bad is that we’re supposed to have a Free-Enterprise system here, and Free Enterprise is supposed to have information flow which is, well, Free, as in free to buy and sell and talk about what you buy and sell; didn’t someone say that Markets are Conversations?
This gets particularly intense where vendors are selling commodity products, like diapers or tires or XML processors; a rational customer doesn’t give a rat’s ass about what’s painted on the outside, but cares a lot about absorbancy or cornering forces or parsing speed. And, of course, price. If you can’t share information about these things, the market is broken. Imagine what would happen if Goodyear tried to keep Consumer Reports from writing about their tires.
A Slightly Different Angle · Joshua Allen wrote me with an interesting subtly-different example: TPC database benchmarks. TPC has all kinds of rules for the hoops you have to jump through and they have to approve before you can publish any kind of TPC results.
Upon reflection, this feels like a really different case, since the benchmark here is the product. I’d call this QA, not spin control.
Freedom is Slavery, etc. · All this would be a fairly typical case of agents of the state intervening to restrain self-destructive behavior in the market, except for much to my surprise, I ran across a few examples of people trying to defend this abominable behavior. Of course the people at Network Associates explained glibly that the rules were only there to make sure people got more and better information, and this was the general tone of the defenders: someone who screws up a benchmark could do real damage, and thus the community would be poorly served by such publication.
Here’s a quote from a Microsoftie who should know better:
The clause is there to avoid the classic problem of people disregarding all published best-practices, and write a test which shows that the product is X times slower than some other product, i.e. it helps protect against overly contrived benchmarks.
Horseshit. That’s what you may be saying, but what I’m hearing is “Freedom is Slavery.” I’m hearing the same tone that I hear from the governments of Egypt and Saudi Arabia and China, when they explain that they’re exercising iron control over the information their citizens are allowed to receive only because they want to ensure that it’s “fair and balanced.”
When a vendor tries to pull this stunt, it’s all about corporate paranoia and spin control, and that’s all it’s about. Yes, benchmarks are tough—I know, I’m the author of a fairly successful and widely-used benchmark, have to write about that here sometime—and indeed, it is 100% certain that someone will publish half-baked, shoddy benchmark results and cause damage. It’s also 100% certain that in every democracy there will be politicians making half-baked, shoddy statements and causing damage. And we just have to deal with it, because the alternative is worse. And it’s not so bad; the market, like the electorate, is in aggregate reasonably good in the long run at sorting out the wheat from the chaff. But only if they’re given a chance.
