There’s this conversation rolling around the developers’ world; it started with chapter in Coders at Work by the one & only JWZ, which I haven’t read, but which Joel Spolsky enthused over, praising the practice of “duct-tape programming”. Uncle Bob Martin weighed in. I have opinions and examples too.
First, Joel is right that you want to pay attention to what JWZ says, since he’s shipped more code that more people have used than any dozen or two average programmers. Also he’s funny. So I’ll buy that book.
Joel is right that complexity is your enemy. We’d all like to write code to a higher level of abstraction than, say, C and Unix system calls provide, but when you’re eight levels up the stack in something like Spring or Websphere, chances are one or two of the levels are going to generally suck, because that’s how life generally is. So it’s really sensible to worry about all those levels.
TDD I · Joel is wrong to piss on unit testing, and buys into the common fantasy that it slows you down. It doesn’t slow you down, it speeds you up. It’s been a while since I’ve run a dev team, but it could happen again. If it does, the developers will use TDD or they’ll be looking for another job.
In related news, I’m trying to learn Haskell, and it’s a good thing that I’m using the online Real World Haskell because I would have launched a physical book across the room during the extended snotty lecture about how those poor dynamic-language programmers have to use unit testing because they lack the miraculous benefits of strong static typing. Static typing is not a substitute for unit testing. Sheesh.
Uncle Bob piles on: “Programmers who say they don’t have time to write tests are living in the stone age. They might as well be saying that man wasn’t meant to fly.”
Problems and Solutions · But then he says (slightly paraphrased): “I think that any programmer that’s not smart enough to use templates, design patterns, multi-threading, COM, etc is probably not smart enough to be a programmer period.”
What?! My experience suggests that there are few surer ways to doom a big software project than via the Design Patterns religion. Also, that multi-threading is part of the problem, not part of the solution; that essentially no application programmer understands threads well enough to avoid deadlocks and races and horrible non-repeatable bugs. And that COM was one of the most colossal piles of crap my profession ever foisted on itself.
I’d like to agree with Bob’s basic argument, but I don’t think we’re in agreement on problems and solutions.
Consider the Passenger · Let’s go back to the issue of complexity and layering. My own most successful software productions have been down-to-the-metal stuff, usually in C, full of memory-mapping and byte-level I/O. But these days, you can get fantastic results with certain highly-layered approaches. Consider my recent write-up on Ravelry, which is built in Rails, itself a pretty high stack of abstractions. They’re running it via Phusion’s Passenger.
Similarly, check out some recent DHH tweets, which I’ll reproduce for convenience:
Basecamp is now handling more than 50 million Rails requests per week. We're peaking out at around 200 req/sec. Damn! (*)
Basecamp's average response time is 90ms and 87% of all requests finish in less than 200ms. Great stats thanks to jumping off virtualization. (*)
Basecamp runs on 4 dedicated Dell R710's (2xL5520@2.27Ghz CPUs with 12GB RAM) serving Rails through Apache/Passenger. Zippy bastards! (*)
Another Passenger success story. And what is Passenger actually? It’s an Apache module. For those who don’t know, an Apache module is about as close to the raw metal as you can get in Web infrastructure; uncompromising C code where you get to munge each request as the server works its highly-optimized way through the business of parsing and dispatching it and delivering the results.
This seems like existence proof of the assertion that layering is your friend (as in all that Rails machinery making Ravelry and Basecamp maintainable), but so is being close to the metal (Passenger’s C code greasing the path between the Internet and the app).
TDD II · If you read the comments around this debate, it’s increasingly obvious that we as a profession don’t have consensus around the value of TDD. Many of our loudmouths (like me for example) have become evangelists probably to the point of being obnoxious. But there remains a strong developer faction out there, mostly just muttering in their beards, who think TDD is another flavor of architecture astronautics that’s gonna slow them down and get in their way.
I think we need to bash this out in public. We need to organize a good head-to-head practitioners’ debate at a conference or on some good online property, and see if we can at the very least come to a good understanding of our disagreements.
Contributions
Comment feed for ongoing:
From: katre (Sep 25 2009, at 14:20)
The main point that people come aground on in TDD is that the benefits aren't immediate. They're down the road, next week when I refactor, next month when Bob quits and Big Customer reports a new bug and I've never run this code before. When people don't think about the medium-to-long term, they just see the short term, which is that any developer looks more productive writing code to ship than code to test.
[link]
From: Steven Cheng (Sep 25 2009, at 15:35)
Testing just makes sense, and is a part of making any serious product, software or not.
That said, TDD needs to be separated from just testing.
My experience was in a compiler test team. The developers write the design doc, design doc goes to the test team, test team develops a test plan, then writes tests. The developers write some unit tests of their own as basic sanity checks.
The tests did not drive the design.
Test Driven Design seems to mean that the tests are created first, or pretty close, and are used as a spec and design doc. I can only see this buying two things:
1. It's one way to make sure the tests actually get created.
I think there are other ways that are just as effective.
2. If you are creating software similar to something you have made before, tests first can let you blast out code until the tests pass. The tests will catch all the small errors here and there, and since you've done it before, there's not much of a chance of higher-level concept type bugs.
I can't see a "code to make the tests pass" approach drive the internal design of a database.
I can't see tests driving the design of an algorithm either.
Design, then test is standard in other industries and it should work for software too.
[link]
From: Danno (Sep 25 2009, at 15:37)
I think you have the meaning of the authors of RWH slightly wrong. They're not saying that you don't have to create automated tests when you use Haskell, but that they're of a more abstract nature (such as QuickCheck property tests) due to the significant class of state and value based errors that are eliminated by Haskell's purity and unique type system.
Note, for example, that there are, strictly speaking, no ways of casting a value, but rather functions that take a value of one type as an input and return a value of another type as the output.
To be sure, there is still a need for unit testing Haskell code, but if you're creating idiomatic Haskell, you should end up with a comparatively smaller volume of traditional unit tests than in other languages.
[link]
From: Sean Hogan (Sep 25 2009, at 16:34)
I think Joel was saying that if you've actually shipped working (mostly) code you have more cred.
He seemed to imply that a significant proportion of people with this cred are duct tape programmers (but not necessarily the other way around).
The main point seemed to be:
- don't feel bad because you don't use complex programming technologies.
- don't feel good unless you actually ship working code.
[link]
From: Michael Schuerig (Sep 25 2009, at 17:04)
From what you're writing, I'm not sure you differentiate (enough) between TDD as doing design by writing tests before and to facilitate writing the production code, versus TDD as writing unit tests before or after the production code.
I agree, that "the profession" is not unanimous on whether TDD-as-design is always the right thing. I strongly disagree that anyone needs to bash this out in public even more. The debate boils down to "Look, this works for me, you ought to do it, too, and I'm sure what you are doing is wrong."
Instead of bandying about strong opinions and egos, I suggest gathering evidence when, where, how, and why specific approaches work or don't work. It might even be possible that there's more than one way to develop software.
[link]
From: Wesley Walser (Sep 25 2009, at 17:33)
“I think that any programmer that’s not smart enough to use templates, design patterns, multi-threading, COM, etc is probably not smart enough to be a programmer period.”
I read his statements in exactly the opposite way that you paraphrased it. He actually tells a story about a group at Netscape over abstracting their code and getting nothing of value done. That story was meant to reinforce his view that people who over use design patters, multi-threading, COM, etc often don't actually ship products. This seems to be the same point that your arguing for.
Can you read that section through once more and let us know if you come to the same conclusion?
[link]
From: dantakk (Sep 25 2009, at 18:11)
TDD promotes coverage as a measure of the quality of a test suite. But programs with high coverage have other costs. The cost of producing the unit test suite is not insignificant. Suites with high coverage can approach the size of the tested code.
And not all the code in a project is of equal importance. Is testing that little logging function to the same extent as your core logic the best use of your resources? For a project with limited resources [1] the time saved by "getting it right the first time" (jwz) can be applied to other priorities.
TDD encourages writing the tests first, which works well if the app domain is well known. When you're prototyping something new there's often large scale changes to the code architecture. If you've been writing your tests upfront you now have to rewrite your tests. But if you'd waited for the code to stabilize you'd only have to write the tests once.
For many classes of application (e.g. graphical apps) the work required to instrument your code for testing is a project in itself. When test suite size and coverage become a metric of code quality developers can become distracted by juking their stats instead of focusing on delivering a product. And developers are always looking for a rabbit hole to jump down :)
Is TDD worth it? I'm sure it is in many cases but in my opinion you can't generalize about this kind of thing, it's project-dependent. From my own experience it's worked well for smaller projects but becomes more cumbersome as the project gets bigger. Judicious use of unit testing combined with conventional best practices has worked better for me, ymmv.
[1] I'm excluding OSS in particular here
[link]
From: Chris Jaynes (Sep 25 2009, at 20:00)
I'm glad somebody else was as concerned as I was about Joel's attitude toward tests in that article. I was all fired up to leave him a comment, but he doesn't allow comments. I guess it's time for me to set up my own blog!
Yes, shipping version 1 is a feature, but shipping version 2 is ALSO a feature!
Sure, a good programming team can squeeze out a first version of a working app without any tests, but sustaining that past the first version is pretty tough without breaking things, and definitely takes even longer with each version released.
Whether you use TDD or not, a good unit test suite definitely speeds up development (even of the first version). Any programmer who says otherwise probably isn't doing their testing right.
(And the odds are, if they haven't figured out how to get their testing right, they probably aren't writing very clean code, which makes the problem even worse, and adds even more time in the long run.)
Thanks, Tim, for sticking up for testing!
[link]
From: Fred Blasdel (Sep 25 2009, at 20:04)
Why do you think your TDD evangelism is structurally any different than the "Design Patterns religion"?
Also, using Apache in any form is in my mind as far as you can get from web-bare-metal without using Java (and the ASF has plenty of that to offer). Bare metal in this case would be using small single-purpose tools directly: Rack, mongrel, nginx, varnish, memcached, etc.
[link]
From: Daniel (Sep 25 2009, at 20:47)
Think of the security. Unit test can prevent regression but it can never found out new issue. "Duct Tape" / "Design Pattern" / "TDD" won't replace a knowledgeable software engineer.
[link]
From: James Abley (Sep 26 2009, at 04:02)
Everyone's selling something. Me, I'm selling the fact that everyone's selling something. Joel sells bug-tracking software. Does this perhaps conflict with having a customer-base that has well-tested, clean code that can evolve over time?
[link]
From: Martin Probst (Sep 26 2009, at 05:02)
“I think that any programmer that’s not smart enough to use templates, design patterns, multi-threading, COM, etc is probably not smart enough to be a programmer period.”
I think what people commonly mess up is the ability to understand a particular piece of technology or concept on the one hand, and the ability to build working products cost effectively out of it on the other hand.
I agree that any programmer should be smart enough to generally understand multi threading, COM, templates, and so on. And any programmer should be able to write small samples that use these technologies and work.
However building a large scale product using these technologies can be very difficult, and academics are prone to oversee that. Certain technologies like threading scale very badly to large software products.
My rule of thumb is that anything that is not obviously correct or wrong from looking at (more or less) a single source file will give you major problems in practical development. Code must always be obvious, otherwise you mess it up in the long run.
And this of course gets worse with subtle problems like threading or memory management that don't necessarily break 100% of the time.
[link]
From: Neil Bartlett (Sep 26 2009, at 14:27)
Static typing is not a substitute for ALL unit testing, but it is certainly a substitute for SOME unit testing.
There are simply some tests we do not have to write because the compiler catches the failure modes for us. The RWH authors are certainly not arguing that statically typed programs do not need to be tested -- see Chapter 11, "Testing and quality assurance" for extensive advice on testing Haskell programs.
[link]
From: JulesLt (Sep 27 2009, at 01:50)
A tangential question - in what way is multi-threading code conceptually different from making parallel (and asynchronous) HTTP requests, and especially if those requests can update shared resources?
(I say this because I've already seen my team have to deal with an HTTP race condition in an AJAX app).
[link]
From: Gabriel Cl. (Sep 27 2009, at 07:58)
To be fair, the RWH authors had to counterbalance extended snotty lectures about how those poor static-language programmers have to suffer the type system and lack the miraculous benefits of dynamic typing. Unit testing is not a substitute for static typing. Sheesh
:)
[link]
From: Preston L. Bannister (Sep 27 2009, at 10:08)
As to: "multi-threading is part of the problem, not part of the solution; that essentially no application programmer understands threads well enough to avoid deadlocks and races and horrible non-repeatable bugs", you would not by any chance be talking about GNU Horde?
How about if you understand threads well enough NOT to use them, except where there is a clear flow of control, and distinct logic?
Then again (as I found this last week) you can create undetected race conditions in a single-threaded application. Threads outside your application can cause you grief. (And in last week's case, only causes problems on a desktop with multiple CPUs.) Calling a system API can cause trouble.
[link]
From: Isaac Gouy (Sep 27 2009, at 10:13)
> TDD
afaict Joel Spolsky didn't mention TDD.
"One of the ironies of TDD is that it isn't a testing technique... It's an analysis technique, a design technique..." p204 Test Driven Development
(Also "Who is TDD intended for?" p200)
> I’m trying to learn Haskell
http://www.haskell.org/haskellwiki/Introduction_to_QuickCheck
> a good understanding of our disagreements
About unit testing or TDD?
Is there a trade off between productivity and lower defect rates?
"Trade-offs between Productivity and Quality in Selecting Software Development Practices" 2003 IEEE SOFTWARE
http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=411969483763E32FCB95EDDCDE72AEDA?doi=10.1.1.61.1633&rep=rep1&type=pdf
[link]
From: Hugh Fisher (Sep 27 2009, at 17:03)
One reason people like Joel (and JWZ) are down on test driven design is that they work in the field of commercial shrink-wrapped software. Read his "Five Worlds" article for an overview of how this is different to other fields.
For people designing software for end users, there doesn't seem to be any evidence that unit testing adds significantly to your chance of succeeding commercially. The mentality behind TDD may even be a hindrance.
Until someone can write unit tests that distinguish, say, between the iPhone and Windows Mobile, TDD is not a useful tool in this field.
[link]
From: Phil (Sep 29 2009, at 10:15)
If Haskell programmers don't need as many unit tests, it's because their codebases exhibit referential transparency and are easier to reason about. Static typing may contribute, but there's really nothing like an absence of side-effects to ease the pain.
[link]
From: Andrew (Sep 29 2009, at 15:41)
The notion of the pragmatic programmer is hardly new. In fact Joel has been banging on about this subject for so long now that although I completely agree with him it's getting a little boring.
In his duct tape essay he makes sure to call out certain aspects of software engineering that are firmly held articles of faith in some communities, which of course increases controversy value of the article.
Like some of your other commenters I think the pragmatic engineer assesses each situation and applies the right set of tools and processes rather than blindly prescribing/proscibing certain approaches. Making these type of critical judgements are how your experienced technical leaders earn their money.
For instance in my world of embedded realtime software development you'd lose development efficiency by applying TDD. The cross-compilation, realtime, hardware dependent environment makes TDD much harder to implement (not that it can't be done, we've had various fresh-faced engineers try over the years and our source code version tree is littered with the droppings of those attempts). Similarly, threads and a priority based scheduler is what makes it possible to meet hard realtime deadlines with software.
Like the famous interview question, "What is your favorite programming language?" the only answer is "It depends on the problem."
[link]
From: calvin (Sep 29 2009, at 19:58)
This might seem odd, but doesn't everyone test?
TDD approaches, design patterns, templates, COM are all great and wonderful. But in my experience, whenever someone starts talking that kind of language, it means they do not understand the problem space. And even more likely do not understand the users.
Here is what I find really interesting about this whole question. When programmers write a programs for themselves, do they use TDD, patterns, templates etc?
How careful are they with their inputs? Do they iterate/rewrite or do they fix bugs in their own code? Most do rewrites because they become better developers over time. As developers become better do they gravitate towards templates or do they gravitate away from templates?
Don't hire people who don't write code for themselves!
When a programmer writes for himself, he combines the problem space with his ability to express in code. In fact, most programmers write things for themselves that push the boundaries of what they can achieve with the language/environment in the problem space. It's more fun that way.
Who writes code for themselves in the "official" ways? I don't know anyone that does that.
When you get a handle on a problem, you naturally start asking questions about how to do something. A programmer who knows one language will try and solve the problem with that language. (when all you have is a hammer). A more experienced programmer asks more complex longer term questions about future dev and other issues.
What I have seen over and over is that some developers, especially if they have learned some new tool, or have drunk some kool-aid, recommend approaches because that is what they know, not because they understand the whole problem space.
This is JWZ's essential criticism of Collabra. Collabra applied what they knew to what they thought was Netscape's problem: developing the next version of the browser.
JWZ knew the problem space better. It was SHIPPING the next version of the browser. JWZ knew that shipping was important, but he also knew that you have to ship something that is actually better. (I wish MS knew that about vista.... )
What templates, or c++, or TDD or design patterns are for is making better software. JWZ knew, like a lot of us, that those things don't necessarily make better software. And do they have little to do with shipping.
That is Spolsky's main point. SHIP!
Now I'm going to get back to work writing some python code, because I have stuff that needs to get out the door and it's a lot easier to conceptualize, test, and develop in python.
[link]
From: Deepak Shetty (Sep 30 2009, at 19:51)
I should stay away from religious wars but
"Joel is wrong to piss on unit testing..<snip/> If it does, the developers will use TDD or they’ll be looking for another job. "
Two things here
a. Personally I believe TDD is flawed because developer's suck at testing. I realized this when I was reading a book on testing that asked me to note down all the test cases which I would write up to cover the case that given the length of 3 sides of a triangle determine if the triangle is valid. Now try this out on any developer. How many cases do they test out? It's no point specifying code coverage by the tests because they'd only implement the cases they thought of. If they don't validate negative numbers , there is no code for it and no test either. i.e. Developers will only test what they have coded for.
It's extremely essential that some tests are conducted by someone with expertise in the domain. TDD gives you a false sense of security. Look , all my tests run, no bug!
b. TDD in the sense write tests first, don't work for everybody. I can't do it. I do however write tests after I can see something working. also since TDD is considered an agile principle isn't the developers will use TDD or else! very un-agile? If TDD doesn't work for your team , perhaps you should use something that does?
[link]
From: David Terrell (Oct 01 2009, at 08:28)
While we're in the middle of reasonably describing technology in the context in which it was developed ("threads and C++ suck ass! in 1994..."), COM wasn't bad for a while either. It solved a number of problems -- multiple allocators/deallocators, type safe plugins, object discovery, and in a way that worked well in C or C++ on low powered hardware.
It didn't get awful until after the VB people got ahold of it ("variants ho!")
[link]
From: W. Lederer (Oct 05 2009, at 17:59)
Peter Seibel has a new post extending this conversation and bringing more points from the other chapters of the book at http://www.gigamonkeys.com/blog/2009/10/05/coders-unit-testing.html.
[link]
From: Curt Sampson <cjs@cynic.net> (Oct 06 2009, at 10:47)
I'm one of those agile/TDD/etc. guys; I've done several major projects in Ruby, and I'm currently doing one in Haskell. I recently presented a paper on this: http://www.starling-software.com/misc/icfp-2009-cjs.pdf
Section 3.2 briefly compares the amount of unit test code in a largish Ruby project and my Haskell project; the summary is that I write about a third the amount of test code when using Haskell (less now than in early 2009, actually) and achieve more confidence in the correctness of my code.
The difference is that with the type system I prove certain aspects of my code to be correct, rather than show via tests that it doesn't fail in certain specific cases.
So I would say that yes, a good type system does reduce the need for unit tests, and the RWH authors are right when they make this claim. That said, using Haskell has certainly not eliminated my need for unit testing.
If I had to go with a generalisation, I'd say you want to do a fair amount of TDD and automated testing. However, I can find plenty of specific situations where you wouldn't want to, and I'd bet that many TDD advocates, if confronted with these situations, wouldn't disagree.
[link]
From: M1EK (Oct 06 2009, at 17:18)
The TDD zealots, like the Agile-in-general-zealots, who imply that it is nigh impossible to create usable, reliable, software without doing it Their Way have a very very strong ability to deal with cognitive dissonance, as they build their wonderful app on top of a stack full of usable, reliable software that was built with Waterfall and without TDD.
[link]