Design and code reviews and attention to requirements, help develop a product, and make sure it has all the features you want in the product, but it doesn't do anything in terms of reducing bugs. Only testing can achieve that, which is why the aforementioned article says nothing about design and code reviews and attention to requirements. In short, design and code reviews and attention to requirements helps make sure the product will do what it's supposed to do. Testing makes sure the product doesn't do what it's not supposed to do.

Chris
2011-06-01 18:34:01

I'm not saying that testing isn't important. Just that an over emphasis on testing often leads to inattention to other quality practices that are almost always a better investment of time than testing is.

A properly performed code review will definitely find bugs, and it will be almost certainly more time-efficient than testing. My personal experience bears this out -- it's around 10x more effective.

You mention "reducing bugs", and in my experience missing features are generally considered bugs by users. Often adding a feature (or changing the way a core feature works) very late in the game ("Stage F" in Jesse's post) costs a lot of time and effort. The test team has to create or revise tests, the documentation team has to create or revise documentation, etc. Not to mention that if your sales guys have promised something to the customer by a certain date, and then that date arrives and the customer hasn't received what he was promised, he's going to be pissed. Some requirements work at the beginning heads off all of these costs.

A bad design (i.e. the kind of trash you get from interns and incompetent engineers) is going to breed bugs in the code. You'll find these bugs during testing -- TONS of them. But by the time you've gotten as far as testing, you've spent a lot of effort implementing a design that's doomed to fail. An hour spent reviewing a bad design, and sending the author back to the drawing board with things to fix, will pay off dozens of hours in coding and test. Anything from Jesse's "Stage C" or later is a horrible place to discover a bad design.

I'm not talking about theory, this is real world experience. Try reviewing your designs and code, keep track of how many defects you find per amount of time and how long it takes to fix each one, and then compare that to the same measurements for testing. Design/code reviews will win every time.

Brian St. Pierre
2011-06-01 19:05:10