Given typical defect densities, any non-trivial design or code is going to contain some errors. Even seemingly trivial maintenance fixes are likely to be defective.

It’s your job as a reviewer to find as many of these defects as possible. If you’re not finding defects, you’re wasting your time on reviews.

That “one simple step”? Remind yourself that there are almost certainly defects in the work product you’re reviewing, and then find them. It’s all about attitude.

Related posts:

1. Who Else Wants Better Short Term Memory? In “Talent is Overrated”, Geoff...
2. 3 Easy Ways to Stick to a Coding Standard When you’re writing python, you...
3. 9 “Must-Have” Tools for Software Teams The items below are useful...

Jason Cohen pointed to an article of his in which he said “List no items that can be automated.” (His emphasis, but I second the motion.)

In the context of the SANS paper, let’s look at five items that should be automated so that no human has to find the defects:

1. CWE-665: Improper Initialization. As the write-up for CWE-665 suggests, you could use a programming language that forces you to explicitly initialize all variables before use. Or, if you’re stuck with something like C, make sure you turn on the warnings in your compiler, or use a static analysis tool (e.g. lint) to verify that all variables are initialized before being used. Tools like Coverity can be very sophisticated in their static analysis.
2. CWE-119: Failure to Constrain Operations within the Bounds of a Memory Buffer. Same goes for this one. First, use a programming language that doesn’t require attention to every tiny little detail of string handling. Failing that, apply static analysis. It’s also worth performing runtime analysis (e.g. electric fence, Purify) with appropriate test cases to verify that you’ve avoided buffer overflows.
3. CWE-404: Improper Resource Shutdown or Release. Even garbage collected languages can have problems with resource release. First, some GC systems have problems with circular references. Second, GC systems are typically worried about releasing memory. You also have to worry about database handles, file handles, and sockets. Configure your static analysis tool to track resources like these so that you don’t have to do it manually.
4. The write-up for CWE-404 also mentions in passing that you should “wash your garbage before you dispose of it”. This is useful for two reasons: first, an attacker won’t have access to the contents of previously used memory, and second, it often makes debugging memory problems easier if you write a known value into freed memory. (I modified malloc() at my last job to write 0xcacacaca into freed memory so that we would know right away when someone stepped on a turd, um, I mean used freed memory.) Configure your libraries to shred objects before you free them and you won’t have to worry about it on a line-by-line basis in the code you review.
5. CWE-362: Race Condition. I’m typically worried less about security than the nightmare of isolating and debugging race conditions. I haven’t seen any tools that detect race conditions well, but Coverity does a decent job of telling you when you’ve mishandled a race condition in a couple of cases: first, it warns you when you fail to release a lock, and second, it warns you when you access a resource that is statistically accessed from within a lock. So you still have to beware of race conditions (at the design level is the bset place to find these), but there is an option for finding tedious errors in the mechanics of dealing with races.

And now I’m going to throw out everything above.  Remember that the tools are built to detect common errors. If it is really important that you find a certain class of bug, put it on your checklist. In fact, if it’s super critical, make it a checklist of length one. For example, I’ve gone over codebases with the single-minded focus of finding race conditions. When you perform a review like this, it’s amazing how many defects you might find in areas that you never suspected.

Related posts:

1. One Simple Step for Avoiding Shallow Reviews It's your job as a reviewer...
2. Who Else Wants Better Short Term Memory? In “Talent is Overrated”, Geoff...
3. An Interesting pid File Race ISC’s dhcpd uses this code...

Why A Review Checklist is Essential

Most (all?) experts recommend the use of a checklist for performing design and code reviews. It’s more effective to ask a set of questions about the item being reviewed than to simply stare at a bunch of code looking for problems. The checklist focuses your attention on aspects that are most important.

If you want to perform a broader review, develop two or three checklists with different foci and give the lists to separate reviewers. The defect lists from each reviewer will have less overlap than if you gave everyone the same list.

What Your Review Checklist Should Look Like

The following advice is partly based on advice from “Software Inspection” by Gilb and Graham. A good checklist is a list of questions. For practical purposes, I’m strongly in favor of a checklist that does not exceed one side of a printed page — at normal font size. For me this means a checklist of about 15 items, 20 as an absolute maximum. Remember, we’re focusing, which means looking harder for just a few types of items. If you’re using online checklists, I’d extend this to mean that it should fit on the screen without scrolling. Either way, I don’t think you should have more than 20 questions or so at the max. Fewer is probably better, more is too overwhelming.

Checklist items are yes/no questions. They should all be phrased such that a “no” answer means something is broken.

When reviewing, proceed through the entire item under review (the entire code listing or design) with each question. Don’t work linearly, this is a horrible way to review code. I prefer to review on paper, simply because I can make little notes and marks all over the paper as I go through it. For the first question, you might work mostly front-to-back. Then the second question might send you backwards through the code. After a couple of passes over the code you have some familiarity with it, so the third question might send you straight to a location that has potential problems.

How To Choose Questions for the Review Checklist

Since the whole point of the checklist (and reviews in general) is to find defects, the questions should focus on finding the highest value defects. Why waste valuable checklist real estate on a question like “Does the code conform to formatting conventions?” Duh. If your team has a formatting convention, and you look at a piece of code that doesn’t conform, it’s going to jump off the page and bite you in the nose. You don’t need a checklist question for this!

Back to the specific problem at hand: security errors. SANS has done a great job of sifting through tons of security errors and boiling them down to the top 25. We could probably develop a 100 question review checklist from their Top 25 list, but it would be too big to be usable. So let’s focus on the most important for our application.

For this exercise, our application is a blog like Wordpress but much simpler: no plugins, no comments, single author. This is not a hosted application like Blogger, so we are less concerned about vulnerabilities like cross-site scripting, SQL injection, and OS command injection. We do need to make sure that only the author is allowed to post, so let’s look at the “Porous Defenses” category.

• CWE-285: Improper Access Control (Authorization)
• CWE-327: Use of a Broken or Risky Cryptographic Algorithm
• CWE-259: Hard-Coded Password
• CWE-732: Insecure Permission Assignment for Critical Resource
• CWE-330: Use of Insufficiently Random Values
• CWE-250: Execution with Unnecessary Privileges
• CWE-602: Client-Side Enforcement of Server-Side Security

The entry for CWE-285: Improper Access Control lists the following for prevention and mitigation:

For web applications, make sure that the access control mechanism is enforced correctly at the server side on every page. Users should not be able to access any information that they are not authorized for by simply requesting direct access to that page. One way to do this is to ensure that all pages containing sensitive information are not cached, and that all such pages restrict access to requests that are accompanied by an active and authenticated session token associated with a user who has the required permissions to access that page.

Here are some straightforward checklist questions we can extract from that description:

1. Does every administrative page enforce access control on the server side?
2. Are unauthenticated (anonymous) users prevented from accessing administrative pages?
3. Is caching disabled for administrative pages?
4. Does every administrative page validate the session token properly?

Now you can go walk through your design and/or code to verify that you can say “yes” to each of these questions.

If you found this post helpful, subscribe to The Daily Build to get updates as they are posted. Thanks!

Related posts:

1. One Simple Step for Avoiding Shallow Reviews It's your job as a reviewer...
2. 9 “Must-Have” Tools for Software Teams The items below are useful...
3. Using Python’s ctypes to Call Into C Libraries The ctypes module makes loading and...

• Codestriker Open source, web based, written in perl I think.
• Code Collaborator Commercial.
• Review Board Open source, web based, appears to be python/Django.
• Rietveld Open source from Google. Guido van Rossum’s 2nd pass at a code review tool. You get three guesses on the implementation language…
• Crucible Commercial from Atlassian.

Bottom line: don’t try to do this by hand, you’ll be wasting a bunch of time packaging patches and tracking defects when a tool can do it automagically for you.

Related posts:

1. 9 “Must-Have” Tools for Software Teams The items below are useful...
2. Data vs Code I’ll take an array over...
3. If the comments are ugly, the code is ugly If the comments are ugly,...

I tweeted this article on Five Reasons to Do Code Reviews from CIO.com last week,and realized that there are much more than the five reasons they give. So I came up with 20 more over the rest of the day. This is a collection of those tweets, with a little more detail given here.

Code review reason 1: Feedback comes back fast enough to give you whiplash. Since code review is done after coding and before integration or system tests, developers don’t have to wait as long to get feedback on code quality. By providing specific, timely feedback, developers can adjust their coding practices to avoid common mistakes.

Code review reason 2: Find more bugs than testing alone. This is obvious: Testing finds bugs. Code review finds different bugs. When combined they find more bugs than you would otherwise remove before shipping to customers.

Code review reason 3: A million monkeys sometimes produce valid syntax, even if it doesn’t make any sense. Compilers can’t catch this. Sometimes you’ll make a syntax error that produces valid syntax so the compiler can’t catch it. The classic here is in C, = (assignment) versus == (boolean equality operator); in this case most compilers can issue a warning but there are other examples. A code review will catch this error quickly, but finding this during testing can be hard to track down.

Code review reason 4: Find and kill bugs where they live, instead of finding turds and having to track them back to the nest. When you get defect reports from a code review, you have in-hand the exact line number and problem description. When you get reports from system test or (horrors!) a customer, you get — at best — a reproducible set of steps to trigger the bug. It’s then up to you to work backwards to the actual defect in the code, which is often time consuming, especially since the code may have been written weeks or months earlier. (See reason #1.)

Code review reason 5: Indoctrinate the n00bs. Code reviews are an effective training tool. First, new software engineers who participate in code reviews get to look at code produced by other engineers and thus get a feel for the coding standard and documentation style. Second, inexperienced engineers will get feedback on their code before they have a chance to infect the system with inferior code. (Because we all know that the code we wrote 10 years ago probably sucks — unless we were fortunate enough to have it reviewed by more senior engineers. If you haven’t already been coding for several years, keep in mind that everything you’re writing today probably sucks but you won’t be able to acknowledge that for another few years; see reason #6.)

Code review reason 6: Uses developer egos to boost quality of code from the beginning. The theory here is that because developers know their code is going to be publicly scrutinized by a group of peers, they will take extra care in coding to avoid the embarrassment caused by having others find all kinds of bugs in their code.

Guest contributor @kyletolle: Code review reason [7]: Explaining code makes sure they understand it well. Will fix the “Uhh, b/c I always do it that way” remarks. After you have to answer this question a few times during a review, you’ll probably ask the question of yourself while coding. (See reason #6.)

Code review reason 8: Makes testing go faster. Heck, it makes the entire development cycle go faster. There are multiple root causes for this. First, since code review finds and fixes defects at the site of the defect (see reason #4), less time is spent fixing these defect. Second, fewer bugs are found by testers so they spend less time writing up issues and the engineering team spends less time tracking and managing issues. Third, testers don’t have to re-run regression tests so many times because fewer release candidates will be needed.. Last, it is less likely that testers will be blocked waiting for fixes to bugs.

Code review reason 9: Fewer angry phone calls means happier tech support staff. Proof: Fewer bugs in shipping code (see reason #2). Customers not disrupted by bugs as often. They don’t get angry as often. So when they call, it’s with songs of praise. QED. (Ok, maybe that singing part isn’t true, but you get the picture.) To prove this from another angle: if you’re spending less time fixing bugs, you can spend more time implementing features that customers are requesting.

Code review reason 10: (Anti-reason) You’ll pay more sales commissions because you’ll be selling more product…wait, that’s a good thing! When customers sing your praises because your product is so reliable, it’s easier both to sell to new customers and to sell more product to your existing customers. Sure, your sales commissions might go up, but I’ve never really heard anyone complain about this too loudly.

Code review reason 11: There are tools that reduce the annoyance factor of old school review processes. After I tweeted this, @kyletolle asked for some recommendations. I will follow up this post with several tools that I’ve found. Stay tuned. (Look at my timeline starting here for now).

Code review reason 12: There’s zero risk. Even not-super-effective reviews beat testing. If I spent a little time I could probably dig up some published research to back this up. But my personal data says my personal code reviews are about four times as effective as unit testing. So even if I perform a half-hearted review, I’m still going to be more effective than testing. (See reasons #2 and #4, also keep in mind reason #5 which doesn’t have an immediate measurable effect.) There might be data out there that says code reviews are a waste of time. Flying reindeer might also exist.

Code review reason 13: (Anti-reason) Testers bored from not finding as many bugs… Oh yeah! They could focus on stress/perf tests. This would be a good problem to have…

Code review reason 14: Reviewing code is good practice for reviewing code. You’ll get better. Your team’s review effectiveness will improve over time. This means that the long term risk of failure is even lower than the short term risk (see reason #12 for why the risk of failure is zero or very close to zero).

Code review reason 15: It’s a great way to build a really useful code review checklist. This is kind of stretching, a useful code review checklist doesn’t really matter to the customer. (I should probably come up with a couple of bonus reasons to account for the “artificial fillers”… Hey, I tweeted this during the 3pm slump before my afternoon cup of tea.)

Code review reason 16: Out on a speculative limb: better staff retention because developers spend less time fixing annoying bugs. I’m not sure about anyone else, but I’ve stayed at a job that was otherwise not-so-great because of the quality of the team I was on and the people around me. Being part of a well-oiled machine feels good. It’s motivating to go to work everyday knowing that you’re going to produce a high quality product and your coworkers are there to back you up.

Code review reason 17: Reduces schedule risk because product quality is higher as it enters downstream phases. Your testers won’t get blocked as much (see reason #8) and testing won’t drag out. Since you get feedback relatively early in the schedule (see reason #1) you have a good idea of how long testing will take.

Code review reason 18: Feedback (especially from peers) at the source level means you will stop making the same mistakes over and over. (See reason #6.) This also stems from the fact that each engineer will have data in-hand about the types of defects he makes and he can come up with ways to prevent this defect from happening to begin with. And this is really when the magic occurs: when you can start thinking about preventing entire classes of defects.

Code review reason 19: Improves the quality of documentation / comments as well as executable code. (See reason #7.) Developers will include better comments to avoid having to answer too many questions from reviewers. If reviewers are looking for inconsistencies between comments and code, this is extra motivation to make sure that code documentation is up to date. Lastly, I suspect that overly complex code will become scarce since this will be scrutinized and questioned more than simple code passages.

Code review reason 20: Customers won’t find as many bugs in their soup. This is maybe redundant to reason #9, but helping customers is why we exist, right? So I’ll allow this one to stand and — like I said above — I’ll work on a couple of bonus reasons to add to this list.

Related posts:

1. One Simple Step for Avoiding Shallow Reviews It's your job as a reviewer...
2. Data vs Code I’ll take an array over...
3. 9 “Must-Have” Tools for Software Teams The items below are useful...

The main thing missing from the discussion is that there are better techniques for finding defects than testing. Like design and code reviews, and especially more attention to requirements. Catch defects as early as possible and reduce costs even further.

Related posts:

1. One Simple Step for Avoiding Shallow Reviews It's your job as a reviewer...
2. 9 “Must-Have” Tools for Software Teams The items below are useful...
3. Open an SSH Tunnel in Four Seconds or Less As I mentioned in a previous...

1. Get the requirements right. It’s so often repeated that it’s almost a cliché to say that requirements errors will cost 10x or more to fix during coding or testing. But it’s true, so I can risk repeating it here. Spend a little more time getting the requirements right, and you’ll get every second back, with interest during the testing phase. The risk here is overanalysis, but if you tend to be the impatient type — jumping into design or coding with very thin requirements — then you’re unlikely to suffer paralysis.
2. Get the design right. It doesn’t have to be an elaborate 500 page document. One of the best designs for a complex system that I’ve ever seen was a state machine described in a two-page document with a big matrix on the first page. Remember, a written design is a communication tool. Just like the rest of your software, it can be the simplest thing that will work. Marker on a whiteboard works, if your team decides that is an effective way to communicate. An extra hour spent preparing and reviewing the design will save at least two hours in testing. Really. I’ve seen too many designs that were rushed (and clearly broken upon review) because of an unexplainable urgent desire to get a document into review so that coding can begin. The state machine I just mentioned went through probably 20 man-hours of reviews in maybe three passes, and we must have found at least 50 defects in it. Those defects took a few seconds each to fix (some may have taken a few minutes), but if there had been code and tests based
   on this broken design it would have taken hours to fix some of them.
3. Get the code right. I used an Extreme Programming mantra above (“the simplest thing that will work”), but I think XP overall often leads to bad practices — primarily an overemphasis on testing. I agree that automated regression tests are a valuable tool. However, it is important to recognize that testing has limits. It takes an immense effort to achieve full branch coverage, and even this can’t be considered “complete” testing. Testing must be used in conjunction with other tools: code review and static analysis are two such tools for removing defects from code. And before you’re at the point where code is ready for review or even the compiler, take the time to consult a reference when you’re unsure how to use an API. I don’t know how many bugs I’ve fixed (including some of my own!) that were a few lines down from a /*TBD: how is this supposed to work?*/ comment. Instead of putting in that comment, take the time to figure out how it is supposed to work.
4. Get the tests right. Ever waste time on a bogus bug report? Testers telling you a feature is broken when they’re just doing the wrong thing? It’s not their fault if you failed to communicate how the feature is supposed to work. In the same way your test team should be insisting on some written communication from you about the feature set, you should insist on some written communication from them about what they’re going to test. Again, this “test plan” doesn’t have to conform to an ISO/IEEE standard that requires it be 1200 pages long and six inches thick. Like with the design, a whiteboard works just fine. Whatever communication method you use, if it helps head off problems then it is working. If you’re finding too many problems late in the game, then maybe you need to have something a little more formal — try posting short documents to an internal wiki.
5. Are you keeping score? A little bit of measurement can do great things for you. I track time, lines of code, and defects. I was reviewing one of my designs this morning and this data helped me decide: (a) how many defects I should expect to find in my design and (b) how long it should take me to find those defects. All because I have historical data that tells me how many defects I find per hour of design review, and how many defects I make per hour while designing. I also know how much testing time this is likely to save… so I don’t feel any particular rush to start coding from a buggy design. I’d rather get the design right first.

As I mentioned above, the risk to getting into the habit of waiting is that you spend too much time analyzing the current phase and don’t just get on with it. But if you start from good requirements, it should be obvious when you’re done with design and coding. And if you’re keeping score, you’ll know how much time to devote to reviews to keep them efficient.

Related posts:

1. One Simple Step for Avoiding Shallow Reviews It's your job as a reviewer...
2. Insist on Automatic Tests At some point your team...
3. Who Else Wants Better Short Term Memory? In “Talent is Overrated”, Geoff...