His simple example is the 13-letter word “lexicographer”. To you and I (assuming you speak English and have a decent vocabulary), it is easy to remember. We don’t have to remember 13 letters, we just remember the whole chunk. But when presented with “trgdpxhdewfwm” for 3 seconds, you probably can’t remember more than half a dozen letters.

Another example is that chess masters can recall board positions after being shown a chess board with pieces on it for just a few seconds. They do this by chunking sets of pieces together — almost like “words” — whereas novices will try to remember individual pieces (“letters”).

It struck me that programmers do the same thing when reading and writing code.

The coding standard helps us, by telling us where the chunks are and how to draw the boundaries between chunks.

If you have a coding standard.

And apply it consistently.

When you choose names at random, you destroy your short term memory. You become a crappy programmer, and you drag everyone around you down too.

Quick example in C. Assume you’re writing a small module for checking environmental status.

BOOL isOverTemp();
BOOL isUnderTemp();

int GetCurTemperature();

BOOL env_isOverVoltage();
BOOL env_isUnderVoltage();

Anyone writing this pile of gibberish should be fired excommunicated.

Why?

For starters, I just wrote it, and I can’t now remember which was camel case and which was underscored. Also: which one was abbreviated?

Before you object that this is a contrived example, two points: (1) yes, it is contrived, that’s the point of an example, and (2) I have worked with far worse on an almost daily basis — the example above is fairly tame.

On the other hand, if you know the coding standard has some simple rules — and they are followed uniformly — you can easily remember the function names.

1. Initial 2-3 letter module prefix (“env” for this example).
2. All functions are lowercase with underscores.
3. No abbreviations.

Then we have:

BOOL env_is_over_temperature();
BOOL env_is_under_temperature();

int env_get_cur_temperature();

BOOL env_is_over_voltage();
BOOL env_is_under_voltage();

Now, when you’re writing, reviewing, or maintaining code, you don’t need to constantly refer to the header or documentation to get it right. A simple three-line coding standard just boosted your memory capacity by over 643%.

(In the interest of full disclosure: I made that number up.)

I realize that implementing even this simple three line standard is controversial because all the camel case folks are pulling out big sticks and the underscore people are grabbing rocks. To which I say: just flip a coin and enjoy the extra brain power. Adopt a three-line standard, follow it, and save the curly-brace debate for the next major coding standard revision.

Through some odd coincidence these posts crossed my path recently:

• Check In Early, Check In Often [comments here are especially good]
• Best Practices for Version Control
• 5 SVN Best Practices
• Continuous Integration is an Attitude, Not a Tool

What I got to thinking about is that version control is one of the foundation tools in your development process. It enables many other processes and practices. The posts above make great points, but I’d like to put them in the context of the overall process. (I’ve also given some thought to the conflicting advice given, again in the context of the forest instead of the trees.)

1. Use version control! If you’re not doing this, you should get out of the profession now.
2. Check in often. This is frequently given advice, but I feel it’s often given out of context. Jeff Atwood includes a quote from AccuRev founder Damon Poole:
   My rule of thumb is “check-in early and often”, but with the caveat that you have access to private versioning. If a check-in is immediately visible to other users, then you run the risk of introducing immature changes and/or breaking the build.
   (My emphasis.) “Private versioning” means a private branch: you can check in without affecting anyone else (see #8). But even with a private branch, consider Salvatore Iovene‘s advice: “SVN is not a backup tool”. He makes the point that you shouldn’t check in before you go home just because it’s the end of the day. Instead check in when you have a logical unit of work (e.g. everything compiles, most of the tests pass but you have to make some experimental changes to finish the feature or perform debugging, etc.). The danger is that if you have to revert because you’ve made some disastrous changes, you won’t have a sane state to revert to. Final point: Salvatore also reminds us that some tools don’t commit to a central repository by default (git, darcs, etc), so checking in doesn’t replace a good backup.
3. Put everything in version control. This is a key enabler for continuous integration (CI). James Shore has several great posts on CI. I think he under-emphasizes the need to check in everything necessary to build. First, it should be easy for a new hire to pull down a source tree and immediately get a good build. It makes life easier for them and for you. Second, your CI server should be able to do the same thing. If someone upgrades one of the tools locally and the build depends on this then the integration build should break. The tool upgrade must be checked in so that everyone picks up the change.
4. Don’t break the build. [When I say "build" I mean "build and tests". It's unfortunate naming but I think this is the way the term is used by a lot of people.] This is a promise that the team has to make to each other: “I will not break the build.” What it means is that every team member agrees not to check in code to the trunk that does not build and pass all tests. If you have jerks on your team that constantly break the build, ask them to play by the rules, and if they can’t then get rid of them because they’re dragging you down.
5. Check in bite sized chunks. Too big and you’ve probably been working too long: did you go dark? Large checkins are also harder to review well, so quality will suffer. If a given feature is large, create a branch and continue to check in bite sized chunks (see #6).
6. Branch, but only when needed. If you don’t have bite sized chunks, or if more than one person needs to work on a feature that can’t go into the trunk, create a branch. When you have a branch, it is critical to update (or merge, or pull, whatever terminology you prefer) into the branch from the trunk on a regular basis. Integration will be a nightmare if you don’t. When the code on the branch is mature enough, merge (or push) it back to the trunk. (Tip: it doesn’t need to be perfect, it just shouldn’t introduce regressions into the trunk.) If more non-bite-sized work is needed on the feature, create a new branch and repeat the process.
7. Check in when you’re ready. But not sooner or later. This should be obvious, but some people don’t seem to get it. If you check in before you’re ready, you break the build (and your promise to your team members). This can stop your entire team dead in their tracks! If you wait too long, you’ve lost the opportunity to get rapid feedback on your work. This practice enables continuous integration. It relies on having work broken down into bite sized chunks (see #5).
8. Use private versioning. Private versioning means that each developer has his own branch (or even multiple branches). With a VC tool like AccuRev, this is automatic: every tree has an associated private stream (branch) on the server. With distributed VC tools like git, this is also automatic: everything is private until you push/pull to the central repository or another developer. It can be simulated with other tools but requires a bit more effort. This allows individual developers to maintain several versions of their work before they are ready to share with the rest of the team. It also allows a developer to keep a sane version of code before merging in changes from the trunk. (If the merge becomes a disaster, he can discard the merge results and go back to the sane version without much effort.)
9. Update/merge/pull regularly. This also seems obvious, but again not everybody gets it. I mentioned it above as part of branching, and it’s especially important if you’re using private branches. You must merge from the trunk to your private branch/workspace/sandbox regularly. If you have private versioning, this is low risk because it’s easy to revert a bad merge. (These two practices create a self-reinforcing virtuous cycle!)
10. Train your users. I’ve worked in shops where it seems like only half of the team understands anything beyond “checkout, update, checkin”. Everyone must know how to perform daily duties like checkout, checkin, update and merge from trunk, merge from private branch to trunk, how to navigate history, perform diffs, etc. Everyone should know how to perform less-frequently-needed duties like odd sorts of merges – cherry picking, etc, though this is maybe less important.
11. Use a common project structure and naming convention. Thanks to Anders Sandvig for this bit of wisdom. I wouldn’t have thought to include it in VC best practices but it makes sense to list it here. A common project structure makes it easier to add every project into the CI server. For example, I’ve used the CI server to enforce the rule that every project must provide a $BUILDROOT/build script that runs the entire build. (The server also enforced placing tests in $BUILDROOT/tests, list of images to be published in $BUILDROOT/MANIFEST, etc.) This also makes it easier to move staff from one project to another.
12. Enable checkin notifications. This is typically seen as an email from a post-commit-hook (or whatever your tool calls it). The checkin notification can also launch CI build. Ideally, the notification will tie each checkin back to whatever issue/defect/task/project tracker you’re using.
13. Write good checkin comments. This makes notifications worthwhile. Remember that VC is a communication tool as much as a time machine! I’ve had several experiences where reading someone else’s checkin comment has spurred a discussion that either causes them to rework their bug fix or it has caused me to change course on something I’ve got in progress.
14. Perform atomic checkins. – When it’s time to check in, don’t do ten different checkins for the ten different files you modified! They all belong together as a logical unit. This makes merging easier. It prevents the CI server from running multiple build and test cycles needlessly (and from reporting spurious breakages because it picked up half a change). This makes history easier to understand when others look at comments. This is part of training, and is reinforced if you expect all developers to merge regularly. (On the theory that merging is easier when checkins are atomic.)
15. [Advanced practice.] Use a holding area. – Instead of checking in to trunk, developers check in to a holding branch (call it “pre-integration” or something similar). You perform code review and run integration tests on that branch. When review has approved the changes and the tests pass, promote (merge) the change from the holding area to trunk. The rest of the team pulls (merges) from the trunk, which is guaranteed safe. I’ve also seen the use of a two-level holding area where the first level is for review and the second level is integration test. This is the default if you use private branches (#8) and perform code reviews out of the private branch before integrating (#16).
16. Perform reviews on checked-in code. This is so much better than manually gathering changes and emailing patches around. (Though tools like git have optimizations specifically for eamiling patches around.) This is easy if you’re using private branches since you can request a review before integrating. Some code review tools enable this.

Design Review

I do a handwritten (paper) design. I like to design offline — I find that this forces me to do two things. First, to slow down and get it right. Second, to stay at about the right level of abstraction instead of drilling down into too much detail. When I’m done the paper design, I review it against a checklist that I build from mistakes found in downstream phases. My design reviews catch about 70% of the mistakes I make while designing. I’d like that to be higher, but it is definitely worth taking the time to do it. This isn’t specific to Django or Python, I do this for all projects.

Written Test Plan

After design review is done, I write (on paper, informally) a unit test plan based on the design. This is only possible when the design is at a low enough level that you can know what the general flow of control looks like through each method. I aim for full branch coverage in my unit tests. This doesn’t mean it will be “fully tested”, but at least I’ll have exercised most of the code. I usually find (and fix) a couple of defects while writing the test plan. This is also something I do for all projects.

Code Review

After I’m done coding, and before I run any of the code, I print it out and review it offline. The following bash alias is helpful (prints 2-up):

# usage: py2pdf OUTPUTFILE INPUTFILE(s)
py2pdf ()
{
    out=$1;
    if [ -f $1 ]; then
        echo $1 exists, aborting;
    else
        shift;
        echo $*;
        a2ps -A fill -E -2 -C -o - $* | ps2pdf - > $out;
    fi
}

It’s controversial, but I like the advice from Watts Humphrey in the PSP: review code before compiling (or in this case, before running any unit tests since there isn’t really a compile phase). It forces me to be much more thorough looking for defects since I know that python hasn’t filtered anything out yet.

To review the code, I use my checklist which is built in the same way as the design review checklist. I also use the test plan, walking through the code with each test case to see if there are any defects that testing will expose. It’s better to fix these now instead of waiting for the unit test to find them. Two reasons: First, it’s faster because you can see what the problem is. Second, you may notice something that the unit test won’t catch for whatever reason — maybe you don’t end up including the right data pattern to catch the defect. Code review is a generic defect removal practice.

Unit Test – XHTML Validation

Again, this seems to be controversial though I don’t really understand why. Some people don’t think valid XHTML is worth much. I validate because I want to make sure that my generated content isn’t horribly broken, and — just like compiler warnings — you either validate cleanly or you don’t validate at all. As soon as you start ignoring “a couple of silly errors” everything goes downhill.

I wrote an XHTML validator middleware for Django but I haven’t yet used it on this project. Why not? Well, I was trying to use Dojo as a javascript library, and it’s not possible (or at least not easy) to write valid XHTML while using Dojo widgets. So I had to write some special code in my test harness to strip out all of the Dojo garbage and then validate the result.

I’m using jQuery now, I’m much happier, and I suspect I’ll be able to validate using the middleware on my next iteration. (Which involves ripping out the Dojo false starts.)

Validation is generic for any web project. The middleware is specific to Django.

Unit Test – Test Plan

I implement the written test plan using PyUnit, the Django enhancements based on PyUnit, and Selenium for browser-based testing.

My test plan usually generates a lot of test cases (a recent app had over 100 test cases for about 800 lines of Python, 700 of Javascript, and 200 XHTML). So I like to keep them in separate files in a “tests” directory in each app. One trick that I use is to implement a suite() function in my APP/tests/__init__.py. The suite function globs “APP_tc_*.py” and sorts them by time stamp, most recent first. This way if a test is failing I can make sure that it runs first when I fix the problem and rerun the suite. The other part of the trick is to add an optional argument to the globbing/sorting function that limits the number of test cases returned. So when I need to rerun a test a few times to fix a tricky problem, I don’t have to wait for the entire suite to run to get the results of the test-under-development. This is all highly specific to Django development.

Unit Test – Pylint

This is still experimental for me. Pylint generates a lot of noise. Most of what it reports is legitimate, but about 90% are minor (i.e. not-user-impacting) defects. I’ve had to spend a fair amount of time tuning the report parameters to stifle the noise. The most time consuming part so far has been filtering out results from the test code. I’d like to figure out a way to make pylint skip the test files but there doesn’t seem to be a setting for this. On the bright side, it nagged me into making some of my test code more maintainable. Obviously pylint is specific to python, I use other lint-like tools for other projects when the signal-to-noise ratio is high enough.

Unit Test – jslint

I’m a javascript novice, so jslint has been helpful in ferreting out problems that I would not have caught through testing. As I let the feedback from the tool and testing form better design and coding habits/patterns, the signal-to-noise ratio may drop so that this is less effective.

These bullet points — the core of his argument — are excellent (quoted here):

• Design quality is the most important factor influencing productivity in software development
• The things that obstruct quality degrade productivity
• The reductions in productivity over time that are typical to tool-driven software development are greater than what can be solved by tools
• The application of tools to these problems exacerbates the quality problem, creating a vicious cycle that accelerates exponentially
• Quality software design is the product of principles and practices, not tools
• The typical degradation in a software’s quality over time isn’t due to the nature of software, it’s due to the nature of the approaches we choose to develop and operate software

Full disclosure here: I am working on a set of software tools, and a big part of my consulting for software teams is helping them install tools to support their improvement initiatives.

I agree with Scott that good design is the root of all that’s virtuous in software. I guess I also agree that tools are often a distraction, though I think he’s drawing a direct line where the root causes go deeper.

Maybe it’s my embedded background, and the fact that I’ve worked on teams that were woefully under-equipped in terms of tools. (Or execs were sold a bill of goods and we got stuck with the resulting boat anchors and paperweights.) I just don’t see an over-emphasis on tools, maybe this happens in IT more.

What I do see are many opportunities for improvement.

• Designs can be better communicated. If designs can be communicated better amongst the team, they can be reviewed better, which will kill bad designs before they get coded — and the feedback will help developers improve their future designs. A tool for formatting and transmission could help here. But such a tool would only be helpful to a team that already has processes in place for creating and reviewing the design.
• Requirements can be managed better. Practices like Scrum seem helpful, but there are still challenges. Again, a tool can help here but only when the team has an understanding of how the tool fits into an existing process.
• Existing processes and tools are overly slanted towards managing code.
  By the time any requirement or design has been reduced to code, it’s way too late to fix it. Start working on improving quality earlier in the cycle, the effort will pay off in a big way.

Putting software tools in the hands of unskilled developers is like letting me go after wood with power tools: it’s an expensive way to make a lot of sawdust. Better to build skills with manual processes and then introduce tools when manual chores become tedious.

The main thing missing from the discussion is that there are better techniques for finding defects than testing. Like design and code reviews, and especially more attention to requirements. Catch defects as early as possible and reduce costs even further.

1. Get the requirements right. It’s so often repeated that it’s almost a cliché to say that requirements errors will cost 10x or more to fix during coding or testing. But it’s true, so I can risk repeating it here. Spend a little more time getting the requirements right, and you’ll get every second back, with interest during the testing phase. The risk here is overanalysis, but if you tend to be the impatient type — jumping into design or coding with very thin requirements — then you’re unlikely to suffer paralysis.
2. Get the design right. It doesn’t have to be an elaborate 500 page document. One of the best designs for a complex system that I’ve ever seen was a state machine described in a two-page document with a big matrix on the first page. Remember, a written design is a communication tool. Just like the rest of your software, it can be the simplest thing that will work. Marker on a whiteboard works, if your team decides that is an effective way to communicate. An extra hour spent preparing and reviewing the design will save at least two hours in testing. Really. I’ve seen too many designs that were rushed (and clearly broken upon review) because of an unexplainable urgent desire to get a document into review so that coding can begin. The state machine I just mentioned went through probably 20 man-hours of reviews in maybe three passes, and we must have found at least 50 defects in it. Those defects took a few seconds each to fix (some may have taken a few minutes), but if there had been code and tests based
   on this broken design it would have taken hours to fix some of them.
3. Get the code right. I used an Extreme Programming mantra above (“the simplest thing that will work”), but I think XP overall often leads to bad practices — primarily an overemphasis on testing. I agree that automated regression tests are a valuable tool. However, it is important to recognize that testing has limits. It takes an immense effort to achieve full branch coverage, and even this can’t be considered “complete” testing. Testing must be used in conjunction with other tools: code review and static analysis are two such tools for removing defects from code. And before you’re at the point where code is ready for review or even the compiler, take the time to consult a reference when you’re unsure how to use an API. I don’t know how many bugs I’ve fixed (including some of my own!) that were a few lines down from a /*TBD: how is this supposed to work?*/ comment. Instead of putting in that comment, take the time to figure out how it is supposed to work.
4. Get the tests right. Ever waste time on a bogus bug report? Testers telling you a feature is broken when they’re just doing the wrong thing? It’s not their fault if you failed to communicate how the feature is supposed to work. In the same way your test team should be insisting on some written communication from you about the feature set, you should insist on some written communication from them about what they’re going to test. Again, this “test plan” doesn’t have to conform to an ISO/IEEE standard that requires it be 1200 pages long and six inches thick. Like with the design, a whiteboard works just fine. Whatever communication method you use, if it helps head off problems then it is working. If you’re finding too many problems late in the game, then maybe you need to have something a little more formal — try posting short documents to an internal wiki.
5. Are you keeping score? A little bit of measurement can do great things for you. I track time, lines of code, and defects. I was reviewing one of my designs this morning and this data helped me decide: (a) how many defects I should expect to find in my design and (b) how long it should take me to find those defects. All because I have historical data that tells me how many defects I find per hour of design review, and how many defects I make per hour while designing. I also know how much testing time this is likely to save… so I don’t feel any particular rush to start coding from a buggy design. I’d rather get the design right first.

As I mentioned above, the risk to getting into the habit of waiting is that you spend too much time analyzing the current phase and don’t just get on with it. But if you start from good requirements, it should be obvious when you’re done with design and coding. And if you’re keeping score, you’ll know how much time to devote to reviews to keep them efficient.

Some of my best project experiences have been those that have involved the testers starting at the requirements phase. These guys have a different attitude and have asked some great questions that prevented us from going down dead-ends.

Great testers find and isolate really hairy bugs. Truly great testers not only find these bugs, but will often find a pattern of bugs and can, for example, point to the portion of the spec that the code failed to implement. They take a bigger picture view of the product and its environment and how to improve the quality by both preventing defects and figuring out how to detect defects during the test phase.

As Jason says, we should let these guys have a little more influence on the development process. For example, they should be insisting on design and code reviews, even if they don’t participate. Because when the development team is in solely in charge of the development process, these steps sometimes get skipped — because the job is a rush, or because the feature is small and doesn’t need to be reviewed. (Even though rush jobs are most in need of reviews, and small features always seem to take longer because they receive less care and attention.)

1. Turn on warnings.

This is a no brainer. Your compiler can tell you when you’re doing things that are dangerous. I’ve found countless bugs from compiler warnings. Sometimes the warnings are annoying because the compiler barks about legitimate code, but once you’re in the habit of coding to avoid warnings it really isn’t that bad. Plus it saves a lot of time in the long run.

2. Make warnings break the build.

After you’ve turned on warnings, eradicate them from your code, and then set up your build so that it fails if there are any compiler warnings. If you use GCC, this is as simple as adding -Werror to your compile flags. With other systems, you may have to parse the build output to search for warnings.

If you skip this step, warnings can creep back into the build unnoticed.

3. Review your code.

Before you even compile a change, manually review it. Spend some time looking at it. A thorough review takes more time than you might think. Five or ten minutes per line of code is not unreasonable. If you are using a defect tracking system, look through the history of defects you have made in the past to know what kind of defects you tend to create. Put these in your checklist. Review your code against the checklist. Then, for every function call that you’ve added or changed, look at the definition for that function and verify that you’re passing the right number and type of arguments.

I’ve worked on a few systems that are not consistent about using 0-based or 1-based arguments, even in the same module, so part of my checklist is to verify whether each argument is 0-based or 1-based. This part of the review has saved me from several embarrassing bugs.

4. Get a review buddy.

Connect with one or more of your peers to review each others’ code. Keep it informal – you won’t need to get management approval. Promise yourselves that you will promptly turn around requests for review. Tell your reviewer what kinds of mistakes you tend to make, and what kind of mistakes you already found in your own review. Over time your review partner will learn what your weaknesses are and can look for those types of defects. You should also seek and provide feedback on these patterns of mistakes to prevent them in the future.

It may be worthwhile to make a couple of different review partners. Different people will have a different focus, find different defect types, and it allows you to spread out the load. It also exposes you to different thinking styles as you read more code from a variety of people. Lastly, if one of your review partners leaves the company, you will already have an existing partnership that you can rely on.

5. Track all of your defects.
   • Date
   • Where found (file, class, and/or function/method)
   • How found (self, compile, peer, test, qa, field)
   • “Type” – some brief description to define the type or category of defect

You are probably already using a bug/issue tracker like Bugzilla or Jira. These are great for tracking issues found in QA or in the field. But they don’t really track defects. One issue in QA may be caused by several defects in the design or code, or one defect may cause several issues in the field. There is no 1:1 correlation.

These systems also miss all of the defects that you catch before your code gets to system test or released to the field. Set up a spreadsheet to track every defect you find in your self reviews, peer reviews, compile, and unit test. (You can track this in a notebook in a pinch, but sifting through the data is more tedious.) Keep track of:

Every couple of weeks, or once a month, generate a graph of the data. Look for files/classes/functions that generate a lot of defects, these may be candidates for rewrite since they are obviously hard to maintain. Look for defect categories that cause you problems. Brainstorm ten strategies to reduce your worst defect category. Implement the top one or two. Thank your review partner for finding so many defects. (Or, if one of your partners isn’t finding much, you may decide to send more “business” to one of your
other reviewers.)