The Daily Build

This post was inspired by recent experience with some horrible build scripts from the open source world — but I’ve seen enough in-house badness over the years that I wanted to establish some basic parameters for build scripts.

This is a really broad way to divide the world, but I see two important categories of build scripts. (By “build script” I mean Makefiles, SConsfiles, Ant files, and their ilk).

1. Build scripts intended for in-house use, never seen by outsiders.
2. Build scripts that you distribute to external users.

Both sets of build scripts are important. Let’s consider the build script (or Makefile system, or whatever you’re using) as a software system independent of the software that is being built. Its primary requirement is to reliably convert your source code into an executable. Any decent build script has numerous other requirements, most of which are probably implied. I have never seen a written requirement for the numerous build scripts I’ve written. I have only rarely heard them discussed — and never actually posed as “requirements”.

I have, always taken the following meta-requirements for granted. Based on my experience with some fairly awful build scripts, I guess these aren’t universally acknowledged.

• The build script should build all the “normal” executables with a single command.
• The build script must encapsulate all environment variables within the script. (This is a corollary of the above.)
• The build script should not unnecessarily rebuild source code in a directory tree that has previously been built. (This is make‘s raison d’etre.)
• The build script should be documented. It doesn’t have to be elaborate, but a five-line comment at the top of the script describing the available command-line variables would be nice.
• make clean or its equivalent must work reliably.
• It should rarely be necessary to run make clean.
• Bonus: the build should be parallelizable, to be able to take advantage of multicore machines and/or distributed builds.

Back to those two categories. If you screw up an in-house build script, it’s primarily your team that’s going to suffer. I’m not aware of any bugs from SQA or Customers on projects I’ve been on that were traced back to a build script, but I can see where it could happen. Mainly what I’ve seen is developers habitually wasting time rebuilding source that should not need to be rebuilt. The worst offender in this respect was a build script that forced a “make clean” at the beginning of every build. In the immortal words of Dave Barry, I’m not making this up. Really.

(Actually, I think it was even worse when I was dealing with a build script that required a “make clean” every time around but didn’t have it coded into the script!)

I can’t say I have a completely clean conscience in this area: I wrote a big pile of Makefiles for a past employer in which I unknowingly used features (bugs?) of a specific version of GNU Make that were “fixed” in a subsequent version. As time went on it became (a) a much bigger task to fix the Makefile code that used the “feature” and (b) harder to keep the specific version of GNU Make that worked with the Makefiles! I imagine they’re still stuck with this situation… sorry guys.

In the second category, when you toss a slapped-together build script to your unsuspecting users you do all of human society a disservice. Now they are not only wasting their time fighting with a bad build script, but they often have the disadvantage of being unable to contact the author directly for support or commiserate with other experienced users of the system.

So do us all a favor: if you don’t know what you’re doing when you start to write your next build script, please ask for help. Otherwise we’ll have to resort to some sort of professional licensing scheme for build script authors.

Thanks for your patience, and for coming back for a discussion of deploying our Django web app.

In case you missed any of the previous posts in this series, here they are:

1. Day One, Get Ready (Concept and prep)
2. Day Two, Mockups (Creating a design)
3. Day Three, Coding (Coding tests, views, templates, and models)

Pre-Deployment

First, we need to make a couple of decisions:

• How are we going to push updates to the live site: FTP, git, svn?
• How are we going to handle backups?

Version Control as Distribution System

In my case, I’m using svn+ssh to push updates. Notice that this does not require special setup on your server — you do not need to install the svn stuff that DreamHost or your host may provide. Just do svn init to create a repository on the server (outside the DocumentRoot!). Then point your development pc to svn+ssh://USERNAME@host.example.com/home/USERNAME/svn/PROJECT. (Git works similarly, no support from your host required except the binaries.)

In the directory on the host where you’re going to store your project files, point to the same URL. (You could use the file:///…/ url, but I prefer to avoid accessing the repo directly. Superstition?)

Now, whenever you make a change on your development system, just "svn ci" and then on the host “svn up” and restart your fcgi to pick up the new code. Presto! The live site is updated with your change.

Backups

You must have a backup strategy: Your app will have users. Your host’s disk will burp. Your users will hate you when the disk burps and you don’t have a good backup.

There are probably 374 different ways of backing up your Django app. The two major things you need to capture are the database and your code. If you are storing objects (e.g. uploaded files) outside the database, you’ll need to back these up too. The option I’m using is django-backup.

Pull the code from subversion into your project. Rename the directory to "django_backup". Add django_backup to your INSTALLED_APPS. Verify it works by running ./manage.py backup -c. Sanity check the backup by doing zless backups/*.gz. We’ll set up a cron job on the host to run this regularly when we deploy.

Deploy

I’ve previously written about deploying Django apps on DreamHost, so I’m not going to duplicate that here. Keep in mind that you want to use the version control strategy outlined above. Read that article, deploy your app and come back here when you’re done.

Post-Deployment

Backup

Let’s add that cron job we previously mentioned. On the host, run crontab -e. If you’re on DreamHost and this if the first time you’ve used cron, it will prompt you for an email address to send output to. Then it will dump you into an editor. (Side note: "joe" is the default. If you want something different, like vim, be sure that EDITOR=/usr/bin/vim is set in your environment.)

Set up a job something similar to the following:

MAILTO="YOU@EXAMPLE.com"

# m h  dom mon dow   command
4 2 * * * (cd /home/PATH/TO/PROJECT; ./manage.py backup -c --email=YOU@EXAMPLE.com)

This will run a backup every day at 02:04 (AM). You will get two emails: one with the output from the job, and one with the compressed backup file. (When you get to the point where your database backups are too big for email, you’ll need to find another strategy.)

Now we’ve got another problem to solve: we’re going to accumulate a bunch of backups on the disk. Let’s get rid of the old backups. This is pretty safe, since we’re receiving backup files via email. Add a cron job like this:

14 2 * * * (cd /home/PATH/TO/PROJECT;
    touch --date=`date --iso --date='10 days ago'` .backup.oldest;
    find ./backups/ -mindepth 1 \! -newer .backup.oldest -execdir rm '{}' +)

(Formatted here for readability — you need to put that all on one line.)

This will remove backup files older than 10 days. You could do this more concisely with a tool like tmpwatch or tmpreaper, but neither is installed on my host and this incantation should work on pretty much any flavor and installation of linux.

At this point we’re deployed and the majority of the work is done. Tomorrow we’ll take a look at some maintenance issues.

I recently found myself wanting to quickly figure out how far (both miles and driving time) it is to various destinations. At first I was keeping a Firefox tab open to Google Maps and kept keying in the address for the destination. Then I realized that I could make a “Quick Search” and get the info faster. Add this link to your Quick Searches folder:

http://maps.google.com/maps?f=d&saddr=Elm+St,+ Manchester,+NH+03101&daddr=%s&output=html

[Formatted here for readability. Put it all on one line with no spaces.]

Edit the properties so that the “saddr=” part is your start address. Change the keyword to “gm”. This is set to only output simple HTML (no map). If you want the full map, remove the &output=html on the end.

Now, when you need to get a quick idea of how long it is going to take you to get somewhere, open a new tab and type “gm city, state”.

Thanks for coming back for Day Three!

[Note: Sorry this post is a day late. It was all ready to go late yesterday, but some of the code included below triggered a bug either in WordPress or ScribeFire and the whole post got mangled. I managed to resurrect it today from drafts, and I think it's coherent, but if you find some problem with it please drop me a note.]

Progress So Far

Yesterday we built some mockups, just HTML and CSS — nothing active. Hopefully you’ve had a chance to run your mockups past a couple of people for feedback and ideas.

Armed with these mockups, we’re ready to get started coding. Today’s post is the longest in the series. Take it in chunks if you need to. (I didn’t write it all at once either.)

Foundation

MySQL

First, go to your web host’s control panel and create two MySQL databases for this project. I created “resumandb” and “test_resumandb”. This latter database is needed for running the built-in test system.

You’ll get an error/warning if the test database exists when you first run tests. However, the control panel needs to set user privileges and it seems like the only way it knows how to do this is by creating the database.

Set up the same databases on your local system:

bash$ mysql -u root
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1969
Server version: 5.0.32-Debian_7etch8-log Debian etch distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> CREATE DATABASE resumandb;
Query OK, 1 row affected (0.00 sec)

mysql> GRANT ALL ON resumandb.* TO 'resuman'@'localhost' identified
    by 'reallY_baD_9passworD!';
Query OK, 0 rows affected (0.11 sec)

mysql> GRANT ALL ON test_resumandb.* TO 'resuman'@'localhost' identified
    by 'reallY_baD_9passworD!';
Query OK, 0 rows affected (0.0 sec)

Notice that you don’t need to create the test database. As mentioned above, this will happen when you run tests.

Project Skeleton

What follows is a bunch of steps: do-this, do-that. At the end of this section you should have a runnable (but empty) project that is ready to start hanging functionality onto.

Then generate a skeleton for your project: django-admin.py startproject YOURPROJECT.

Then dive right in to settings.py and change a few things:

• Set the admin email to your email address.
• Set the MySQL database according to your web host’s settings.
• Timezone.
• MEDIA_ROOT, MEDIA_URL and variants — see note below.
• TEMPLATE_DIRS — see note below.

To make it easier to use the same settings file to test both locally and on your host, I add the following to my settings.py to set MEDIA_ROOT and TEMPLATE_DIRS:

import os

def full_path_to(path):
    '''This makes this settings file relocatable.'''
    return os.path.join(os.getcwd(), path)

MEDIA_ROOT = full_path_to('static/')
TEMPLATE_DIRS = (
    full_path_to('templates')
)

Then you just have to make sure to cd to the directory where your settings.py lives whenever the app runs (more on this when we deploy to the host). I find this easier than monkeying with PYTHONPATH.

Similarly, the following will change your URLs based on whether you’re running locally or on the host. (Just make sure you test for a directory that only exists locally! My paths are a little different on the host.)

MEDIA_URL = 'http://media.resuman.com/resuman/static/'
if os.path.exists('/home/brian/projects/resuman'):
    MEDIA_URL = 'http://localhost/apache2-default/resuman-static/'

ADMIN_MEDIA_PREFIX = 'http://media.example.com/admin_media/'
if os.path.exists('/home/brian/projects/resuman'):
    ADMIN_MEDIA_PREFIX = 'http://localhost/apache2-default/resuman-admin/'

Now it’s time to generate the app: ./manage.py startapp YOURAPP. Notice that the app name should be different from the project name (otherwise it gets too confusing later on).

Edit your YOURPROJECT/urls.py to include a reference to YOURAPP:

    (r'^funnel/$', include('resuman.jobfunnel.urls')),

Also uncomment the admin lines so you can use the built-in admin app.

Now it’s time to generate the app: ./manage.py startapp YOURAPP.

Now edit YOURAPP/urls.py — paste in the needed bits from the existing urls.py, but drop the admin lines and the include().

In your settings file, add the admin and admindoc apps and “YOURPROJECT.YOURAPP” to INSTALLED_APPS.

Under the YOURPROJECT directory, create a templates directory and a static directory. Copy your HTML files from the mockup to the templates directory. Copy the CSS file to the static directory.

My system is configured by default to serve from /var/www/apache2-default/, and we specified above to fetch static files from http://localhost/resuman-static/, so we need to do the following to make this possible (substituting your paths, of course):

bash$ ln -s /home/brian/projects/resuman/static /var/www/apache2-default/resuman-static
bash$ ln -s /home/brian/projects/django/git/django/contrib/admin/media/ \
    /var/www/apache2-default/resuman-admin

Restart apache.

Change directory to YOURPROJECT and ./manage.py syncdb; ./manage.py runserver.

Browse to http://127.0.0.1:8000/admin/. Log in, and you should see the admin app in all its glory. If the stylesheet didn’t load (ie. it looks really ugly), View Source on the page. At the top, find the URL to that ends something like …/resuman-admin/css/base.css. Copy-paste this into the address bar. It probably doesn’t load. Verify that it is the right URL — if not then change your settings.py to have the right URL base. If the URL is right, then you need to fix your symlink, server config, or permissions (I often get bit by having the wrong permissions).

Adding Some Meat

This wouldn’t be a bad time to push a snapshot into your version control system (e.g. git init; git add .; git commit -m'YOURPROJECT skeleton done').

Now we’re finally ready to write the first view for this app. Edit YOURAPP/urls.py. Lay out the URL map that you want to use for your app. We’re on a tight five day schedule, so don’t go nuts! There’s only time to get a couple of pages done. Don’t worry, you can add more later. Here’s what mine looks like:

from django.conf.urls.defaults import *

urlpatterns = patterns(
    '',
    (r'^$', 'resuman.jobfunnel.views.dashboard'),
    (r'^add/$', 'resuman.jobfunnel.views.add_job'),
    (r'^edit/$', 'resuman.jobfunnel.views.edit_job'),
)

Remember that my toplevel urls.py includes this based on the ^funnel/$ pattern, so each of the patterns above will have …funnel/ as a prefix in the URL.

Now let’s write our first couple of tests. Edit YOURAPP/tests.py and add something similar to the code below. You’ll have to change URLs and logins. The class ViewTestCase is defined in viewtestcase.py, a convenience TestCase subclass I wrote for testing Django views. Copy that file into YOURAPPNAME directory.

from viewtestcase import ViewTestCase
class DashboardViewTestBase(ViewTestCase):
    # Override in subclass to use post/head/etc. Must match a
    # method defined in django.test.TestCase.
    TESTMETHOD = 'get'

    # Override.
    TESTURL = '/funnel/'
    TESTARGS = {}
    TEMPLATE = 'dashboard.html'

class DashboardViewLoginTest(DashboardViewTestBase):
    # We're expecting an error, so set TEMPLATE to None to avoid getting a bogus test failure.
    TEMPLATE = None

    def test_login_required(self):
        """
        Tests that a login is required to view the page.
        """
        self.expect_login_redirect()
        return

class DashboardViewTest(DashboardViewTestBase):
    # This uses TEMPLATE from the parent class.

    # Set username and password and the base class will automagically login the test client.
    USERNAME = 'brian'
    TESTLOGIN = (USERNAME, 'a')

    def test_logged_in_ok(self):
        pass

Now run the test: ./manage.py test. You should see exactly two failures. If a bunch of stuff fails (like built-in django tests), fix whatever is wrong before continuing.

bash$ ./manage.py test
Creating test database...
Creating table django_admin_log
Creating table auth_permission
Creating table auth_group
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table jobfunnel_job
Installing index for admin.LogEntry model
Installing index for auth.Permission model
Installing index for auth.Message model
Installing index for jobfunnel.Job model
Installing json fixture 'initial_data' from '/home/brian/projects/resuman/../resuman/jobfunnel/fixtures'.
Installed 1 object(s) from 1 fixture(s)
..........EE......
======================================================================
ERROR: test_login_required (resuman.jobfunnel.tests.DashboardViewLoginTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/brian/projects/resuman/../resuman/jobfunnel/viewtestcase.py", line 98, in setUp
    self.fetch_view(self.TESTMETHOD, self.TESTURL, self.TESTARGS)
  File "/home/brian/projects/resuman/../resuman/jobfunnel/viewtestcase.py", line 79, in fetch_view
    self.response = function(testurl, testargs, **extra)
  File "/usr/lib/python2.5/site-packages/django/test/client.py", line 277, in get
    return self.request(**r)
  File "/usr/lib/python2.5/site-packages/django/core/handlers/base.py", line 77, in get_response
    request.path_info)
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 183, in resolve
    sub_match = pattern.resolve(new_path)
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 183, in resolve
    sub_match = pattern.resolve(new_path)
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 124, in resolve
    return self.callback, args, kwargs
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 136, in _get_callback
    raise ViewDoesNotExist, "Tried %s in module %s. Error was: %s" % (func_name, mod_name, str(e))
ViewDoesNotExist: Tried dashboard in module resuman.jobfunnel.views. Error was: 'module' object has no attribute 'dashboard'

======================================================================
ERROR: test_logged_in_ok (resuman.jobfunnel.tests.DashboardViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/brian/projects/resuman/../resuman/jobfunnel/viewtestcase.py", line 98, in setUp
    self.fetch_view(self.TESTMETHOD, self.TESTURL, self.TESTARGS)
  File "/home/brian/projects/resuman/../resuman/jobfunnel/viewtestcase.py", line 79, in fetch_view
    self.response = function(testurl, testargs, **extra)
  File "/usr/lib/python2.5/site-packages/django/test/client.py", line 277, in get
    return self.request(**r)
  File "/usr/lib/python2.5/site-packages/django/core/handlers/base.py", line 77, in get_response
    request.path_info)
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 183, in resolve
    sub_match = pattern.resolve(new_path)
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 183, in resolve
    sub_match = pattern.resolve(new_path)
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 124, in resolve
    return self.callback, args, kwargs
  File "/usr/lib/python2.5/site-packages/django/core/urlresolvers.py", line 136, in _get_callback
    raise ViewDoesNotExist, "Tried %s in module %s. Error was: %s" % (func_name, mod_name, str(e))
ViewDoesNotExist: Tried dashboard in module resuman.jobfunnel.views. Error was: 'module' object has no attribute 'dashboard'

----------------------------------------------------------------------
Ran 18 tests in 4.364s

FAILED (errors=2)
Destroying test database...

Let’s write the view so the test will pass. Edit views.py:

@login_required
def dashboard(request):
    return render_to_response('dashboard.html',
                              {'title': 'Dashboard'},
                              context_instance=RequestContext(request))

This view uses a template called dashboard.html. Let’s make that template. Go back to the mockup for the dashboard. Copy everything into templates/base.html, then rip out the content so all you have left is the generic skeleton of a page in your app. Something like this (notice that this is using the title variable passed into the context by the view):

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<!--
# Resuman: keep track of job applications, cover letters, and resumes
#
# Copyright (c) 2009, Blakita Software LLC
# All rights reserved.
-->

<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
  <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />

  <title>
    {{ title }}
  </title>

  <style type="text/css">
    @import "{{ MEDIA_URL }}base.css";
  </style>

  {% block scripts %}
  {% endblock scripts %}

</head>

<body>

<div id="container">

<div id="menu">
TBD
</div> <!-- end div=menu -->

<div id="header">
<h1>{{ title }}</h1>
</div> <!-- end div=header -->

<div id="content">

{% block content %}
{% endblock content %}

</div> <!-- end div=content -->

<div id="footer">
<div class="nav">
  <ul>
    <li><a href="/">Home</a></li>
    <li><a href="/blog/about">About</a></li>
    <li><a href="/privacy.html">Privacy</a></li>
  </ul>
</div>

<div class="copyright">Copyright © 2009, Blakita Software LLC</div>
</div> <!-- end div=footer -->

</div> <!-- end div=container -->

</body>
</html>

Now take the content you ripped out and put it into dashboard.html:

{% extends "base.html" %}

{% block content %}
<ul id="funnel">
  <li class="phase first">
    <div class="phase">Applied</div>
    <div class="companies">
    <ul class="companies">
        <li class="company">AAA</li>
    </ul>
    </div>
  </li>
  <li class="phase">
    <div class="phase">Confirmed</div>
    <div class="companies">
    <ul class="companies">
        <li class="company">BBB</li>
        <li class="company">CCC</li>
    </ul>
    </div>
  </li>
  <li class="phase">
    <div class="phase">Screen</div>
    <div class="companies">
    <ul class="companies">
        <li class="company">Fubar</li>
    </ul>
    </div>
  </li>
  <li class="phase">
    <div class="phase">Interview</div>
    <div class="companies">
    <ul class="companies">
        <li class="company">Rabuf</li>
    </ul>
    </div>
  </li>
  <li class="phase">
    <div class="phase">Offer</div>
    <div class="companies">
    <ul class="companies">
        <li class="company">Oof Rab</li>
    </ul>
    </div>
  </li>
  <li class="phase last">
    <div class="phase">Start!</div>
    <div class="companies">
    <ul class="companies">
        <li class="company">Rab Oof</li>
    </ul>
    </div>
  </li>
</ul>

{% endblock content %}

Obviously this still just has dummy static data. We’ll get some active data in there very soon, but now we’re ready to rerun the test. It should pass this time. If not, fix the problem. When you get all tests passing, celebrate!

For real data, we need to write the model(s) used by this app. Let’s add another test:

class DashboardViewTest(DashboardViewTestBase):
    USERNAME = 'brian'
    TESTLOGIN = (USERNAME, 'a')

    def test_company_list(self):
        self.expect_div_content('content', 'Foobar Corp')
        return

Edit models.py to add the model:

from django.contrib.auth.models import User
from django.db import models

class Job(models.Model):
    applicant = models.ForeignKey(User)
    company = models.CharField(max_length=80)

The test depends on having a job in the database. Let’s set up a test fixture. Edit YOURAPPNAME/fixtures/initial_data.json:

[{"pk": 1, "model": "auth.user", "fields":
    {"username": "brian", "first_name": "", "last_name": "",
     "is_active": 1, "is_superuser": 0, "is_staff": 0,
     "last_login": "2009-02-06 13:50:02", "groups": [], "user_permissions": [],
     "password": "sha1$0fcc8$c4cf5184f5c005f90165e782fb090e7d75b72986",
     "email": "brian@example.com", "date_joined": "2009-02-06 13:44:04"}},
 {"pk": 2, "model": "auth.user", "fields":
    {"username": "alan", "first_name": "", "last_name": "",
     "is_active": 1, "is_superuser": 0, "is_staff": 0,
     "last_login": "2009-02-06 13:50:02", "groups": [], "user_permissions": [],
     "password": "sha1$0fcc8$c4cf5184f5c005f90165e782fb090e7d75b72986",
     "email": "alan@example.org", "date_joined": "2009-02-06 13:44:04"}},
 {"pk": 1, "model": "jobfunnel.job", "fields":
    {"position_url": "http://example.com/career/", "title": "Foobar Eng",
     "company_url": "http://example.com/", "company": "Foobar Corp", "applicant": 1,
     "phase": "Apply", "date": "2009-02-11", "position": "Engineer",
     "notes": "Applied via website"}}
]

This will populate the database with a couple of users, both with password “a”, and a job before each test runs.

Run this test, expecting exactly one failure — the content div does not contain the expected string. Let’s grab the list of jobs in the view and pass it into the template:

@login_required
def dashboard(request):
    jobs = models.Job.objects.all()
    return render_to_response('dashboard.html',
                              {'title': 'Dashboard',
                               'jobs': jobs,
                               },
                              context_instance=RequestContext(request))

I won’t paste all of the code here again, but we need to edit the template to use the jobs list:

    <ul class="companies">
      {% for job in jobs %}
        <li class="company">{{ job.company }}</li>
      {% empty %}
        <li class="company">No jobs in this phase.</li>
      {% endfor %}
    </ul>

Now rerun the test and expect it to pass. Hooray!

One last refinement before we quit for today: users shouldn’t be able to see each others applications. The way this is coded, all jobs are going to show up on everybody’s dashboards. Not good. Here’s another test that checks that the application for Foobar Corp only shows up on Brian’s dashboard, not on Alan’s.

class PrivateDashboardViewTest(DashboardViewTestBase):
    USERNAME = 'alan'
    TESTLOGIN = (USERNAME, 'a')

    def test_private_applications(self):
        assert('Foobar' not in self.get_div_content('content'))
        return

Run the test, watch it fail, and then change one line in the view:

    jobs = models.Job.objects.filter(applicant=request.user)

Now the all the tests should pass.

Push a copy of your code into your version control tool. Take a break, you deserve it.

For “homework”, put together your other views in the same way as this one. We’ll look at deployment tomorrow.

Welcome back.

Ready for Day Two? Did you get your “hello world” app running on your host?

Where We Stand

Yesterday we nailed down our concept, bought a domain name and hosting, set up our toolkit, and deployed a practice app on the host.

Here’s what I’m going to build: a web app to keep track of job applications, cover letters, and resumes. The working name for the project is “resuman”. Unfortunately, that obvious domain name was already registered and I’m not up for buying from the current owner. So after trying a bunch of more-or-less obvious combinations, I grabbed “yresu.me”, and I’ll deploy the app at “trackm.yresu.me”.

Notes on Domain Registration

It’s worth making a couple of notes here on the domain registration process. I use GoDaddy for my registrations. In the past I’ve done registration with my preferred host, but if you end up switching hosts it’s a hassle to move the domain registrations. Another complication: hosting services like DreamHost only support registrations on the major TLDs (.com, .org, .net, .info), while a full-service registrar like GoDaddy supports alternatives like .me.

Lastly, I like the bulk domain search service from Dotster. You can search for up to 50 names at a time. So if your first choice is unavailable, use Dotster’s bulk search tool to enter a whole mess of variations and see what’s available. This is the process I used to find yresu.me — the search included resu.me, esu.me, su.me, and myresu.me, all of which were taken.

Make Some Mockups

Before we commit any design ideas to HTML, let’s take some great advice from 37signals: sketch a bunch of drafts on paper first. To back up this idea, check out this story posted by Jeff Atwood about quantity vs. quality:

The ceramics teacher announced on opening day that he was dividing the class into two groups. All those on the left side of the studio, he said, would be graded solely on the quantity of work they produced, all those on the right solely on its quality. His procedure was simple: on the final day of class he would bring in his bathroom scales and weigh the work of the “quantity” group: fifty pound of pots rated an “A”, forty pounds a “B”, and so on. Those being graded on “quality”, however, needed to produce only one pot – albeit a perfect one – to get an “A”.

Well, came grading time and a curious fact emerged: the works of highest quality were all produced by the group being graded for quantity. It seems that while the “quantity” group was busily churning out piles of work – and learning from their mistakes – the “quality” group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay.

So my advice to you is to produce no fewer than five different designs on paper. It only takes a few minutes to sketch something out and see what it looks like. I did a handful of drawings on paper with my kids’ crayons, and then some refinements on my whiteboard once I had chosen the general design. So use whatever media you have handy, and don’t worry about getting it perfect. Run your sketches by a couple of innocent bystanders for some feedback.

Once you’ve got a good design on paper, make mockups in HTML and CSS. Steve Dennis at subcide.com has a great walkthrough on creating a CSS layout from scratch. If you don’t have an established process for building up a design, follow his tutorial. Putting together the HTML+CSS takes a little more time than sketches on paper, but it’s still worth doing a couple of different designs to see what grabs you. I worked up two different CSS layouts on top of the same HTML. Run these designs by some people — now that you’ve got some code, you can upload it to your site and email a link to some friends. Pick the better of the two (or three).

Better yet, upload the mockups to your host and leave a comment here with a link to them. Traffic here is low enough that I should be able to reply with feedback. (Although feel free to circulate a link to this series to a hundred of your friends, or post it to digg. I’ll at least reply to the first 50 comments…)

Tomorrow we’ll start writing some code.

This is the first in a series of posts that will walk through the steps of designing, building, and deploying a complete web app using Django. I’m going to assume you know the basics when it comes to Python, Django, HTML, CSS, Javascript and some basic tools. This will be a part time effort, so you should be able to work alongside me in a couple of hours a day.

A word about costs: you’ll need to spend ten bucks or so for a domain name, and anywhere from $30 to $150+ for hosting if you don’t already have these set up. Of course, you can just follow along on your server at home for free, but you’ll learn more if it’s all live when everything is said and done.

Let’s get started.

Day 1: Get Ready

Concept

We need to figure out what we’ll be building, and we need to get a few things set up.

First, you need an idea. It doesn’t have to be the next Twitter. Try to pick something simple that you’ll be able to code up in a couple of evenings, but useful enough that it’s worth doing right.

Let that simmer on the back burner, and let’s get some stuff set up.

Hosting

You will need someplace to host the app. As noted above, you can do this on your own machine at home, but it’s good practice to set it up live — the learning will be stickier too.

Webfaction is highly recommended for Django hosting, but it can be a bit pricey if you’re just experimenting.

Deploying Django on Dreamhost will suffer a bit performance-wise, but if you google for a coupon you may be able to get up to $97 off. So for $30 you can get a year’s worth of hosting. Not bad.

Go off and do this. Now. This post will still be here when you come back.

Domain

Obviously, you’re going to need a domain name where your app will live.

How’s that idea you’ve been simmering for the past 20 minutes? Good, now pick a domain name to go with it. You can let this simmer for a while too, but remember that the name can take a day or two to propagate through DNS once you’ve purchased it.

Not sure about Webfaction, but you can buy this through Dreamhost for $10 or $15. Depending on what kinds of promotions they’re offering, you may even have a credit for a free domain name when you signed up for hosting.

Go ahead and snag that domain now, I’ll wait patiently…

Tools

Since you don’t want to develop on the hosting platform, you’ll need to install some things on your own machine so you can develop and test. This is the minimum I’ve been using, please leave a comment if there’s something you think I’m missing.

• Apache with mod_python (used at Webfaction) or mod_fastcgi (used at Dreamhost). If you have a different host (or you’re reading this at some future date and the options have changed), just make sure that what you’re using matches them.
• Editor
• Firefox with the Firebug and Web Developer add-ons.
• Python
• Django. I use the trunk. (Actually, I pull from my github fork.)
• Selenium

It will be helpful if you can install the tools in the same way — including the same paths — that you have on your web host. If this isn’t practical, it isn’t a big deal, but it will make your life easier down the road when you’re trying to figure out why something works locally but doesn’t work on the live site.

That’s it for today. As homework, set up a practice “hello world” app on your host so that you know everything is working.

Update: continue this series with day two: mockups.

Subscribe to The Daily Build to make sure you get all the posts in this series!

Jesse Noller has been republishing articles he wrote for Python Magazine. These are very informative: I learned new things about the “with” statement in python 2.5 and tonight I was introduced to Paramiko, a library for using SSH.

This is a rundown of the things you should have in your toolkit — doesn’t matter whether you call yourself a software engineer, programmer, developer, code monkey, etc.

Editor

You must have an excellent editor. It should slice, dice, puree and mince, all with minimal effort. Seriously, it should have support for searching across files, “tags” (jumping to the definition of an identifier), ideally provide tooltips for function calls while you’re writing, automatic template generation, and pushbutton compilation that can step you through the list of compile errors, if any.

Pick a good editor, and become an expert at using it. It’s worth the investment.

Toolchain (Compiler)

I can’t really say compiler any more, since many of the languages we’re using aren’t compiled in the traditional sense. So instead I’ll call this the toolchain. For your chosen language, you need to choose the most appropriate toolchain. It may sometimes seem like you don’t have much choice: there’s only one python, right? Wrong. You have the choice of targeting different platforms (CPython, Jython or IronPython), and also of choosing the version you’ll use. You may need to choose an older version so that you can match what’s installed on your hosting provider. You may even need to keep around multiple versions — maybe you’re a library writer and need to ensure portability across various versions.

Things that are part of your toolchain: compiler or interpreter and runtime. In other words, whatever it takes to make the software run on your customers’ computer.

Version Control

AKA VC, software configuration management (SCM), source control. This is a tool that keeps track of the various versions of your software and allows you to work effectively with other engineers. There are a number of different VC tools, with a wide variety of features, complexity and costs.

If you haven’t used VC before, take a look at subversion. It’s free (open source), widely used, easy to install and use, and stable. There are other tools that may have better technical merits, especially for certain environments, but I think subversion is the best place to start from an ease-of-learning perspective.

Build

For your software to have value for your customers, you need to be able to convert program text into executable software. If your software is a one-file script, maybe all you need to do is publish it to your website. In this is the case, you’re allowed a pass on having a build tool, but you should still consider an automated solution.

If you have to deliver something more complex, an automated build tool is required.

Even if you’re just publishing a handful of scripts to your website as a tar or zip file, you still need to perform the following: gather the scripts, tar them, name the tarball something sensible, and upload. Do this by hand, and eventually you will skip a step and your customers will end up downloading an unusable mess.

At the opposite extreme, you may have hundreds of files that need to be compiled into a handful of libraries and then linked into an executable. You may have dozens of dependencies on external libraries. Translations to other natural languages. All of this needs to be portable across a dozen platforms.

This is why "make" was invented, and why other tools with similar goals continue to evolve to solve these complex problems. Ant, Maven, Scons, an endless parade of Make variants, etc. Pick the build tool that best works with your toolchain and specific situation.

Review

You need a way to review your work. If you’re working alone, it may be enough to simply review your work in your editor — though I’d recommend at least printing it out. For whatever reason, it seems harder / less effective to review a work product in the medium that produced it. A better alternative may be to use the view / diff functionality in your VC tool.

If you’re working with others, it will be helpful to install a code review tool that helps package your work product, distribute it to your peers for review, and manage feedback.

Test

Some languages come with support for unit test built in — python includes it in the standard library. You can find a handful of different libraries for c++, and for seemingly any other programming language known to man.

Your testing needs may also require a framework like selenium, which allows automated testing in a web browser.

If you’re doing embedded software, you’re probably going to have to jump through a bunch of hoops to automate your testing. Unfortunately, you’ll have to figure out how big the hoops are, how many you need, how far off the ground they need to be, whether they should be electrified, and then you’ll have to build them yourself. Annoying, but very much worth it in the end.

Continuous Integration / Automated Build

Again: if you’re working alone you can skip this, though I still think it’s useful. To be clear, “continuous integration” is more than a tool. It’s a way of life. However, this post is about your toolkit, and CI is a very useful category of tool. There are a number of different CI tools available, each with a slightly different focus and set of features. Pick the tool that has the features you need, connect it to your VC system, and enjoy the automation.

Lint

For those of you who aren’t familiar, lint is an old-school tool for finding potential trouble spots in C code. There are many tools available for C and other programming languages that help to sniff out trouble spots in code. Some tools can even spot design anti-patterns. C has lint. Python has pylint. Coverity Prevent is a commercial tool (expensive) that can find deep bugs in C, C++ and Java.

I don’t want to advocate sloppy programming practices: you shouldn’t need to run lint on your code. But if there’s a tool available that can effectively remove defects from my software, I’m going to use it.

(“Effective” defined to mean costing less to remove defects than some alternative, which is often either system testing or "customer testing". Before you go spending tons of money on an expensive lint, are you doing code reviews? This is an extremely cost effective practice. Spend your time and money there first.)

I’ll continue this in another post. Meanwhile, what’s in your toolkit? Favorites from the categories above? Categories of software tools that you want to make sure I don’t miss?

The Dreamhost wiki article on Django helped, but all the steps starting from scratch aren’t really documented in one place. Hopefully the list below will help, but since I’m writing it after the fact and I had to go through a couple of iterations to get it right, there are probably some things that aren’t 100% right.

1. Read the python article. Set up virtualenv into $HOME/local.
2. If you’ve already messed around with installing MySQLdb and/or other packages, remove them and start over.
3. Install ez_setup (easy_install) as describe in the article.
4. Install the MySQL egg: easy_install MySQL_python
5. Follow the setup steps in the Django article. Do the “myproject” test using sqlite3. Really. It helped me find a couple of things I was doing wrong with my real (i.e. more complicated) MySQL project.
6. As a deviation from the Django setup instructions, I prefer to use a fork of the Django codebase with some of my own patches. If you’re in the same situation, use github. I forked the github “unofficial copy” of the subversion code, added a couple of patches that aren’t in the trunk yet and a couple of my own, and cloned a copy into my dreamhost account.

This is the second half of an article I posted about using a checklist to prevent security errors. There, I said that you have 15 checklist items max, and you shouldn’t waste any of those questions on silly things like “Does the code follow the coding standard?”.

Jason Cohen pointed to an article of his in which he said “List no items that can be automated.” (His emphasis, but I second the motion.)

In the context of the SANS paper, let’s look at five items that should be automated so that no human has to find the defects:

1. CWE-665: Improper Initialization. As the write-up for CWE-665 suggests, you could use a programming language that forces you to explicitly initialize all variables before use. Or, if you’re stuck with something like C, make sure you turn on the warnings in your compiler, or use a static analysis tool (e.g. lint) to verify that all variables are initialized before being used. Tools like Coverity can be very sophisticated in their static analysis.
2. CWE-119: Failure to Constrain Operations within the Bounds of a Memory Buffer. Same goes for this one. First, use a programming language that doesn’t require attention to every tiny little detail of string handling. Failing that, apply static analysis. It’s also worth performing runtime analysis (e.g. electric fence, Purify) with appropriate test cases to verify that you’ve avoided buffer overflows.
3. CWE-404: Improper Resource Shutdown or Release. Even garbage collected languages can have problems with resource release. First, some GC systems have problems with circular references. Second, GC systems are typically worried about releasing memory. You also have to worry about database handles, file handles, and sockets. Configure your static analysis tool to track resources like these so that you don’t have to do it manually.
4. The write-up for CWE-404 also mentions in passing that you should “wash your garbage before you dispose of it”. This is useful for two reasons: first, an attacker won’t have access to the contents of previously used memory, and second, it often makes debugging memory problems easier if you write a known value into freed memory. (I modified malloc() at my last job to write 0xcacacaca into freed memory so that we would know right away when someone stepped on a turd, um, I mean used freed memory.) Configure your libraries to shred objects before you free them and you won’t have to worry about it on a line-by-line basis in the code you review.
5. CWE-362: Race Condition. I’m typically worried less about security than the nightmare of isolating and debugging race conditions. I haven’t seen any tools that detect race conditions well, but Coverity does a decent job of telling you when you’ve mishandled a race condition in a couple of cases: first, it warns you when you fail to release a lock, and second, it warns you when you access a resource that is statistically accessed from within a lock. So you still have to beware of race conditions (at the design level is the best place to find these), but there is an option for finding tedious errors in the mechanics of dealing with races.

And now I’m going to throw out everything above.  Remember that the tools are built to detect common errors. If it is really important that you find a certain class of bug, put it on your checklist. In fact, if it’s super critical, make it a checklist of length one. For example, I’ve gone over codebases with the single-minded focus of finding race conditions. When you perform a review like this, it’s amazing how many defects you might find in areas that you never suspected.