The Daily Build

ISC’s dhcpd uses this code to check for an already-running daemon:

/* Read previous pid file. */
if ((i = open (path_dhcpd_pid, O_RDONLY)) >= 0) {
    status = read (i, pbuf, (sizeof pbuf) - 1);
    close (i);
    if (status > 0) {
        pbuf [status] = 0;
        pid = atoi (pbuf);

        /* If the previous server process is not still running,
           write a new pid file immediately. */
        if (pid && (pid == getpid() || kill (pid, 0) < 0)) {
            unlink (path_dhcpd_pid);
            if ((i = open (path_dhcpd_pid,
                           O_WRONLY | O_CREAT, 0644)) >= 0) {
                sprintf (pbuf, "%d\n", (int)getpid ());
                write (i, pbuf, strlen (pbuf));
                close (i);
                pidfilewritten = 1;
            }
        } else
            log_fatal ("There's already a DHCP server running.");
    }
}

The problem with this strategy is that, if the box dies, there’s a stale pid file left in /var/run/dhcpd.pid. This wouldn’t be so bad — the code above checks [using kill(pid, 0)] to see if there’s a process running with that pid. But when the box is restarting, there will be a bunch of processes all starting in similar sequence each time. So on one boot, you might see dhcpd with a pid of 1001 and ntpd with a pid of 1002. If the box dies violently (e.g. power cut), the dhcpd pid file will contain 1001. On the second boot, assume ntpd starts first and gets a pid of 1001 and dhcpd is 1002. Now, the kill(pid, 0) will succeed, making it appear that dhcpd is already running, and dhcpd will exit.

How to fix this?

1. Explicitly put the pid file under /tmp. Getting this right is fussy — make sure you avoid the race conditions associated with creating temp files. Use dhcpd’s “-pf” flag to tell it where to use the pid file. This avoids spurious “already running” messages, because dhcpd will never read a pid from an existing pid file. [You could also just remove the /var/run/dhcpd.pid file, but I'd rather explicitly provide the path in my startup script in case some dim bulb decides to change the compiled-in default.]
2. Be careful in your restart code to kill any existing dhcpd (assuming you really want a new dhcpd), or avoid trying to start a new one (assuming you want to use an already running dhcpd). pgrep(1) and pkill(1) will be useful here.

In researching this, I saw this bit of wisdom from Henning Brauer: “pid files are useless.”.

I heartily agree…

<shameless_plug>
I’m in the middle of putting together packages for the analog call generator my startup is building.
</shameless_plug>

The process for building debian packages is actually very well documented, though many of the tutorials you will find are aimed at people who are packaging third-party software instead of their own.

Also, I’m running on Debian’s Lenny, and I need to produce packages that install cleanly on Ubuntu’s Jaunty Jackalope. It sounds worse than it is.

Take this with a grain of salt — it works for me, but I’m certainly not an expert in this area.

Steps:

1. Debianize each package. (Here’s an excellent quick & dirty guide to getting a package created.)
2. Build the packages: (cd MYPROJECT; fakeroot ./debian/rules binary). Don’t distribute these packages, they are probably wrong! But this is the quickest way to get everything mostly-working.
3. Sanity check the packages: dpkg-deb -c MYPROJECT.deb; dpkg-deb -I MYPROJECT.deb; dpkg -i MYPROJECT.deb

4. Check in your MYPROJECT/debian/ directory to source control (e.g. git, svn, etc).
5. This is the tricky step, and one that I didn’t see explicitly documented.You need a version of debootstrap that knows about ubuntu distributions. (Ubuntu debootstrap knows about Debian, but not the other way around.)
   1. Download the jaunty (or whatever ubuntu version) debootstrap source package. Grab the .dsc and .tar.gz from that page.
   2. Run dpkg-source -x debootstrap_1.0.20~jaunty1.dsc — this extracts the source.
   3. (cd debootstrap-1.0.20~jaunty1; debuild -uc -us) — this will build a .deb you can install on your lenny machine. Install package “devscripts” if you don’t have debuild.
   4. sudo dpkg -i debootstrap_1.0.20~jaunty1_all.deb
6. Install pbuilder. This will help you build the packages in a clean chroot — i.e. without all the pollution you have on your machine that could be making the packages work even though they will break on your users’ machines.
7. Put the code found here in your ~/.pbuilderrc.
8. Apply this sudoers change (optional but convenient).
9. sudo DIST=jaunty pbuilder create … then wait. (If you have lame internet service (Hughes) and a daily bandwidth cap, you may want to schedule this for the free-for-all period: echo sudo DIST=jaunty pbuilder create | at 2:30am tomorrow)

If you see anything I’m missing or would like to add, please drop a comment below. Thanks!

Exploring zsh features made me want to figure out some of the history-editing wizardry. (Bash has similar history tricks, I just never bothered to dive too deeply into them.)

If you want to experiment with history expansion a bit, you can echo the result instead of executing it:

hostname:~/dir% ls /some/long/path/to/file_0.1-2_i386.changes
hostname:~/dir% echo !?ls?:s/-2/-3/
echo ls /some/long/path/to/file_0.1-3_i386.changes

In this case, what I wanted to do is repeat a long command that referenced a file with a version number in it — but I wanted to use a different version number (-3 instead of the previously given version -2).

History expansion uses “!” (“bang”, or exclamation point). If you want to select the previous command to expand based on a string match, “!?str” will find the previous command that contains “str”. If you want to substitute some part of that previous command, put another question mark on the end, add a colon, and use the the “s” modifier. In the example above, “!?ls?” means expand the previous command containing “ls”. The “:s/-2/-3/” means modify that expanded command to replace the occurrence of “-2″ with “-3″.

(Of course, to actually get all of that figured out and working, I had to iterate the sequence of commands above about a dozen times.)

Leave a comment with your favorite zsh history trick. Thanks!

I’ll take an array over a giant switch-case statement any day.

1. The array definition will be more compact and easier to see all at once.
2. Defining actions in an array enforces uniformity.
3. You can put checks in the code to automatically verify that the array definition is complete. (I.e. verify it contains a definition for every item it should have.) Yes, some tools can do this for certain types of switch statements. Using an array-based check is more portable and more foolproof.

If you have a switch statement of mostly cut and paste cases, you can probably convert it to an array very easily, and then rewrite the switch statement to look up the value in the array and do whatever thing is supposed to be done, either via function pointers or by using an associated value from the array.

I did this on a horrible switch statement once. Dozens of cases, 80% were nearly identical but the few that weren’t were awful to untangle. Once I pulled the case bodies into separate functions, put function pointers into a table, and replaced the switch body with a lookup loop it was much cleaner. The code for the odd cases was eventually pushed out (it was a symptom of bad design). The whole exercise enabled another round of changes that allowed the functions for the case bodies to be collapsed back into the table — we ended up removing an entire unnecessary layer of indirection and made the design much easier to grok.

Since the beginning of time, all the cool kids have had really cool shell prompts. It’s a great place to display helpful information, and zsh has features that let you have a flexible, informative, unobtrusive prompt.

Set your prompt by setting $PROMPT. If you do PROMPT='foo ', the shell will give you a foo prompt for every command. Not terribly useful but you get the point.

There are a bunch of codes you can use in the value of PROMPT to get useful output. For example, %m gives the name of the machine you’re running on, and %~ gives the name of the current working directory. For a list of all the codes, check the “zshmisc” man page under the section “SIMPLE PROMPT ESCAPES”. Note that there are codes for boldface, underline, and colors here too.

The drawback to having all kinds of information in your prompt is that you limit the length of commands that you can enter without scrolling. Scrolling stinks because the command is harder to read. Enter the “right prompt”. Set RPS1 to contain the lengthy part of your prompt, and it will be displayed on the right margin of your terminal. Then when commands get long and encroach on the right prompt, it will conveniently disappear. Read Quentin’s comment on this zsh post.

Here’s a gotcha with RPS1 and fancy prompt formatting: gnome-terminal does not behave well when these are set. I’m trying out alternatives to see which will be better.

Lastly, a trick I picked up from Justin. Zsh provides hooks that you can use to do things before and after a command runs. Just define the functions preexec and precmd. This example sets the title of the xterm while a command is running:

function title() {
    # escape '%' chars in $1, make nonprintables visible
    local a=${(V)1//\%/\%\%}

    # Truncate command, and join lines.
    a=$(print -Pn "%40>...>$a" | tr -d "\n")

    case $TERM in
        screen*)
            print -Pn "\e]2;$a @ $2\a" # plain xterm title
            print -Pn "\ek$a\e\\"      # screen title (in ^A")
            print -Pn "\e_$2   \e\\"   # screen location
            ;;
        xterm*)
            print -Pn "\e]2;$a @ $2\a" # plain xterm title
            ;;
    esac
}

# precmd is called just before the prompt is printed
function precmd() {
    title "zsh" "%m:%55<...<%~"
}

# preexec is called just before any command line is executed
function preexec() {
    title "$1" "%m:%35<...<%~"
}

Update: Set variable a to local in function title above as suggested in the comment below.

At some point your team is going to be gone. Not all at once (well, maybe, but in that case you won’t care), but over time turnover will completely replace your team.

If you are the manager, and you outlast the team, you’re going to pay for low quality code when you try to bring in new people and they end up breaking everything becuase there’s no tests to check their bug fixes and/or enhancements.

If you are a surviving member of the team, you’re going to pay with huge headaches because you’ve got to fix code where you have no idea what might be broken as a result.

Change is scary when you don’t have automatic tests. Just suck it up and write the tests!

Snippet from my .zshrc:

# This controls what the line editor considers a word. By default it
# includes '/', which makes it so that when I M-del (attempting to erase
# a directory in a path), I erase the whole path. Annoying.
# WORDCHARS='*?_-.[]~=/&;!#$%^(){}<>' # (default)
WORDCHARS='*?_-.[]~=&;!#$%^(){}<>'


After living with this for a while, I realize that I should probably remove the underscore too, since that’s what I was used to in bash.

To get started:

1. sudo aptitude install zsh
2. chsh /bin/zsh

That’s pretty simple.

Of course, you’re not running zsh yet… either logout and log back in or just run zsh at the prompt. You’ll get a series of prompts to configure a .zshrc. It only takes a few minutes, so run through the options and save the file.

Next up: setting a custom prompt.

The items below are useful systems based on my experience working with a bunch of different software teams at a handful of companies over the past decade-plus. I haven’t bothered to list things like compilers, interpreters, libraries, etc. If you don’t have those, you aren’t making software…

1. Source control. This almost belongs in the “if you don’t have it, you aren’t making software” category. There are so many free and simple to use tools that it is ridiculous to not use source control. Pick one of: subversion, git, or any of the dozens of other free tools and use it. I guess I’d say “if you don’t have it, you are making a mess”.
2. Bug/issue tracker. Trac is a decent tool for keeping tabs on development tasks and bugs. Bugzilla is ok too. If you have a budget, there is an apparently infinite set of commercial systems available of varying quality. Almost anything is better than a list of scribbles on your whiteboard.
3. Backup server. You can do software without it, but the odds are against you lasting long.
4. Build system. Automated nightly/continuous builds are awesome. Whether you DIY, install something free like hudson, or purchase a commercial system, this is a key piece of making sure that the software maintains at least a minimum level of quality (i.e. it is always buildable). After having worked on a couple of teams that didn’t have such a system, and the build was always broken, I would never work without it again.
5. IM / Jabber. Especially with a conference server where the rooms are archived and available via the web. Searchable is nice, but making them available where they can be grepped isn’t bad either. I didn’t become convinced of the value here until last summer.It sounds like an extra distraction and sometimes it can be, but you can turn it off when you’re heads-down coding. And it has the potential to reduce the number of interruptions from people walking into your office with questions that could have been answered in an IM. It’s lighter weight than email and rooms are broadcast by nature so you don’t have to spam a huge list of people to find the answer you’re looking for.
6. Email lists. Must have web-browsable archives; the list without this has much less value. Search would be a nice bonus, though I haven’t seen it done well. This lets teams communicate a bit more formally than IM and provides a record of certain discussions and decisions.
7. FTP server. Bonus points for a TFTP server. Maybe it’s because of the nature of embedded software, but every team I’ve been on has always had to fling around large files and images. Email is horribly inefficient for this. Also, most of the products I’ve worked on needed an FTP (or TFTP) server to boot from. You’re likely to install something on your development machine, which is fine, but it’s convenient to have a central team/company internal server so you can just tell someone that a file is in your public directory on “the ftp server”. Microsoft shops and folks that don’t need to boot via FTP can probably get away with samba shares.
8. Doc server. Wiki is probably the best way to go if your team doesn’t have to produce overly-formal documentation. Wikis and formal document control systems that I’ve used have had horrible search, which sucks but I haven’t found a great solution. In the past I’ve done this with a combination of a formal doc tool plus a blog for the informal “howto” kind of notes that are always needed.
9. Code review. I can’t say I’ve worked anywhere that had a dedicated code review tool. However, when all of the development machines are on NFS so that everyone has a view of your workspace, IM fits the bill reasonably well: post a review request to the chat room with the path to your changes and get an ad hoc on the spot code review. With some discipline and a supportive culture this works very smoothly. (This arrangement could probably be done with an internal pastebin, which I haven’t listed here as must-have since I haven’t really seen it used.)

What would you add to this list? Any other tools or systems that, if missing, would make you think twice about taking a job?

(This is part five in a series of posts on ssh.)

Let’s say you have a half-dozen machines at work you want to log into. Instead of setting up a remote forwarding connection from each of those machines, you can have the connection from your main machine perform multiple forwardings instead of just one. This even works if some of the machines don’t support ssh.

It shouldn’t surprise you at this point that you can do this with your config file. On your work machine, you might have something like:

Host tunnel
  HostName cloud.example.com
  User mycloudusername
  IdentityFile ~/.ssh/id_dsa
  Port 22
  RSAAuthentication yes
  PubkeyAuthentication yes
  ExitOnForwardFailure yes
  # tunnel ssh to myworkmachine
  RemoteForward 4022 localhost:22
  # tunnel remote desktop to mywindowsbox via myworkmachine
  RemoteForward 5389 192.168.4.10:3389
  # tunnel http to mywindowsbox via myworkmachine
  RemoteForward 5080 192.168.4.10:80
  # tunnel remote desktop to otherwindowsbox via myworkmachine
  RemoteForward 6389 192.168.4.11:3389
  # tunnel ssh to workserver via myworkmachine
  RemoteForward 7022 192.168.4.2:22

You can add a bunch of forwardings as shown above. Each entry will open the given port on cloud and forward it to the specified port on the specified machine. Now when you run “ssh tunnel” on your work machine, it will connect to cloud and set up the five port forwardings specified in your config file.

Then when logged in to cloud.example.com, you can do, for example, “ssh -p 7022 myserverlogin@localhost” to log into the machine called workserver.

If you mirror the remote forwardings in your home config file as local forwardings, then when you “ssh work” from home you can remote desktop to a windows machine from your home pc by doing “rdesktop -u myworkwinuser localhost:5389″ and it will use the tunnel. (The connection will go from your home pc to cloud, to myworkmachine, to mywindowsbox.) The windows machine does not need to know anything about ssh.