The whole process is much easier with compilation-perl.el. This handy bit of code sets up patterns for compilation-mode so that test failures (among other things) are recognized. Just add compilation-perl.el to your emacs lisp directory and follow the directions at the top of the file for installation instructions.

So my test process is now:

1. Hit F4 (which I have mapped to compile).
2. If necessary, change the command to either make test or to run a specific test I’m working on (e.g. perl t/MyModule.t).
3. If there’s a failure, press C-F4 (which I have mapped to next-error) and it brings me right to the failure.
4. Repeat from step 1 (but the second time around I don’t need to do step 2 since the command is already set properly).

If you want to experiment with history expansion a bit, you can echo the result instead of executing it:

hostname:~/dir% ls /some/long/path/to/file_0.1-2_i386.changes
hostname:~/dir% echo !?ls?:s/-2/-3/
echo ls /some/long/path/to/file_0.1-3_i386.changes

In this case, what I wanted to do is repeat a long command that referenced a file with a version number in it — but I wanted to use a different version number (-3 instead of the previously given version -2).

History expansion uses “!” (“bang”, or exclamation point). If you want to select the previous command to expand based on a string match, “!?str” will find the previous command that contains “str”. If you want to substitute some part of that previous command, put another question mark on the end, add a colon, and use the the “s” modifier. In the example above, “!?ls?” means expand the previous command containing “ls”. The “:s/-2/-3/” means modify that expanded command to replace the occurrence of “-2″ with “-3″.

(Of course, to actually get all of that figured out and working, I had to iterate the sequence of commands above about a dozen times.)

Leave a comment with your favorite zsh history trick. Thanks!

1. Source control. This almost belongs in the “if you don’t have it, you aren’t making software” category. There are so many free and simple to use tools that it is ridiculous to not use source control. Pick one of: subversion, git, or any of the dozens of other free tools and use it. I guess I’d say “if you don’t have it, you are making a mess”.
2. Bug/issue tracker. Trac is a decent tool for keeping tabs on development tasks and bugs. Bugzilla is ok too. If you have a budget, there is an apparently infinite set of commercial systems available of varying quality. Almost anything is better than a list of scribbles on your whiteboard.
3. Backup server. You can do software without it, but the odds are against you lasting long.
4. Build system. Automated nightly/continuous builds are awesome. Whether you DIY, install something free like hudson, or purchase a commercial system, this is a key piece of making sure that the software maintains at least a minimum level of quality (i.e. it is always buildable). After having worked on a couple of teams that didn’t have such a system, and the build was always broken, I would never work without it again.
5. IM / Jabber. Especially with a conference server where the rooms are archived and available via the web. Searchable is nice, but making them available where they can be grepped isn’t bad either. I didn’t become convinced of the value here until last summer.It sounds like an extra distraction and sometimes it can be, but you can turn it off when you’re heads-down coding. And it has the potential to reduce the number of interruptions from people walking into your office with questions that could have been answered in an IM. It’s lighter weight than email and rooms are broadcast by nature so you don’t have to spam a huge list of people to find the answer you’re looking for.
6. Email lists. Must have web-browsable archives; the list without this has much less value. Search would be a nice bonus, though I haven’t seen it done well. This lets teams communicate a bit more formally than IM and provides a record of certain discussions and decisions.
7. FTP server. Bonus points for a TFTP server. Maybe it’s because of the nature of embedded software, but every team I’ve been on has always had to fling around large files and images. Email is horribly inefficient for this. Also, most of the products I’ve worked on needed an FTP (or TFTP) server to boot from. You’re likely to install something on your development machine, which is fine, but it’s convenient to have a central team/company internal server so you can just tell someone that a file is in your public directory on “the ftp server”. Microsoft shops and folks that don’t need to boot via FTP can probably get away with samba shares.
8. Doc server. Wiki is probably the best way to go if your team doesn’t have to produce overly-formal documentation. Wikis and formal document control systems that I’ve used have had horrible search, which sucks but I haven’t found a great solution. In the past I’ve done this with a combination of a formal doc tool plus a blog for the informal “howto” kind of notes that are always needed.
9. Code review. I can’t say I’ve worked anywhere that had a dedicated code review tool. However, when all of the development machines are on NFS so that everyone has a view of your workspace, IM fits the bill reasonably well: post a review request to the chat room with the path to your changes and get an ad hoc on the spot code review. With some discipline and a supportive culture this works very smoothly. (This arrangement could probably be done with an internal pastebin, which I haven’t listed here as must-have since I haven’t really seen it used.)

What would you add to this list? Any other tools or systems that, if missing, would make you think twice about taking a job?

Let’s say you have a half-dozen machines at work you want to log into. Instead of setting up a remote forwarding connection from each of those machines, you can have the connection from your main machine perform multiple forwardings instead of just one. This even works if some of the machines don’t support ssh.

It shouldn’t surprise you at this point that you can do this with your config file. On your work machine, you might have something like:

Host tunnel
  HostName cloud.example.com
  User mycloudusername
  IdentityFile ~/.ssh/id_dsa
  Port 22
  RSAAuthentication yes
  PubkeyAuthentication yes
  ExitOnForwardFailure yes
  # tunnel ssh to myworkmachine
  RemoteForward 4022 localhost:22
  # tunnel remote desktop to mywindowsbox via myworkmachine
  RemoteForward 5389 192.168.4.10:3389
  # tunnel http to mywindowsbox via myworkmachine
  RemoteForward 5080 192.168.4.10:80
  # tunnel remote desktop to otherwindowsbox via myworkmachine
  RemoteForward 6389 192.168.4.11:3389
  # tunnel ssh to workserver via myworkmachine
  RemoteForward 7022 192.168.4.2:22

You can add a bunch of forwardings as shown above. Each entry will open the given port on cloud and forward it to the specified port on the specified machine. Now when you run “ssh tunnel” on your work machine, it will connect to cloud and set up the five port forwardings specified in your config file.

Then when logged in to cloud.example.com, you can do, for example, “ssh -p 7022 myserverlogin@localhost” to log into the machine called workserver.

If you mirror the remote forwardings in your home config file as local forwardings, then when you “ssh work” from home you can remote desktop to a windows machine from your home pc by doing “rdesktop -u myworkwinuser localhost:5389″ and it will use the tunnel. (The connection will go from your home pc to cloud, to myworkmachine, to mywindowsbox.) The windows machine does not need to know anything about ssh.

As I mentioned in a previous post on ssh configuration, your config file can specify a variety settings for each server.

In fact, the Hosts you use don’t even have to exist! (The HostName is the important part.) Consider the following snippet in your ~/.ssh/config.

#
Host work
  HostName localhost
  User myworklogin
  IdentityFile ~/.ssh/id_dsa
  Port 4022
  RSAAuthentication yes
  PubkeyAuthentication yes
  LocalForward 4022 localhost:4022

I’m going to assume remote forwarding is set up and the connection is open from work to cloud as described in this post on remote forwarding; and you’ve got local forwarding set up from home to cloud as described in this post on local port forwarding.

Now you can do “ssh work” from your home pc, and it will automatically log you into your work pc with the right credentials using the tunnel on cloud.example.com. And the scp example above simplifies to “scp work:/tmp/foo.txt ~/foo.txt” — you don’t have to remember the forwarded port numbers.

Typing “ssh work” is nine keystrokes (eight letters plus enter). If you can type 40 wpm, that’s 200 keystrokes per minute, or 3.33 keystrokes per second, which means you can open the tunnel in four seconds!

If you add “alias ssw=’ssh work’” to your ~/.bashrc, you’re down to four keystrokes.

Local port forwarding is the same as remote port forwarding but works in the opposite direction. An example is the clearest way to explain.

Assuming you’ve done the steps in the previous posts, then at home you can run “ssh -L 4022:localhost:4022 me@cloud.example.com”. This listens on TCP port 4022 on your home machine. Any connections there will be forwarded through the ssh connection to port 4022 on cloud… which, as we recall, gets forwarded to port 22 (ssh) at work. If you leave this connection open, you can run “ssh -p 4022 localhost” on your home machine and it will connect to work in just one hop. This means that you can use scp to copy files from home to work or vice versa. For example, “scp -P 4022 localhost:/tmp/foo.txt ~/foo.txt” will copy a file from work to home. (Note: scp needs capital “-P” to give the port. I got it wrong the first time.)

Ssh tunneling can be a bit mind bending at first, but it’s simple when you get used to it. Assume that you’re trying to ssh between two sites that do not allow incoming ssh. Maybe your IT at work is unenlightened and doesn’t have an ssh gateway. And your ISP has braindead configuration rules that don’t allow incoming ssh or they make it difficult.

What you need to get around this is a server “in the cloud” that permits ssh logins. This could be a hosting server that you pay for, or even a friend with an enlightened ISP who will give you a login account.

On your work PC, use ssh to login to the “cloud” server. Using the “-R” argument, you tell ssh to listen on a TCP port on the cloud server. Any connection coming in to this server will be forwarded back through the ssh connection to the TCP port you specify. For example, on mymachine.work.com, “ssh -R 4022:localhost:22 me@cloud.example.com” tells ssh to listen on cloud’s port 4022. Incoming connections to that port on cloud will be forwarded to port 22 (ssh) on mymachine.

By default, ssh will only listen to port 4022 on cloud’s localhost interface. So to log in to work, you will first need to log into cloud, and then use “ssh -p 4022 myworklogin@localhost” to log into work.

We’ll work around this limitation in the next post in this series.

The capability that you probably aren’t using, and the easiest to use, is customizing your config file (in ~/.ssh/config) for the various servers you log into.

For example, I frequently log into about ten different servers using at least four different usernames. By default, if I type “ssh server” the client will use my login name on the client machine to try to log into the server — which is usually wrong. Instead you can tell your ssh client which username to use on each server. (Thanks to Rich Adams for the tip.)

You can customize a variety of settings — not just the username. For example, I specify a different identity file for a couple of servers.

This saves a bunch of typing and occasional confusion. (By avoiding login errrors as I try to log into a server using the wrong username and can’t figure out why my password isn’t working…)

(This is the first post in series of posts about how to get the most out of ssh. Make sure you don’t miss the rest of the series: subscribe to my feed or follow me on twitter.)

Why do this? Well, for one thing it makes code reviews easier when everyone follows the same conventions. It also makes maintenance easier.

1. pep8.py is a style checker that enforces the rules of PEP 8. The “official home” (?) at browsershots.org was dead as I was writing this. (Thanks GitHub!) Run pep8.py on your code and it will tell you where you’ve drifted from the standard.
2. pylint is lint for python. It provides more functionality than pep8.py, but is not a strict superset. (pep8.py is pickier about whitespace issues.) Pylint is also configurable to enforce various naming rules. Like most lints, the SNR is pretty low, but you can turn off most of the noise and get a reasonable signal for the things you want to check.
3. Subversion (as well as most other tools) can be configured to run a script every time you check in code. Run one or both of the above tools in the pre-commit hook and bad code will be rejected. (I’d be wary of doing this with pylint unless you’ve got the categories of “noisy” warnings turned off.) I don’t have anything cookbook for this yet, but I’m working on it…

This is a really broad way to divide the world, but I see two important categories of build scripts. (By “build script” I mean Makefiles, SConsfiles, Ant files, and their ilk).

1. Build scripts intended for in-house use, never seen by outsiders.
2. Build scripts that you distribute to external users.

Both sets of build scripts are important. Let’s consider the build script (or Makefile system, or whatever you’re using) as a software system independent of the software that is being built. Its primary requirement is to reliably convert your source code into an executable. Any decent build script has numerous other requirements, most of which are probably implied. I have never seen a written requirement for the numerous build scripts I’ve written. I have only rarely heard them discussed — and never actually posed as “requirements”.

I have, always taken the following meta-requirements for granted. Based on my experience with some fairly awful build scripts, I guess these aren’t universally acknowledged.

• The build script should build all the “normal” executables with a single command.
• The build script must encapsulate all environment variables within the script. (This is a corollary of the above.)
• The build script should not unnecessarily rebuild source code in a directory tree that has previously been built. (This is make‘s raison d’etre.)
• The build script should be documented. It doesn’t have to be elaborate, but a five-line comment at the top of the script describing the available command-line variables would be nice.
• make clean or its equivalent must work reliably.
• It should rarely be necessary to run make clean.
• Bonus: the build should be parallelizable, to be able to take advantage of multicore machines and/or distributed builds.

Back to those two categories. If you screw up an in-house build script, it’s primarily your team that’s going to suffer. I’m not aware of any bugs from SQA or Customers on projects I’ve been on that were traced back to a build script, but I can see where it could happen. Mainly what I’ve seen is developers habitually wasting time rebuilding source that should not need to be rebuilt. The worst offender in this respect was a build script that forced a “make clean” at the beginning of every build. In the immortal words of Dave Barry, I’m not making this up. Really.

(Actually, I think it was even worse when I was dealing with a build script that required a “make clean” every time around but didn’t have it coded into the script!)

I can’t say I have a completely clean conscience in this area: I wrote a big pile of Makefiles for a past employer in which I unknowingly used features (bugs?) of a specific version of GNU Make that were “fixed” in a subsequent version. As time went on it became (a) a much bigger task to fix the Makefile code that used the “feature” and (b) harder to keep the specific version of GNU Make that worked with the Makefiles! I imagine they’re still stuck with this situation… sorry guys.

In the second category, when you toss a slapped-together build script to your unsuspecting users you do all of human society a disservice. Now they are not only wasting their time fighting with a bad build script, but they often have the disadvantage of being unable to contact the author directly for support or commiserate with other experienced users of the system.

So do us all a favor: if you don’t know what you’re doing when you start to write your next build script, please ask for help. Otherwise we’ll have to resort to some sort of professional licensing scheme for build script authors.