If you want to experiment with history expansion a bit, you can echo the result instead of executing it:

hostname:~/dir% ls /some/long/path/to/file_0.1-2_i386.changes
hostname:~/dir% echo !?ls?:s/-2/-3/
echo ls /some/long/path/to/file_0.1-3_i386.changes

In this case, what I wanted to do is repeat a long command that referenced a file with a version number in it — but I wanted to use a different version number (-3 instead of the previously given version -2).

History expansion uses “!” (“bang”, or exclamation point). If you want to select the previous command to expand based on a string match, “!?str” will find the previous command that contains “str”. If you want to substitute some part of that previous command, put another question mark on the end, add a colon, and use the the “s” modifier. In the example above, “!?ls?” means expand the previous command containing “ls”. The “:s/-2/-3/” means modify that expanded command to replace the occurrence of “-2″ with “-3″.

(Of course, to actually get all of that figured out and working, I had to iterate the sequence of commands above about a dozen times.)

Leave a comment with your favorite zsh history trick. Thanks!

Related posts:

1. Open an SSH Tunnel in Four Seconds or Less As I mentioned in a previous...
2. Make the zsh zle handle “words” correctly Snippet from my .zshrc: #...
3. Set Your zsh Prompt Since the beginning of time, all...

1. Source control. This almost belongs in the “if you don’t have it, you aren’t making software” category. There are so many free and simple to use tools that it is ridiculous to not use source control. Pick one of: subversion, git, or any of the dozens of other free tools and use it. I guess I’d say “if you don’t have it, you are making a mess”.
2. Bug/issue tracker. Trac is a decent tool for keeping tabs on development tasks and bugs. Bugzilla is ok too. If you have a budget, there is an apparently infinite set of commercial systems available of varying quality. Almost anything is better than a list of scribbles on your whiteboard.
3. Backup server. You can do software without it, but the odds are against you lasting long.
4. Build system. Automated nightly/continuous builds are awesome. Whether you DIY, install something free like hudson, or purchase a commercial system, this is a key piece of making sure that the software maintains at least a minimum level of quality (i.e. it is always buildable). After having worked on a couple of teams that didn’t have such a system, and the build was always broken, I would never work without it again.
5. IM / Jabber. Especially with a conference server where the rooms are archived and available via the web. Searchable is nice, but making them available where they can be grepped isn’t bad either. I didn’t become convinced of the value here until last summer.It sounds like an extra distraction and sometimes it can be, but you can turn it off when you’re heads-down coding. And it has the potential to reduce the number of interruptions from people walking into your office with questions that could have been answered in an IM. It’s lighter weight than email and rooms are broadcast by nature so you don’t have to spam a huge list of people to find the answer you’re looking for.
6. Email lists. Must have web-browsable archives; the list without this has much less value. Search would be a nice bonus, though I haven’t seen it done well. This lets teams communicate a bit more formally than IM and provides a record of certain discussions and decisions.
7. FTP server. Bonus points for a TFTP server. Maybe it’s because of the nature of embedded software, but every team I’ve been on has always had to fling around large files and images. Email is horribly inefficient for this. Also, most of the products I’ve worked on needed an FTP (or TFTP) server to boot from. You’re likely to install something on your development machine, which is fine, but it’s convenient to have a central team/company internal server so you can just tell someone that a file is in your public directory on “the ftp server”. Microsoft shops and folks that don’t need to boot via FTP can probably get away with samba shares.
8. Doc server. Wiki is probably the best way to go if your team doesn’t have to produce overly-formal documentation. Wikis and formal document control systems that I’ve used have had horrible search, which sucks but I haven’t found a great solution. In the past I’ve done this with a combination of a formal doc tool plus a blog for the informal “howto” kind of notes that are always needed.
9. Code review. I can’t say I’ve worked anywhere that had a dedicated code review tool. However, when all of the development machines are on NFS so that everyone has a view of your workspace, IM fits the bill reasonably well: post a review request to the chat room with the path to your changes and get an ad hoc on the spot code review. With some discipline and a supportive culture this works very smoothly. (This arrangement could probably be done with an internal pastebin, which I haven’t listed here as must-have since I haven’t really seen it used.)

What would you add to this list? Any other tools or systems that, if missing, would make you think twice about taking a job?

Related posts:

1. How to Use SSH Remote Port Forwarding to Set Up Secure Tunnels Ssh tunneling can be a bit...
2. Insist on Automatic Tests At some point your team...
3. An Interesting pid File Race ISC’s dhcpd uses this code...

Let’s say you have a half-dozen machines at work you want to log into. Instead of setting up a remote forwarding connection from each of those machines, you can have the connection from your main machine perform multiple forwardings instead of just one. This even works if some of the machines don’t support ssh.

It shouldn’t surprise you at this point that you can do this with your config file. On your work machine, you might have something like:

Host tunnel
  HostName cloud.example.com
  User mycloudusername
  IdentityFile ~/.ssh/id_dsa
  Port 22
  RSAAuthentication yes
  PubkeyAuthentication yes
  ExitOnForwardFailure yes
  # tunnel ssh to myworkmachine
  RemoteForward 4022 localhost:22
  # tunnel remote desktop to mywindowsbox via myworkmachine
  RemoteForward 5389 192.168.4.10:3389
  # tunnel http to mywindowsbox via myworkmachine
  RemoteForward 5080 192.168.4.10:80
  # tunnel remote desktop to otherwindowsbox via myworkmachine
  RemoteForward 6389 192.168.4.11:3389
  # tunnel ssh to workserver via myworkmachine
  RemoteForward 7022 192.168.4.2:22

You can add a bunch of forwardings as shown above. Each entry will open the given port on cloud and forward it to the specified port on the specified machine. Now when you run “ssh tunnel” on your work machine, it will connect to cloud and set up the five port forwardings specified in your config file.

Then when logged in to cloud.example.com, you can do, for example, “ssh -p 7022 myserverlogin@localhost” to log into the machine called workserver.

If you mirror the remote forwardings in your home config file as local forwardings, then when you “ssh work” from home you can remote desktop to a windows machine from your home pc by doing “rdesktop -u myworkwinuser localhost:5389″ and it will use the tunnel. (The connection will go from your home pc to cloud, to myworkmachine, to mywindowsbox.) The windows machine does not need to know anything about ssh.

Related posts:

1. Open an SSH Tunnel in Four Seconds or Less As I mentioned in a previous...
2. Use Local SSH Forwarding to Reduce the Number of Manual Hops Local port forwarding is the same...
3. How to Use SSH Remote Port Forwarding to Set Up Secure Tunnels Ssh tunneling can be a bit...

As I mentioned in a previous post on ssh configuration, your config file can specify a variety settings for each server.

In fact, the Hosts you use don’t even have to exist! (The HostName is the important part.) Consider the following snippet in your ~/.ssh/config.

#
Host work
  HostName localhost
  User myworklogin
  IdentityFile ~/.ssh/id_dsa
  Port 4022
  RSAAuthentication yes
  PubkeyAuthentication yes
  LocalForward 4022 localhost:4022

I’m going to assume remote forwarding is set up and the connection is open from work to cloud as described in this post on remote forwarding; and you’ve got local forwarding set up from home to cloud as described in this post on local port forwarding.

Now you can do “ssh work” from your home pc, and it will automatically log you into your work pc with the right credentials using the tunnel on cloud.example.com. And the scp example above simplifies to “scp work:/tmp/foo.txt ~/foo.txt” — you don’t have to remember the forwarded port numbers.

Typing “ssh work” is nine keystrokes (eight letters plus enter). If you can type 40 wpm, that’s 200 keystrokes per minute, or 3.33 keystrokes per second, which means you can open the tunnel in four seconds!

If you add “alias ssw=’ssh work’” to your ~/.bashrc, you’re down to four keystrokes.

Related posts:

1. Use Local SSH Forwarding to Reduce the Number of Manual Hops Local port forwarding is the same...
2. Use SSH to Forward Multiple Protocols to Multiple Machines Let's say you have a half-dozen...
3. How to Use SSH Remote Port Forwarding to Set Up Secure Tunnels Ssh tunneling can be a bit...

Local port forwarding is the same as remote port forwarding but works in the opposite direction. An example is the clearest way to explain.

Assuming you’ve done the steps in the previous posts, then at home you can run “ssh -L 4022:localhost:4022 me@cloud.example.com”. This listens on TCP port 4022 on your home machine. Any connections there will be forwarded through the ssh connection to port 4022 on cloud… which, as we recall, gets forwarded to port 22 (ssh) at work. If you leave this connection open, you can run “ssh -p 4022 localhost” on your home machine and it will connect to work in just one hop. This means that you can use scp to copy files from home to work or vice versa. For example, “scp -P 4022 localhost:/tmp/foo.txt ~/foo.txt” will copy a file from work to home. (Note: scp needs capital “-P” to give the port. I got it wrong the first time.)

Related posts:

1. How to Use SSH Remote Port Forwarding to Set Up Secure Tunnels Ssh tunneling can be a bit...
2. Open an SSH Tunnel in Four Seconds or Less As I mentioned in a previous...
3. Use SSH to Forward Multiple Protocols to Multiple Machines Let's say you have a half-dozen...

Ssh tunneling can be a bit mind bending at first, but it’s simple when you get used to it. Assume that you’re trying to ssh between two sites that do not allow incoming ssh. Maybe your IT at work is unenlightened and doesn’t have an ssh gateway. And your ISP has braindead configuration rules that don’t allow incoming ssh or they make it difficult.

What you need to get around this is a server “in the cloud” that permits ssh logins. This could be a hosting server that you pay for, or even a friend with an enlightened ISP who will give you a login account.

On your work PC, use ssh to login to the “cloud” server. Using the “-R” argument, you tell ssh to listen on a TCP port on the cloud server. Any connection coming in to this server will be forwarded back through the ssh connection to the TCP port you specify. For example, on mymachine.work.com, “ssh -R 4022:localhost:22 me@cloud.example.com” tells ssh to listen on cloud’s port 4022. Incoming connections to that port on cloud will be forwarded to port 22 (ssh) on mymachine.

By default, ssh will only listen to port 4022 on cloud’s localhost interface. So to log in to work, you will first need to log into cloud, and then use “ssh -p 4022 myworklogin@localhost” to log into work.

We’ll work around this limitation in the next post in this series.

Related posts:

1. Use Local SSH Forwarding to Reduce the Number of Manual Hops Local port forwarding is the same...
2. Open an SSH Tunnel in Four Seconds or Less As I mentioned in a previous...
3. Use SSH to Forward Multiple Protocols to Multiple Machines Let's say you have a half-dozen...

The capability that you probably aren’t using, and the easiest to use, is customizing your config file (in ~/.ssh/config) for the various servers you log into.

For example, I frequently log into about ten different servers using at least four different usernames. By default, if I type “ssh server” the client will use my login name on the client machine to try to log into the server — which is usually wrong. Instead you can tell your ssh client which username to use on each server. (Thanks to Rich Adams for the tip.)

You can customize a variety of settings — not just the username. For example, I specify a different identity file for a couple of servers.

This saves a bunch of typing and occasional confusion. (By avoiding login errrors as I try to log into a server using the wrong username and can’t figure out why my password isn’t working…)

(This is the first post in series of posts about how to get the most out of ssh. Make sure you don’t miss the rest of the series: subscribe to my feed or follow me on twitter.)

Related posts:

1. Open an SSH Tunnel in Four Seconds or Less As I mentioned in a previous...
2. How to Use SSH Remote Port Forwarding to Set Up Secure Tunnels Ssh tunneling can be a bit...
3. Use SSH to Forward Multiple Protocols to Multiple Machines Let's say you have a half-dozen...

Why do this? Well, for one thing it makes code reviews easier when everyone follows the same conventions. It also makes maintenance easier.

1. pep8.py is a style checker that enforces the rules of PEP 8. The “official home” (?) at browsershots.org was dead as I was writing this. (Thanks GitHub!) Run pep8.py on your code and it will tell you where you’ve drifted from the standard.
2. pylint is lint for python. It provides more functionality than pep8.py, but is not a strict superset. (pep8.py is pickier about whitespace issues.) Pylint is also configurable to enforce various naming rules. Like most lints, the SNR is pretty low, but you can turn off most of the noise and get a reasonable signal for the things you want to check.
3. Subversion (as well as most other tools) can be configured to run a script every time you check in code. Run one or both of the above tools in the pre-commit hook and bad code will be rejected. (I’d be wary of doing this with pylint unless you’ve got the categories of “noisy” warnings turned off.) I don’t have anything cookbook for this yet, but I’m working on it…

Related posts:

1. Who Else Wants Better Short Term Memory? In “Talent is Overrated”, Geoff...
2. Using Python’s ctypes to Call Into C Libraries The ctypes module makes loading and...
3. Data vs Code I’ll take an array over...

This is a really broad way to divide the world, but I see two important categories of build scripts. (By “build script” I mean Makefiles, SConsfiles, Ant files, and their ilk).

1. Build scripts intended for in-house use, never seen by outsiders.
2. Build scripts that you distribute to external users.

Both sets of build scripts are important. Let’s consider the build script (or Makefile system, or whatever you’re using) as a software system independent of the software that is being built. Its primary requirement is to reliably convert your source code into an executable. Any decent build script has numerous other requirements, most of which are probably implied. I have never seen a written requirement for the numerous build scripts I’ve written. I have only rarely heard them discussed — and never actually posed as “requirements”.

I have, always taken the following meta-requirements for granted. Based on my experience with some fairly awful build scripts, I guess these aren’t universally acknowledged.

• The build script should build all the “normal” executables with a single command.
• The build script must encapsulate all environment variables within the script. (This is a corollary of the above.)
• The build script should not unnecessarily rebuild source code in a directory tree that has previously been built. (This is make’s raison d’etre.)
• The build script should be documented. It doesn’t have to be elaborate, but a five-line comment at the top of the script describing the available command-line variables would be nice.
• make clean or its equivalent must work reliably.
• It should rarely be necessary to run make clean.
• Bonus: the build should be parallelizable, to be able to take advantage of multicore machines and/or distributed builds.

Back to those two categories. If you screw up an in-house build script, it’s primarily your team that’s going to suffer. I’m not aware of any bugs from SQA or Customers on projects I’ve been on that were traced back to a build script, but I can see where it could happen. Mainly what I’ve seen is developers habitually wasting time rebuilding source that should not need to be rebuilt. The worst offender in this respect was a build script that forced a “make clean” at the beginning of every build. In the immortal words of Dave Barry, I’m not making this up. Really.

(Actually, I think it was even worse when I was dealing with a build script that required a “make clean” every time around but didn’t have it coded into the script!)

I can’t say I have a completely clean conscience in this area: I wrote a big pile of Makefiles for a past employer in which I unknowingly used features (bugs?) of a specific version of GNU Make that were “fixed” in a subsequent version. As time went on it became (a) a much bigger task to fix the Makefile code that used the “feature” and (b) harder to keep the specific version of GNU Make that worked with the Makefiles! I imagine they’re still stuck with this situation… sorry guys.

In the second category, when you toss a slapped-together build script to your unsuspecting users you do all of human society a disservice. Now they are not only wasting their time fighting with a bad build script, but they often have the disadvantage of being unable to contact the author directly for support or commiserate with other experienced users of the system.

So do us all a favor: if you don’t know what you’re doing when you start to write your next build script, please ask for help. Otherwise we’ll have to resort to some sort of professional licensing scheme for build script authors.

Related posts:

1. 9 “Must-Have” Tools for Software Teams The items below are useful...
2. Using Lenny to Build Jaunty Packages <shameless_plug> I’m in the middle...
3. 3 Easy Ways to Stick to a Coding Standard When you’re writing python, you...

Editor

You must have an excellent editor. It should slice, dice, puree and mince, all with minimal effort. Seriously, it should have support for searching across files, “tags” (jumping to the definition of an identifier), ideally provide tooltips for function calls while you’re writing, automatic template generation, and pushbutton compilation that can step you through the list of compile errors, if any.

Pick a good editor, and become an expert at using it. It’s worth the investment.

Toolchain (Compiler)

I can’t really say compiler any more, since many of the languages we’re using aren’t compiled in the traditional sense. So instead I’ll call this the toolchain. For your chosen language, you need to choose the most appropriate toolchain. It may sometimes seem like you don’t have much choice: there’s only one python, right? Wrong. You have the choice of targeting different platforms (CPython, Jython or IronPython), and also of choosing the version you’ll use. You may need to choose an older version so that you can match what’s installed on your hosting provider. You may even need to keep around multiple versions — maybe you’re a library writer and need to ensure portability across various versions.

Things that are part of your toolchain: compiler or interpreter and runtime. In other words, whatever it takes to make the software run on your customers’ computer.

Version Control

AKA VC, software configuration management (SCM), source control. This is a tool that keeps track of the various versions of your software and allows you to work effectively with other engineers. There are a number of different VC tools, with a wide variety of features, complexity and costs.

If you haven’t used VC before, take a look at subversion. It’s free (open source), widely used, easy to install and use, and stable. There are other tools that may have better technical merits, especially for certain environments, but I think subversion is the best place to start from an ease-of-learning perspective.

Build

For your software to have value for your customers, you need to be able to convert program text into executable software. If your software is a one-file script, maybe all you need to do is publish it to your website. In this is the case, you’re allowed a pass on having a build tool, but you should still consider an automated solution.

If you have to deliver something more complex, an automated build tool is required.

Even if you’re just publishing a handful of scripts to your website as a tar or zip file, you still need to perform the following: gather the scripts, tar them, name the tarball something sensible, and upload. Do this by hand, and eventually you will skip a step and your customers will end up downloading an unusable mess.

At the opposite extreme, you may have hundreds of files that need to be compiled into a handful of libraries and then linked into an executable. You may have dozens of dependencies on external libraries. Translations to other natural languages. All of this needs to be portable across a dozen platforms.

This is why "make" was invented, and why other tools with similar goals continue to evolve to solve these complex problems. Ant, Maven, Scons, an endless parade of Make variants, etc. Pick the build tool that best works with your toolchain and specific situation.

Review

You need a way to review your work. If you’re working alone, it may be enough to simply review your work in your editor — though I’d recommend at least printing it out. For whatever reason, it seems harder / less effective to review a work product in the medium that produced it. A better alternative may be to use the view / diff functionality in your VC tool.

If you’re working with others, it will be helpful to install a code review tool that helps package your work product, distribute it to your peers for review, and manage feedback.

Test

Some languages come with support for unit test built in — python includes it in the standard library. You can find a handful of different libraries for c++, and for seemingly any other programming language known to man.

Your testing needs may also require a framework like selenium, which allows automated testing in a web browser.

If you’re doing embedded software, you’re probably going to have to jump through a bunch of hoops to automate your testing. Unfortunately, you’ll have to figure out how big the hoops are, how many you need, how far off the ground they need to be, whether they should be electrified, and then you’ll have to build them yourself. Annoying, but very much worth it in the end.

Continuous Integration / Automated Build

Again: if you’re working alone you can skip this, though I still think it’s useful. To be clear, “continuous integration” is more than a tool. It’s a way of life. However, this post is about your toolkit, and CI is a very useful category of tool. There are a number of different CI tools available, each with a slightly different focus and set of features. Pick the tool that has the features you need, connect it to your VC system, and enjoy the automation.

Lint

For those of you who aren’t familiar, lint is an old-school tool for finding potential trouble spots in C code. There are many tools available for C and other programming languages that help to sniff out trouble spots in code. Some tools can even spot design anti-patterns. C has lint. Python has pylint. Coverity Prevent is a commercial tool (expensive) that can find deep bugs in C, C++ and Java.

I don’t want to advocate sloppy programming practices: you shouldn’t need to run lint on your code. But if there’s a tool available that can effectively remove defects from my software, I’m going to use it.

(“Effective” defined to mean costing less to remove defects than some alternative, which is often either system testing or "customer testing". Before you go spending tons of money on an expensive lint, are you doing code reviews? This is an extremely cost effective practice. Spend your time and money there first.)

I’ll continue this in another post. Meanwhile, what’s in your toolkit? Favorites from the categories above? Categories of software tools that you want to make sure I don’t miss?

Related posts:

1. 9 “Must-Have” Tools for Software Teams The items below are useful...
2. 3 Easy Ways to Stick to a Coding Standard When you’re writing python, you...
3. Using Lenny to Build Jaunty Packages <shameless_plug> I’m in the middle...