Dec  8 03:19:33 localhost sshd[4718]: User root from 10.1.2.3 not allowed [...]
Dec  8 03:19:35 localhost sshd[4721]: Invalid user db2inst1 from 10.1.2.3
Dec  8 03:19:38 localhost sshd[4723]: User root from 10.1.2.3 not allowed [...]

fail2ban is configured to (temporarily) block these after a certain number of attempts, but they keep coming back. One particular IP address was hitting ssh constantly (except for the ban periods) for several days, so I added a rule to drop everything from that address — but this strategy isn’t scalable.

The simple, obvious solution is to only allow access from known hosts. It would be very rare that I need to access this server from anywhere except a small number of addresses. I started with a ufw (Ubuntu’s “uncomplicated firewall”) ruleset like this:

% sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
Anywhere                   DENY        10.1.2.3
Anywhere                   DENY        172.16.99.88
22                         ALLOW       Anywhere

To avoid getting locked out, I added (temporarily) a crontab entry for root:

*/15 * * * * /usr/sbin/ufw allow ssh

This will allow access to ssh from any un-banned IP address (the current policy) every 15 minutes. Then I inserted a rule that allows access to ssh from my home ISPs netblock. (You can figure out the netblock by doing a whois lookup on your external IP address; you may need to add multiple netblocks if your ISP has several allocated. Be careful: it’s no fun to get locked out!) This action is safe because I have not yet removed global ssh access. Note that I’m using “insert 3″ to add this rule at a specific position in the list.

sudo ufw insert 3 allow proto tcp from 10.9.8.0/18 to any port 22

And I added a rule to permit access from another server with a fixed IP I have access to (for the rare case where I need to access this server when I’m not at home):

sudo ufw insert 3 allow proto tcp from 172.17.101.102 to any port 22

Now my ruleset looks like:

Status: active

To                         Action      From
--                         ------      ----
Anywhere                   DENY        10.1.2.3
Anywhere                   DENY        172.16.99.88
22/tcp                     ALLOW       172.17.101.102
22/tcp                     ALLOW       10.9.8.0/18
22                         ALLOW       Anywhere

So far, so good, but ssh access is still permitted from anywhere — because of that last rule. This is the dangerous part… you could get locked out if you haven’t set the rules correctly. (You set up that crontab entry, right?)

sudo ufw delete allow ssh

Wait for the prompt… hooray, I’m still connected! After checking that I can access ssh from home and the other server, I know it’s safe to remove the crontab job. (If the cron job has already fired, you’ll need to rerun the ufw delete allow ssh command.)

At this point I can delete the first two rules that ban specific IPs, since they’re outside my netblock and won’t be allowed anyway.

Now I can enjoy quieter logs without all those access attempts from China and Croatia!

SSH makes it trivial to use that VPS as a SOCKS5 proxy. Just do:

ssh -D 8080 myvps.example.com

Then set your browser’s SOCKS proxy to localhost:8080. In Firefox on Linux, this is Edit > Preferences > Advanced > Network tab > (Connection) Settings > Manual Proxy Configuration. Leave all fields blank except for SOCKS Host and Port — localhost and 8080, respectively. Choose SOCKS5. Then browse to about:config and change change network.proxy.socks_remote_dns to true. This tells Firefox to ask the proxy to resolve names instead of trying to resolve them using your ISP. Chrome worked for me without hassle.

Go to test-ipv6.com to test that it works. If your results from test-ipv6.com indicate that IPv6 name lookups are failing, make sure you’ve got that about:config setting mentioned above changed.

1. Install wine. (sudo aptitude install wine)
2. Install python into the wine environment. (Download an msi from python.org and run msiexec /i python-x.x.x.msi.)
3. Install whatever prerequisite packages you need (e.g. wxPython) using wine or msiexec.
4. When you’ve got everything ready to build, just do wine c:/Python27/python.exe setup.py bdist_wininst and look in ./dist/ for your exe!

/* Read previous pid file. */
if ((i = open (path_dhcpd_pid, O_RDONLY)) >= 0) {
    status = read (i, pbuf, (sizeof pbuf) - 1);
    close (i);
    if (status > 0) {
        pbuf [status] = 0;
        pid = atoi (pbuf);

        /* If the previous server process is not still running,
           write a new pid file immediately. */
        if (pid && (pid == getpid() || kill (pid, 0) < 0)) {
            unlink (path_dhcpd_pid);
            if ((i = open (path_dhcpd_pid,
                           O_WRONLY | O_CREAT, 0644)) >= 0) {
                sprintf (pbuf, "%d\n", (int)getpid ());
                write (i, pbuf, strlen (pbuf));
                close (i);
                pidfilewritten = 1;
            }
        } else
            log_fatal ("There's already a DHCP server running.");
    }
}

The problem with this strategy is that, if the box dies, there’s a stale pid file left in /var/run/dhcpd.pid. This wouldn’t be so bad — the code above checks [using kill(pid, 0)] to see if there’s a process running with that pid. But when the box is restarting, there will be a bunch of processes all starting in similar sequence each time. So on one boot, you might see dhcpd with a pid of 1001 and ntpd with a pid of 1002. If the box dies violently (e.g. power cut), the dhcpd pid file will contain 1001. On the second boot, assume ntpd starts first and gets a pid of 1001 and dhcpd is 1002. Now, the kill(pid, 0) will succeed, making it appear that dhcpd is already running, and dhcpd will exit.

How to fix this?

1. Explicitly put the pid file under /tmp. Getting this right is fussy — make sure you avoid the race conditions associated with creating temp files. Use dhcpd’s “-pf” flag to tell it where to use the pid file. This avoids spurious “already running” messages, because dhcpd will never read a pid from an existing pid file. [You could also just remove the /var/run/dhcpd.pid file, but I'd rather explicitly provide the path in my startup script in case some dim bulb decides to change the compiled-in default.]
2. Be careful in your restart code to kill any existing dhcpd (assuming you really want a new dhcpd), or avoid trying to start a new one (assuming you want to use an already running dhcpd). pgrep(1) and pkill(1) will be useful here.

In researching this, I saw this bit of wisdom from Henning Brauer: “pid files are useless.”.

I heartily agree…

So I wrote a little script (or a template for scripts) that runs the sudo job in the background (but preserves stdout/stderr) and relies on bash to clean up the job when you ^C the script. Only gotcha with this is that you may have to retype your sudo password when you ^C if your authentication has timed out by the time you get around to killing it.

#!/bin/bash

function cleanup()
{
    sudo kill $job_pid
    wait $job_pid
    exit 0
}

trap cleanup SIGTERM
trap cleanup SIGINT

sudo long_running_foreground_process &
job_pid=$!
wait

>>> from ctypes import CDLL
>>> libc = CDLL('libc.so.6')
>>> print libc.strlen('abcde')
5

As with everything else in python, it gets even better when you scratch the surface. In the example above, CDLL returns an object that represents the dynamic library. You can access the functions in that library by attribute access (“libc.strlen”) or item access (“libc['strlen']“). Both access mechanisms return a callable object.

This callable object has an “errcheck” attribute that can be assigned a callable. We can use this for error-checking our calls into the library. Let’s write a simple version of the “kill” command that uses the kill(2) system call.

import sys
from ctypes import *

# Load the library.
libc = CDLL('libc.so.6')

# Our error checking function. This will receive the
# return value of the library function, the function that
# was called, and the arguments passed to the function as a
# tuple.
def kill_errcheck(retval, func, funcargs):
    '''Check for error -- retval == -1.'''
    if retval < 0:
        raise Exception('kill%s failed' % (funcargs, ))
    return True

# Get the kill function from the standard library.
kill = libc.kill

# Set the error checker for kill().
kill.errcheck = kill_errcheck

# Pass the command line argument as a pid to kill, with
# SIGSEGV (11).
pid = int(sys.argv[1])
kill(pid, 11)

Save this as kill.py. Then, in your shell, try something like this:

# Notice that the 3401 is the pid of the process
# we're putting into the background. Yours will
# be different.
bash$ sleep 120&
[1] 3401
bash$ python kill.py 3401
[1]+ Segmentation Fault         sleep 120
bash$ python kill.py 3401
Traceback (most recent call last):
  File "kill.py", line 17, in
    kill(pid, 11)
  File "kill.py", line 10, in kill_errcheck
    raise Exception('kill%s failed' % (funcargs, ))
Exception: kill(3401, 11) failed

At line 4 of the output we run sleep in the background. At line 5 we learn the pid of this process. At line 6 we run our kill program, giving it the pid we just spawned, and we see the notification from bash that the process was killed (with signal 11, segmentation fault). At line 8 we run our kill program again on pid 3401, but it doesn’t exist, the kill system call returns -1, and our error checker raises an exception when it detects the system call failure.

But wait, there’s more… I’m working on a follow up post that combines ctypes.Structure with calls into a linux system call.