#!/bin/sh

# From /media/cryptofs/apps/usr/palm/applications/org.webosinternals.ubuntu-natty-chroot/bin

CHROOT=/media/ext3fs/ubuntu-natty-chroot

cp /etc/resolv.conf ${CHROOT}/etc/resolv.conf

mount --bind /dev ${CHROOT}/dev
mount --bind /dev/pts ${CHROOT}/dev/pts
mount --bind /proc ${CHROOT}/proc
mount --bind /sys ${CHROOT}/sys
mount --bind /tmp ${CHROOT}/tmp

chroot ${CHROOT} /bin/bash

umount ${CHROOT}/tmp
umount ${CHROOT}/sys
umount ${CHROOT}/proc
umount ${CHROOT}/dev/pts
umount ${CHROOT}/dev

#!/bin/sh

set -u
set -e

# non-wp tables
mysqlhotcopy --addtodest 'wordpress./^(?!wp_\d)./' /backup/

# wp tables not starting with a digit
mysqlhotcopy --addtodest 'wordpress./^wp_[^\d]/' /backup/

# tables 1-99
mysqlhotcopy --addtodest 'wordpress./^wp_\d{1,2}_/' /backup/

# tables 100-999
mysqlhotcopy --addtodest 'wordpress./^wp_\d{3}_/' /backup/

# tables 1000-1499
mysqlhotcopy --addtodest 'wordpress./^wp_1[0-4]\d{2}_/' /backup/

# tables 1500-1999
mysqlhotcopy --addtodest 'wordpress./^wp_1[5-9]\d{2}_/' /backup/

# tables 2000-2499
mysqlhotcopy --addtodest 'wordpress./^wp_2[0-4]\d{2}_/' /backup/

# tables 2500-2999
mysqlhotcopy --addtodest 'wordpress./^wp_2[5-9]\d{2}_/' /backup/

Scary.

• Encrypted file transfer
• Ability to work without interaction
• Secure (i.e. locked down to specific tools/files)

OpenSSH public/private keypairs and rsync over SSH was a logical starting point, but I was missing a piece to limit the rsync to specific files: the authorized_keys command="command" option.

The Setup

Hostnames have been changed to protect the innocent. For this description:

• coriander is the server, which has some MySQL files we need to back up
• penny is the client, pulling files from coriander using rsync

I regularly SSH to the host I wanted to backup, so this setup needed to work around a potential key conflict. Easy enough using ~/.ssh/config on penny:

Host coriander
    HostName coriander.example.com
    User adam
    IdentityFile ~/.ssh/id_dsa

Host mysql-binlog
    HostName coriander.example.com
    User adam
    IdentityFile ~/.ssh/mysql-binlog-key

I run an ssh-agent, so I have to run things through env -i to prevent the rsync from using my normal key.

In this case, I manually copied my key into adam@coriander‘s ~/.ssh/authorized_keys file:

ssh-rsa AAAAB3NzaC1yc2EAAAA…BIwAAAQEA2GNx7diU== mysql-binlog-key

Next I prepended some options to this key:

command="~/mysql-backup/rsync-control" ssh-rsa AAAAB3NzaC1yc2EAAAA…BIwAAAQEA2GNx7diU== mysql-binlog-key

My wish is your command=”command”

At this point, let’s digress to look at the SSH command option. Try this style authorized_keys on your server to echo the environment and exit when the user logs in over SSH:

command="env" ssh-rsa AAAAB3NzaC1yc2EAAAA…BIwAAAQEA2GNx7diU== mysql-binlog-key

SSH to the host and see the output:

ambackstrom@fsck:~:0$ env -i ssh mysql-binlog
SHELL=/bin/bash
SSH_CLIENT=10.0.0.2 56490 22
USER=adam
PATH=/bin:/usr/bin
PWD=/home/adam
SHLVL=1
HOME=/home/adam
SSH_CONNECTION=10.0.0.2 56490 10.0.0.1 22
_=/usr/bin/env

Then once more, specifying a command to run on the server:

ambackstrom@fsck:~:0$ env -i ssh mysql-binlog 'ls -lAF'
SHELL=/bin/bash
SSH_CLIENT=10.0.0.2 56546 22
USER=adam
PATH=/bin:/usr/bin
PWD=/home/adam
SHLVL=1
HOME=/home/adam
SSH_CONNECTION=10.0.0.2 56546 10.0.0.1 22
SSH_ORIGINAL_COMMAND=ls -lAF
_=/usr/bin/env

Our script becomes the middleman between the client and the requested command, and that command is placed in the $SSH_ORIGINAL_COMMAND environment variable. We can analyze this command and allow, deny, or modify it before execution. Given this rsync command:

env -i rsync -avzP mysql-binlogs:/tmp/ /tmp/test/

We get this remote command:

SSH_ORIGINAL_COMMAND=rsync --server --sender -vlogDtprz . /tmp/

Security via Misdirection

Let’s place a new middleman script in our authorized_keys file on coriander:

command="~/rsync-control" ssh-rsa AAAAB3NzaC1yc2EAAAA…BIwAAAQEA2GNx7diU== mysql-binlog-key

Within ~/rsync-control we’ll analyze the incoming command and take some action:

#!/bin/sh

if [ "$SSH_ORIGINAL_COMMAND" = "rsync --server --sender -vlogDtprz . logs/" ] ; then
    rsync --server --sender -vlogDtprz . /var/lib/mysql/binlog/
    exit $?
fi

exit 1

What we’ve actually done here is obfuscated the real binary log directory from the originating command on penny: the client requests the logs/ directory, but we launch an rsync server for /var/lib/mysql/binlog/. If any other remote command is issued, the script exits with an error code.

I’m sure there are some other great uses for the command option. I know of git server like gitolite that take advantage of this feature, and I’d love to find more.

You’ll want to have the YUI Compressor and Google Closure Compiler installed for minification of CSS and JavaScript, respectively. I installed them to my ~/bin directory, and referenced that directory in my Makefile.

Note that I’m using these two pieces of software mostly out of habit; I haven’t evaluated them or their competitors recently.

The Meat

Customizing for Your Project

To make this file useful for your own project, you’ll need to point it at your CSS and JS files.

JS_TARGETS =
CSS_TARGETS =
CLEANUP =

The first two variables define what scripts will be minified and compressed when you type “make js” or “make css,” respectively. Both of these commands will run when you type “make all” or simply “make.” CLEANUP allows you to specify additional files that will be removed when you type “make clean.”

Some sample customizations are mentioned in the file.

CSS_TARGETS = $(shell cat manifest.txt)

If you would rather organize your list of CSS or JavaScript targets into their own files, you can automatically expand that manifest file using the cat command.

CLEANUP = $(CSS_TARGETS) $(JS_TARGETS)

Some more advanced setups may combine several CSS files before minification, or use a custom target to concatenate JavaScript using the closure compiler. If you find that all your targets can be cleaned, you can simply reference them automatically as above.

concatenated.min.js: file1.js file2.js
	java -jar ~/bin/compiler.jar $(addprefix --js=,$^) >$@

The closure compiler requires a prefix for all input JavaScript files. Use the make function “addprefix” to format the argument list.

custom-concat.css: file1.css file2.css file3.css
	cat $^ >$@

Automatic variables greatly simplify most of the functionality within this makefile. To create a concatenated CSS file, simplify specify the target to the left of the colon, and its dependencies (the files to concatenate) to the right; the cat command never needs modification.

Of course, you can always define custom rules, ie. this one to fetch the newest development version of jQuery:

jquery:
	curl -o jquery.js http://code.jquery.com/jquery-git.js

Other Magic

Some other goodness happens below the “you shouldn’t need to edit past here” line. After we configure a few settings, we define how to create files based on suffixes: .min.css gets passed through YUI Compressor, .min.js through Closure, and .gz through gzip. We configure what files we care about, and make handles the rest.

.DEFAULT_GOAL := all

Set a default rule, rather than using the first rule in the file as the default.

.PHONY: css js

Prevent files named “css” or “js” from interfering with the css and js rules.

%.gz: %
	gzip -9 <$< >$@

One of three pattern rules. This enables you to compress any file (not just CSS or JS) by running “make filename.gz.” I precompress my files because I’m using gzip_static in nginx.

CSS_GZIP = $(CSS_TARGETS:.css=.css.gz)

One of several variables that modifies filenames found in $(CSS_TARGETS), changing the suffix from .css to .css.gz. Note that the colon/equals syntax is shorthand for patsubst. Given CSS_TARGETS = text.css tables.css, we would get CSS_GZIP = text.css.gz tables.css.gz, which would be sent as targets to the more general css rule, and in turn would be handled by the general-purpose %.gz rule above.

See Also

• Advanced Makefile Tricks — macros ($@ et al) and wildcard rules
• GNU make manual — functions, automatic variables, pattern rules

Before the Storm: Finnix

I had no experience with Finnix before this upgrade, but it was exactly what I needed throughout the process. It’s built into the Linode dashboard so you can use it as a recovery console if things go wrong. I also used it as my boot disk while cloning the Linode hard disk to my VM.

The Backup

rsync is my weapon of choice, since it allows for transport over SSH and incremental backups. I used this article to get started with some suggested rsync flags. I ran this several times throughout the process:

rsync -avzPH --numeric-ids --delete --delete-excluded --exclude-from=backup.lst root@server.example.com:/ /mnt/sda1/

Here’s my backup.lst exclusion file:

+ /dev/console
+ /dev/initctl
+ /dev/null
+ /dev/zero
- /dev/*
- /proc/*
- /sys/*
- /tmp/*
- *lost+found

After rsync, chroot to the cloned filesystem:

mount -R /proc /mnt/sda1/proc
mount -R /dev /mnt/sda1/dev
chroot /mnt/sda1 /bin/bash
vi /etc/fstab # update mountpoints. change xvda to sda1, xvdb to sda2

Install a bootloader. The Linode VPS is in Xen and doesn’t normally boot its own kernel. (I know next to nothing about Xen, but this is what I’ve gleaned.)

apt-get install grub
mkdir -p /boot/grub
cp -r /usr/lib/grub/i386-pc/{stage1,stage2,e2fs_stage1_5} /boot/grub
apt-get install linux
echo defoptions=vga=791 >>/boot/grub/menu.lst
update-grub

Reboot. Make sure you kill your cron jobs while the VM is on, as many are probably inappropriate for a backup server.

sudo service cron stop

I recommend taking a snapshot of your backup, here. It may actually be faster to just rsync again, but it’s nice to have a complete backup at the ready.

I used a couple commands to update IP addresses on my backup so I could more accurately test services post-upgrade:

sudo ~/bin/ack --follow -al '207\.192\.74\.235' /etc | sudo xargs sed -i.bak 's/207\.192\.74\.235/172.16.226.130/g'
sudo ~/bin/ack --follow -al '69\.164\.216\.5' /etc | sudo xargs sed -i.bak 's/69\.164\.216\.5/172.16.226.131/g'

The Upgrade

The following notes were taking during my test upgrade, and referenced during the real upgrade. I referenced the Ubuntu Community Documentation project’s “Upgrading to Ubuntu 10.04 LTS” page. There is also a Linode article on upgrading to Ubuntu 10.04.

On the first pass, python-setuptools killed my install. This was fixed by manually reinstalling before upgrade:

sudo dpkg -r python-setuptools && apt-get install python-setuptools

Ensure the update manager is installed, and issue the upgrade command:

sudo apt-get install update-manager-core
sudo do-release-upgrade

I encountered conflicts in the following packages, where I had modified local files. Also included is my resolution, which may or may not be helpful to anyone else.

• /etc/securetty — keep currently installed version; package adds some unneeded ones, and removes tty0 (but does include the Xen ones added by Linode)
• /etc/sysctl.conf — use new; uncomment “kernel.prink = 4 4 1 7” and add “fs.inotify.max_user_watches = 524288“
• /etc/mysql/my.cnf — kept my version
• /etc/dhcp3/dhclient.conf — kept Linode version: “request subnet-mask, broadcast-address, time-offset, routers, domain-name, domain-name-servers, host-name, ntp-servers;“
• /etc/dbconfig-common/config — nothing meaningful in the diff; use new
• /etc/dovecot/dovecot.conf — kept mine
• /etc/default/openvpn — was missing; installed package version
• /etc/snmp/snmpd.conf — kept mine
• /etc/apache2/apache2.conf — kept mine, as well as other apache2 files (but I did use the new mime.types file)
• /etc/stunnel/stunnel.conf — kept mine
• /etc/default/stunnel4 — kept mine
• /etc/udev/rules.d/75-persistent-net-generator.rules — kept installed, looked like the GOTO="persistent_net_generator_end" at the start was intentional

Got a warning about vulnerable SSL keys generated by older versions of Ubuntu; solution is to regenerate your keys. The dialog recommended using openssl-vulnkey and openvpn-vulnkey to test.

Other than having to resolve the file conflicts, the upgrade was very smooth. I found some notes about fixing the Xen serial console, and syslog breaking in 10.04. I also upgraded to a newer paravirt kernel in the Linode dashboard.

python-setuptools error

More about that error with python-setuptools: this thread was helpful. From /var/log/apt/term.log:

Log started: 2010-10-06  08:35:15
Setting up python-setuptools (0.6c9-0ubuntu1) ...
pycentral: pycentral pkginstall: not overwriting local files
pycentral pkginstall: not overwriting local files
dpkg: error processing python-setuptools (--configure):
 subprocess post-installation script returned error exit status 1
Errors were encountered while processing:
 python-setuptools
Log ended: 2010-10-06  08:35:15

Per the article, ran “apt-get remove python-setuptools” and “apt-get install python-setuptools”

Post-upgrade Recompiling

Something required apt-get install libltdl-dev. Useful info for this post, no?

I recompiled all my custom stuff, including nginx, memcached, mod_php and php-fpm, and mysqld.

I’ve been trimming down my Inbox, filing things more aggressively and unsubscribing from the dozens of newsletters that have accumulated over the years. I’ve also created a Django app to log cron output, simultaneously reducing the amount of email I get and enabling me to make cron output more verbose. Recent log entries are exposed via an Atom feed, which I’ve plugged into Google Reader (above).

I spend a lot of time in Google Reader, but it’s stuff like this that makes it feel truly useful rather than just convenient.

It’s too bad I’ve had such a hard time wrapping my head around the programmable completion toolkit, complete and compgen.

Getting There

I have a function that works like cd, but prepends a specific directory. Our web files are stored in /some/dir/webapp, and I want that directory at my fingertips at all times. Here’s the function:

wa() { cd /web/pscpages/webapp/$1 ; }

With this function, wa brings me to webapp; wa project1 brings me to webapp/project1; and so on. I just provide the full sub-path from webapp. Ideally, I would be able to tab-complete directories in webapp.

complete can pull a list of possible completions from a number of sources: “actions” (like files, directories, commands, shell keywords), command output, a wordlist separated by some whitespace, or the output of a bash function, to name a few. What you’ve typed so far (the “current word”) will be used to filter all the possible completions returned by that source. Say you’ve typed “pro” and then hit tab to autocomplete. The returned completions need to match “pro” at the start of the string, meaning you can’t match against absolute paths like /some/dir/webapp/project1.

compgen can be used to generate a list of possible completions. Matches will be output one per line, and can be piped around for transformations just like any other shell command.

Between these two tools, we have everything we need to autocomplete paths starting in a certain directory. Here’s a compgen that gives us directories matching a specified string:

compgen -d /some/dir/webapp/

Sample output:

/some/dir/webapp/.svn
/some/dir/webapp/project1
/some/dir/webapp/templates
/some/dir/webapp/images

We need to trim leading directories so “pro” matches “project1.” We should also append / to the pathnames, since we’re always matching directories:

compgen -S/ -d /some/dir/webapp/ | cut -b 18-

Playing around with compgen‘s arguments, we can further filter the completion list by appending to our string, sort of an implied glob. Use /some/dir/webapp/p, and subdirectories starting with “p” will be returned. This is exactly what we want: compgen takes care of all the filtering for us. We have access to a couple special variables to examine the word the user is expanding. For now, it’s enough just to grab ${COMP_WORDS[COMP_CWORD]} and append it to our path.

When completions are generated by a function, they’re passed back to complete by the $COMPREPLY environment variable. Pulling this all together, we can now create our completion function:

_webapp() {
    local cur
    cur=${COMP_WORDS[COMP_CWORD]}
    COMPREPLY=( $( compgen -S/ -d /some/dir/webapp/$cur | cut -b 18- ) )
}

All that’s left is to tell bash to use this function to complete our argument to wa.

complete -o nospace -F _webapp wa

The Fruits

So, that does it. Our original wrapper to cd, combined with our autocomplete functionality, looks like this:

wa() { cd /some/dir/webapp/$1 ; }
_webapp() {
    local cur
    cur=${COMP_WORDS[COMP_CWORD]}
    COMPREPLY=( $( compgen -S/ -d /some/dir/webapp/$cur | cut -b 18- ) )
}
complete -o nospace -F _webapp wa

Voila. Tab completion in a directory that’s not $PWD, and it even works with subdirectories. I hope this makes autocompletion a little clearer for others.

1. :syn on. These colors suck. Look for themes.
2. 4 bit themes suck. Let’s set up an 8 bit color terminal.
3. Leopard’s Terminal.app fails, install iTerm 0.9.6.
4. iTerm works, but I use GNU screen all day. Recompile screen with --enable-colors256.
5. screen wins, but doesn’t play nice with the delete key in iTerm. Fix it.
6. Compile Vim 7.2 for 8 bit support.
7. Get an 8 bit theme. set t_Co=256 in .vimrc.
8. " syntax highlight html files using the php highlighter
autocmd BufRead,BufNewFile *.html set filetype=php

9. Install FuzzyFinder for good measure now that we’re out of Vim 6.3 territory.
10. Start saving seconds/minutes/hours of my life that before were consumed by syntax errors.

Installation

During the machine setup, I was prompted to do a “Linux Easy Install.” The software auto-populated my full name and username, and asked me for a new password. I let it do its thing, having already pointed it to the Ubuntu installation ISO. A short while later, with no additional interaction, I was at the Gnome login screen. The username and password I had chosen during installation worked great.

Interface

The feature that finally pushed me over the 2.0 edge was Unity support in Linux. In the past I have only run server applications on my VM, but I had to see Linux Unity in action, layering my Linux applications with Mac OS X applications. It doesn’t disappoint.

At the other end of the spectrum is “Headless” mode. I’ve been waiting for this for a long time. When I’m using the server capabilities of my VM, I don’t want an extra application cluttering up my alt-tab. Now: “Enter Headless,” quit VMWare Fusion. Simple. I assume the VM will suspend if I shut down the computer, but I haven’t tested it yet.

The new library window is a bit more attractive, and the machine settings list is more compact. The list also shows a live thumbnail of your VM’s display, be it console or graphical. Along the same lines, the console display can now be resized to zoom the terminal text in or out.

The Gnome application menu is also accessible from the Mac OS X Dock. Apps can be launched in Unity mode without restoring the Gnome desktop.

Seamless cursor movement between the Linux and Mac desktop, along with VM resolution adjusting to the size of the machine window, are also great features, though I suspect they were present in Fusion 1.0.

Some Other Section

Looks like the VMware Fusion team has been up to good things since the first release. I haven’t yet tried Parallels, but I can’t see a reason to try anything new from where I’m sitting.