Paged Diffing

Many times I require my svn diff output to be paged through less. Rather than append | less continually, I created the following wrapper script as ~/bin/svn:

#!/bin/bash
[ "$1" = "less" ] && shift && exec /usr/bin/svn diff "$@" | less
exec /usr/bin/svn "$@"

With this script, I have a new svn less operation that pages diff output.

Note that ~/bin/svn must be before /usr/bin in your $PATH, or the real svn binary will take precedence over the wrapper.

Verbose Commits

I would wager that many git users type git commit -av more than once a day to commit all changed files and display the current diff in their editor. Very handy if you need a reference when creating your commit log message. Subversion will tell you what files have changed, but does not display the diff. I use the following vim mapping, adapted from the Vim wiki:

map <leader>d :vnew<CR>:read !svn diff<CR>:set syntax=diff buftype=nofile<CR>ggdd

This command opens a new window vertically, then fills that window with the diff of the current directory. Unfortunately this is of only limited usefulness: since vim does not know what files you are committing, it displays a diff of all changes in the current directory. Also, svn commit with no arguments seems to move the current directory up one level, which can give an inaccurate diff. I use svn commit * when I need to work around this feature. The wiki page has some variations that appear to work around these problems.

The incoming traffic (blue line) was maxed out at my bandwidth cap, even though I wasn’t downloading anything. After a few failed attempts at figuring out what was going on, I started closing browser tabs, resulting in the sustained dropoff near the end of the graph. I reopened the tabs (small bump closer to the end) but the incoming traffic did not spike.

Gmail and a few other pages were open, but nothing that would explain the traffic spike I saw. Maybe the ghost in the machine was watching Hulu.

Looking through the archives, it seems I never documented the level 70 ding for Cetoh. The screenshot has been lost, probably to a dead hard drive, so the exact date escapes me. But, the rogue was first to 60, the priest was first to 70, and the death knight was first to 80. Old characters lag behind: the rogue sits at 68, and the priest hasn’t moved past 70. Gotta wonder what I’ll be playing at level 85.

The world looks a bit different since I last hit the cap. Cataclysm has been announced, and promises to reshape Azeroth. The achievements system has been brought online. Mounts are available at level 20. People have started getting their Reins of the Violet Proto-Drake.

My death knight is Exalted with two factions (my first two on any character). 1,008 quests brought her from level 55 to level 80, over the course of nearly 11 months. (Winning a duel is a requirement of the death knight starter quest line, and my character became Duel-icious on 2008-11-13.) She Toured the Fjord, got Bored in Borean, saw the Might of Dragonblight, and was just two quests shy of that Grizzly Shizzle at ding. She has caught 1,405 fish and “other things”; overall her skills languish at Cooking 206, Fishing 340, Herbalism 158, and Mining 169. She is still waiting for Artisan flying.

The ssh client included in OpenSSH includes a very easy to use SOCKS5 proxy server: ssh -D 9999 server.example.com opens a normal connection to server.example.com, but also creates a SOCKS5 server at localhost:9999 that routes traffic from supported applications through the remote server. Firefox includes support for SOCKS5, and add-ons like the Web Developer toolbar and FoxyProxy can quickly enable or disable the proxy settings. I’ve used this to great success in the past, but, as with most things, there is room for improvement.

Server names and IPs have been modified from their original values.

Fine-tuning Firefox

While you can provide Firefox with a simple SOCKS5 host and port, the software also supports proxy auto-config (PAC) files. Both Firefox (natively) and Safari (via Network preferences) support PAC. The following PAC file tells the application to route traffic for anything at example.com through the proxy server, and directly connect to everything else. This keeps your non-business traffic from routing through business servers, so you can stream Netflix and still access those servers behind the firewall.

function FindProxyForURL(url, host)
{
	if(shExpMatch(url, '*.example.com/*')) return "SOCKS localhost:9999";
	return "DIRECT";
}

Proxy Everything

While ssh can create a proxy server, there is no built-in support for connecting to a host through a proxy. The same is true for many other applications, particularly console apps. Enter tsocks, a wrapper library that intercepts TCP calls and routes them through a SOCKS proxy. tsocks exploits something call preloaded libraries: when ssh is launched, special flags in the shell force a load of the tsocks library. My preferred way of running tsocks is to run it with no arguments, which sets up the preloading for any program run in the shell from that point forward.

Marc Abramowitz did some great work getting tsocks to compile under Mac OS X. I made a few tweaks to his patch, and release it here. To apply the patch, take the same steps that Marc mentions on his site, but use my patch instead.

After tsocks is installed, running something through your proxy is easy.

$ ssh -f -N -D 9999 host.example.com
$ tsocks
$ ssh firewalled.example.com

The first line creates a proxy server, the second prepares tsocks, and the third opens an SSH connection using the proxy defined in /etc/tsocks.conf. For completeness, here is my tsocks.conf:

local = 10.0.0.0/255.0.0.0

path {
	reaches = 129.21.0.0/255.255.0.0

	server = 127.0.0.1
	server_port = 9999

	server_type = 5
}

That’s all. Secure access to firewalled servers through HTTP and SSH. Works great, as long as you have one server that can be used as a gateway (available for SSH from outside the network, but able to access things inside).

I could not find a way to remove the partition via the Mac OS X Disk Utility, so I resorted to some terminal work. By clobbering the first few bytes of the disk’s partition map, I essentially made the whole disk look like random garbage, and TrueCrypt no longer presented a warning. This will cause data loss, so be careful.

First, find the drive’s device number using Disk Utility. Right click on the drive (not one of the partitions) and select “Information.”

Look for “Disk Identifier” in the property list. In this case, my drive is called disk3, which corresponds to /dev/disk3 on the filesystem. Tomorrow it might be disk4 or disk6, so don’t make any assumptions here. Failure to find the correct drive number in this step will cause you to erase the wrong drive in a future step.

Open up your terminal and umount this disk’s volumes:

• sudo umount -f /dev/<identifier>*

Unmounting drives in the Finder will make the disk unavailable in /dev, and we need this block device if we’re going to clobber the filesystem.

Once the partitions are unmounted, we can write junk to the partition table and essentially unformat the drive. This is the dangerous part, so don’t screw it up or you will destroy the wrong drive, lose all your data, and seriously hate life.

• sudo dd if=/dev/zero of=/dev/<identifier> bs=1 count=1024

After writing zeroes to the front of the drive, TrueCrypt happily made an encrypted volume on the disk.

My biggest problem with being a data packrat is organization. It took me countless searches to find the above article at scienceblogs.com, in part because I first combed through a number of my own repositories before falling back on Google searches. Did I put that in Delicious? Is it in my feed reader’s history? Repeat that for 20 minutes.

It’s revealing to enumerate all the places I might put a piece of data:

1. Twitter
2. P2 blog
3. Delicious
4. Google Reader (which offers “share,” “like” and “star,” as well as custom tags)
5. MediaWiki (public)
6. MediaWiki (private, mostly a relic from before Google Docs)
7. This blog
8. DEVONthink
9. Firefox bookmarks (I generally avoid these)

That doesn’t even hit possible storage locations for images. I can tick off five of those without any effort. Most of these repositories will become obsolete over time, having fallen out of fashion or been replaced by something bigger and better. Some data moves forward into the new tool, some stagnates out of sight.

I have no answers, here.

I will begin taking this for granted immediately. Thanks, Firefox 3.5b4.

Update: Ogg Theora has been backed out of the HTML5 draft spec due to lack of consensus from the big browser makers. Like the <img> tag, support for formats will be left in the hands of developers. A bit sad, really.

It’s too bad I’ve had such a hard time wrapping my head around the programmable completion toolkit, complete and compgen.

Getting There

I have a function that works like cd, but prepends a specific directory. Our web files are stored in /some/dir/webapp, and I want that directory at my fingertips at all times. Here’s the function:

wa() { cd /web/pscpages/webapp/$1 ; }

With this function, wa brings me to webapp; wa project1 brings me to webapp/project1; and so on. I just provide the full sub-path from webapp. Ideally, I would be able to tab-complete directories in webapp.

complete can pull a list of possible completions from a number of sources: “actions” (like files, directories, commands, shell keywords), command output, a wordlist separated by some whitespace, or the output of a bash function, to name a few. What you’ve typed so far (the “current word”) will be used to filter all the possible completions returned by that source. Say you’ve typed “pro” and then hit tab to autocomplete. The returned completions need to match “pro” at the start of the string, meaning you can’t match against absolute paths like /some/dir/webapp/project1.

compgen can be used to generate a list of possible completions. Matches will be output one per line, and can be piped around for transformations just like any other shell command.

Between these two tools, we have everything we need to autocomplete paths starting in a certain directory. Here’s a compgen that gives us directories matching a specified string:

compgen -d /some/dir/webapp/

Sample output:

/some/dir/webapp/.svn
/some/dir/webapp/project1
/some/dir/webapp/templates
/some/dir/webapp/images

We need to trim leading directories so “pro” matches “project1.” We should also append / to the pathnames, since we’re always matching directories:

compgen -S/ -d /some/dir/webapp/ | cut -b 18-

Playing around with compgen’s arguments, we can further filter the completion list by appending to our string, sort of an implied glob. Use /some/dir/webapp/p, and subdirectories starting with “p” will be returned. This is exactly what we want: compgen takes care of all the filtering for us. We have access to a couple special variables to examine the word the user is expanding. For now, it’s enough just to grab ${COMP_WORDS[COMP_CWORD]} and append it to our path.

When completions are generated by a function, they’re passed back to complete by the $COMPREPLY environment variable. Pulling this all together, we can now create our completion function:

_webapp() {
    local cur
    cur=${COMP_WORDS[COMP_CWORD]}
    COMPREPLY=( $( compgen -S/ -d /some/dir/webapp/$cur | cut -b 18- ) )
}

All that’s left is to tell bash to use this function to complete our argument to wa.

complete -o nospace -F _webapp wa

The Fruits

So, that does it. Our original wrapper to cd, combined with our autocomplete functionality, looks like this:

wa() { cd /some/dir/webapp/$1 ; }
_webapp() {
    local cur
    cur=${COMP_WORDS[COMP_CWORD]}
    COMPREPLY=( $( compgen -S/ -d /some/dir/webapp/$cur | cut -b 18- ) )
}
complete -o nospace -F _webapp wa

Voila. Tab completion in a directory that’s not $PWD, and it even works with subdirectories. I hope this makes autocompletion a little clearer for others.

Here’s a small sample of personas. I usually hide some toolbars for Firefox screenshots, but personas tend to look better with a larger canvas.

Hulu’s desktop application has been released.