pdiff: Comparing process output
The diff command is an essential piece of any programmer’s or administrator’s toolbox, to show the difference between two files. Or if not that tool exactly, there are many like it. But sometimes I want to compare the output of two commands, often really the same command operating on different inputs. You could save the output of each to separate files and diff that, or in a shell like bash you can use “<(cmd)” fed directly to diff. For example, when I was recently dealing with an elfutils bug with prelinked binaries, I ran a command like this:
diff <(eu-readelf -hlS foo) <(eu-readelf -hlS foo.prelink)
That works just fine, though it’s a little redundant, but it gets really bad if the command gets much more complex. And if you need to adjust the command, you have to do it in both places, or the diff falls apart. So after getting annoyed by this, I decided to scratch that itch and write this “pdiff” script:
#!/bin/bash CMD= CMDARG= DIFF= OPTS=$(getopt -o d:gvc:I: -n pdiff -- "$@") || exit eval set -- "$OPTS" while true; do case "$1" in -d) DIFF="$2"; shift 2;; -g) DIFF=gvimdiff; shift;; -v) DIFF=vimdiff; shift;; -c) CMD="$2"; shift 2;; -I) CMDARG="$2"; shift 2;; --) shift; break;; *) echo "Internal error!" >&2; exit 2 ;; esac done function run() { local cmd test -n "$CMDARG" && cmd="${CMD/$CMDARG/$@}" || cmd="$CMD $@" echo "# $cmd" eval $cmd } case $# in 2) exec ${DIFF:=diff -u -p} <(run $1) <(run $2);; 3) exec ${DIFF:=diff3} <(run $1) <(run $2) <(run $3);; 4) test -n "$DIFF" && exec $DIFF <(run $1) <(run $2) <(run $3) <(run $4);;& *) echo "Unable to diff $# args" >&2; exit 1 ;; esac
It even has option processing, like -v to open in Vim, -g for gVim, or -d for a custom diff command. Then -c gives a shared command, and -I lets you specify a replacement string within the command (rather than placing arguments at the end). Any remaining arguments get fed to separate invocations of the command for comparison. That’s well beyond the one-liner I started with, but now I can reuse this to make future comparisons easier. The first command I gave is now more simply:
pdiff -c 'eu-readelf -hlS' foo foo.prelink
That command can now get more complex, with compound pipelines or whatever else I need, without being such a pain to compose.
I did a little searching afterward and found that there are already other tools called pdiff, like here and here, so if I ever want to publish this more formally I should find a new name. But for now, it meets my needs, and maybe it can be useful for someone else too.
ssh-pageant 1.0
ssh-pageant now has its first release, version 1.0 — fulfill your authentication agent needs between Cygwin’s OpenSSH and Pageant today! It’s documented, and pre-built binaries are available. And there will be cake, of course.
Any “1.0” feels like it must have some magical aura, but in the end it is just a number. The program works as expected, and has the features I felt were needed to be complete, so 1.0 it is. The next step is to get this packaged and included in the official Cygwin repositories, which I would welcome volunteer help in doing.
Two tiny additions to FOSS
The quietness around here shows that I’m not much of a blogger. That’s ok, I guess — it’s not like I wanted to make a living from it. Well anyway, I’ve published a couple of tools I wrote a while ago, and they deserve at least a tiny announcement. So here are two new contributions from me to the FOSS world.
ssh-pageant is an SSH authentication agent for Cygwin that links OpenSSH to PuTTY’s Pageant. It acts very much like ssh-agent, except it leaves the key storage to Pageant. You can use ssh-pageant to automate SSH connections from the Cygwin shell, and I find this is most helpful for those services built on top of SSH, like SFTP file transfers or pushing to secure git repositories. It is even said to be literally life-saving.
MouseWinX is a Windows tray application that lets you quickly toggle X-mouse window activation, where the window focus follows position of the mouse cursor without having to click anything. I wrote this back in college when I was using Magic VLSI for one of my classes, and I’ve also found it helpful for navigating GIMP. Both are applications with multiple windows to interact with, and hot-tracking the mouse makes them a bit easier to use. But the rest of the time I don’t like having the window focus jump around, so MouseWinX gives a simple icon in the tray which toggles the setting when clicked.
Both of these projects are in an unpolished state, like so many open source projects. They work perfectly well, as far as I know, but I haven’t done the finishing touches, like writing documentation and packaging releases. For now, at least the source code is out there, and hopefully this post will help them be found by those in need of such tools.
Hacking Linux Filenames
I recently read an LWN article on David A. Wheeler’s essay, “Fixing Unix/Linux/POSIX Filenames.” The gist is that he thinks the filename rules are too permissive — we have ‘/’ as the path separator, and a raw 0 terminates the path, but anything else is fair game. On the surface, this has a certain beautiful simplicity to it. However, there are characters that have special meaning depending on the context, so almost any code that actually tries to interpret a filename will have to add a lot of complexity to be robust. The essay delves into many ways that things can go wrong.
Filenames have been this way in for a long time though, and I don’t expect that this will change officially anytime soon. Still, my day job now is developing SystemTap, and this sort of problem is one of many sorts that SystemTap can address. Here’s a script to show how a system administrator could patch the kernel with their own addendum to the filename rules:
#!/usr/bin/stap -g # badname.stp # Prevent the creation of files with undesirable names. # return non-zero if the filename should be blocked function filter:long (name:string) { return euid() && isinstr(name, "XXX") } global squash_inode_permission probe kernel.function("may_create@fs/namei.c") { # screen out the conditions which may_create will fail anyway if ($child->d_inode || $dir->i_flags & 16) next # check that the new file meets our naming rules if (filter(kernel_string($child->d_name->name))) squash_inode_permission[tid()] = 1 } probe kernel.function("inode_permission@fs/namei.c").return !, kernel.function("permission@fs/namei.c").return { if (!$return && squash_inode_permission[tid()]) $return = -13 # -EACCES (Permission denied) delete squash_inode_permission[tid()] }
The script starts by defining a filter function. It first check whether the effective user ID is non-zero, so the root user can bypass the filter. Then, for the prude admins out there, I’ve chosen to block filenames that contain the string “XXX”. I intentionally kept this part small for this example, but you could easily write a function covering all of the new rules that Wheeler suggests.
After that is a probe on the may_create
function, which is what the kernel calls to validate permissions for new files. We can call our filtering function from here to see if the filename is OK, but since may_create
is an inline, we don’t have a direct way to influence its result. The last thing may_create
does though is copy the result of inode_permission
(or permission
in earlier kernels), which we can override. So, we save the filtering decision in a global, and then in a return probe on inode_permission
, we can change the successful $return
code to our own error value. Now, any attempt to create a file that doesn’t pass our rules will get an error of “Permission denied”.
This sort of script is really just a band-aid, and it doesn’t do anything to deal with files that already have “bad” names. Still, I hope this is an interesting example of how easily one can modify kernel behavior with SystemTap. This script can be a starting point to define and try out your own filename rules, and changes can be reloaded and tested without ever having to reboot. Once your policy has been decided, you can configure the script to load as soon as the system boots, so you’re always running with your improved filename rules, even across kernel upgrades.
It’s powerful stuff, but don’t let it get to your head… 🙂
Free Wireless
I made an upgrade on my netbook today. Can you spot the difference?
If you guessed that it’s now 100% open-source compatible, you are correct!
Even though I ordered the Linux package, Dell cheaped out with a Broadcom wireless card that doesn’t have very good Linux support. Broadcom does have drivers available, but you have to download and compile the wrapper yourself. Downloading a new driver and all of the kernel-devel packages is a little harder when your network is not connected…
So to replace the Broadcom, I ordered an Intel 3945ABG card, because it is well supported in Linux. The kernel has the right drivers already included, so Fedora works right out of the box, even booting off of a live cd. And not only are the drivers open-source, but they even work better. My connection time shrunk from 20-30 seconds down to about 5 seconds. I can live with that!
Optimism
These election results are as good a reason as any to break blog silence — congratulations to our President-elect, Barack Obama! And while I’m sad about California’s current results on Prop 8, overall I feel much more optimistic about our immediate future.
Obama is not the messiah, and he doesn’t have a magic wand to solve all of our problems, but he has inspired me and so many others in a way I’ve not experienced before. I hope to see great things in the next four years…
Broken Psychedelics
This is a DLP color wheel:
This is a DLP color wheel on drugs:
And sadly, that’s a picture of mine. 😦
On the plus side, it was pretty easy for me to get to the part, so I’m going to order a new one and try the replacement myself. Professional TV repair costs way too much for my taste. Anyway, I would like to be more handy around the house, and starting with something I know (electronics) seems like a safe bet.
Update (5/19): Success! It wasn’t that much different than changing parts in my computer, which I’ve done for years, except this had the added stress of moving around delicate optics. Time to catch up on everything my DVR saved for me…
WoT will be completed
Pardon the slight pun, but there’s now an end in sight for The Wheel of Time!
[…] Brandon Sanderson has signed on to complete A Memory of Light, with scheduled delivery of the manuscript in December 2008 and a planned publication date of Fall 2009.
Two years! I guess that’s not so bad. I’ve not read anything by Brandon Sanderson before, but maybe while I’m waiting for this one I’ll read some of his own books. I can’t imagine the pressure — Jordan’s fans will never forgive Sanderson if he screws this one up. At least he’s not starting from scratch though, as Jordan wrote a large part of the book before his passing and had notes on much of the rest. So here’s hoping for a proper closure to one of my favorite series…
Cold Battlestar
I’ve got a bit of a cold right now, and after working all weekend to meet a deadline, I’m pretty tired too. So, do you think I might pass on attending an advance screening of Battlestar Galactica: Razor? No frakkin’ way!
The movie-sized episode fills in more of the background of the Pegasus, and it doesn’t disappoint. It’s been a while since the last season finale, and this was perfect to whet my appetite. I’m sure I’ll watch it again when it airs on SCI FI; I’m just not sure whether it’s worth the standalone DVD investment. I do want to have it, but I already got hit when I bought the miniseries and then found that it was also included in the season one set. I have a feeling that they’ll do the same with Razor and season four. Of course, maybe Santa will be nice and I won’t have to worry about it…
Anyway, here’s hoping that the writer’s strike is resolved soon, so I can look forward to season four without interruption…
Inconvenient world affairs
I finally watched An Inconvenient Truth this weekend, and it was definitely eye-opening. I was annoyed a little by Gore tooting his own horn — the election scandal was especially off-topic — but the message is still powerful. While I generally understood and believed already that global warming is an issue, I didn’t know the raw scale of it. So now that I’ve received the warning, I’ll start watching for ways that my own habits can be improved, and I encourage others to do the same.
On another side of humans deteriorating our own condition, we have Musharraf declaring emergency rule in Pakistan. I remember seeing him as a guest on The Daily Show, and he seemed like a genuinely nice guy. He has the charisma that makes for a powerful leader, which makes his current actions even more frightening. People will genuinely believe in him and his justifications for extreme actions, much more than the puppet show that our own administration puts on. I hope for Pakistanis that this can be resolved peacefully, but it’s hard to see how when he has already defied diplomatic efforts.
It’s funny — I’m still one of many young adults who get their primary news from Comedy Central, but I’m developing an interest in politics and world affairs that used to bore me. Maybe this is just part of getting older…