Linux on Hashbang.ca

Diagnosing performance degradation under adverse circumstances

Wed, 03 Jun 2015 21:31:00 -0700

[This post is a few years old and was never published. Recently, I was reminded about memcached slab imbalance, which in turn reminded me of this post.]

At work, we encountered a sudden and precipitous performance regression on one particular page of a legacy application. It’s a Perl web application, running under mod_perl, using ModPerl::RegistryLoader to compile scripts at server startup, and Apache::DBI to provide persistent database connections.

Our users suddenly began complaining about one particular page being “three times slower than normal.” Later examination of the Apache logs showed a 20x(!!) slowdown.

Investigating this performance problem was interesting because we didn’t have good access to required data, and our technology choices slowed us down or completely prevented us from collecting it. Although we solved the mystery, the experience had several important lessons.

Validating SSL certificates for IRC bouncers

Sat, 14 Sep 2013 10:28:00 -0400

IRC bouncers are sort of like a proxy. Your bouncer stays online, connected to IRC, all the time, and then you connect to the bouncer using a normal IRC client. I connect to my bouncer with an SSL-encrypted connection, but I hadn’t been validating the certificate until now. Validating the SSL certificate is critical for thwarting man-in-the-middle (MITM) attacks.

In a MITM attack, the victim connects to the attacker, thinking it is the service they want to talk to (the IRC bouncer in this case). The attacker then forwards the connection to the service. Both connections might use SSL, but in the middle, the attacker can see the plaintext. They can simply eavesdrop, or modify the data flowing in both directions. SSL is supposed to prevent that, but if you don’t validate the certificate, then you don’t know who you’re talking to. I want to know I’m really talking to my IRC bouncer, so let’s figure out how to validate that certificate.

Adding LUKS hard disk encryption on LVM after the fact

Sat, 15 Jun 2013 17:20:00 -0400

I have an external hard disk enclosure with two disks. I used Logical Volume Manager to create a single logical volume that spanned them, and slowly filled it to about 60% capacity. Lately, I’ve been trying to be more conscious of using encryption, and this was one area where I hadn’t done so. At the time I felt like learning how to do LVM was enough, and LUKS could wait until later. Well, it’s later. Here’s how I added encryption after the fact (without a spare hard disk).

Introducing mvr: like mv, but clever

Fri, 19 Apr 2013 19:11:00 -0400

I wanted to move a large number of files from one directory to another, but the target directory already had many of the filenames already used. This is a common enough problem – digital cameras use DSC#, video downloaders often append numbers to get a unique filename, and so on. In both those examples, the sequence restarts if you empty the program’s work directory. So, you’ll end up with DSC0001.jpg every time you empty your camera’s memory card. If you’re trying to move such files into a single directory, you’ll get conflicts every time.

Instead of manually renaming the files before transferring them, I wrote a simple script to give each file a unique name in the destination directory.

Book review: 'Coding Freedom' by Gabriella Coleman

Mon, 21 Jan 2013 22:35:00 -0400

I’ve just finished reading Gabriella Coleman’s new book “Coding Freedom: The ethics and aesthetics of hacking” (2013, Princeton University Press) which culminates over a decade of field research, in-depth interviews, observation, and participation in the hacker scene globally. In this case, “hacker” refers to the free/open-source software (FOSS) hacker, and in particular the Debian project.

Hacking is where craft and craftiness converge.

Automating server build-out with Module::Build

Thu, 08 Nov 2012 13:25:00 -0400

At Pythian, we have one application that is composed of several components, the deployment of which needs to conform to our slightly peculiar server setup. Until recently, this required manually deploying each component. I did this a couple weeks ago, and it took me something like 40 hours to figure out and complete. As I went, I started reading up on Module::Build, trying to figure out how to automate as much as possible. It turns out that this core module gives us a surprisingly powerful tool for customized deployment. First, it will help to understand a few aspects of how our code is deployed.

Introducing File::Symlink::Atomic

Sat, 26 May 2012 00:39:00 -0400

In Tips & tricks from my 4 months at Pythian, I showed how to give a symlink a new target atomically. I wasn’t aware of any module to encapsulate that, so I quickly put together File::Symlink::Atomic.

This module is useful because it eliminates the need to know how to do this safely - simply use File::Symlink::Atomic and you get a drop-in replacement for CORE::symlink. It creates a temporary symlink (using File::Temp to get a unique pathname) pointing to your new target, then moves it into place with a rename call. On POSIX systems, the rename system call guarantees atomicity.

Tips & tricks from my 4 months at Pythian

Sat, 19 May 2012 21:15:00 -0400

After working with Yanick Champoux on a few little Perl projects here and there, we finally met face-to-face at YAPC::NA last summer. A few months later, when I was looking for a co-op position, I immediately thought of Pythian.

Wherein I realize the bliss of writing init scripts with Daemon::Control

Mon, 16 Apr 2012 16:59:00 -0400

Init scripts are annoying little things – almost entirely boilerplate. Here’s how I learned to stop struggling, and love Daemon::Control to control my daemons.

The module really is as simple as the synopsis – you describe the daemon, have it write an init script (which actually just runs your Daemon::Control script) for you, then update-rc.d and you’re golden. It really is that simple.

Trimming whitespace in gedit with Perl

Sun, 14 Aug 2011 00:03:00 -0400

With gedit plugins, you can turn this simple text editor into a lightweight IDE. It’s fast, has good syntax highlighting, and can have code completion, shell integration, and many similar feature you might expect from an IDE. One feature it lacked was trimming whitespace from files. I searched for plugins to do this, and found several, but none of them quite met my expectations, because none were configurable. I typically want my files to end with one and only one newline. Of course, the solution is Perl.

Understanding load averages

Sat, 29 Jan 2011 16:50:00 -0400

Load averages are at once hugely simple and hideously complex. Understading what these numbers mean is important for correctly applying this simple indicator of system health.

First, a load average is not CPU percentage. That is simply a snapshot of how often a process was found being executed on the CPU. The load average differs in that it includes all demand for CPU, not just what is currently running.

If Linux were popular

Mon, 20 Dec 2010 16:48:00 -0400

This essay was originally written in Portuguese, and subsequently translated to English. I rewrote the English version so it reads like an anglophone wrote it.

I understand that some people have problems switching from Windows to linux. Since all my friends keep telling me how great Windows is, I thought I’d try an experiment.