Diagnosing performance degradation under adverse circumstances

[This post is a few years old and was never published. Recently, I was reminded about memcached slab imbalance, which in turn reminded me of this post.]

At work, we encountered a sudden and precipitous performance regression on one particular page of a legacy application. It’s a Perl web application, running under mod_perl, using ModPerl::RegistryLoader to compile scripts at server startup, and Apache::DBI to provide persistent database connections.

Our users suddenly began complaining about one particular page being “three times slower than normal.” Later examination of the Apache logs showed a 20x(!!) slowdown.

Investigating this performance problem was interesting because we didn’t have good access to required data, and our technology choices slowed us down or completely prevented us from collecting it. Although we solved the mystery, the experience had several important lessons.

Validating SSL certificates for IRC bouncers

IRC bouncers are sort of like a proxy. Your bouncer stays online, connected to IRC, all the time, and then you connect to the bouncer using a normal IRC client. I connect to my bouncer with an SSL-encrypted connection, but I hadn’t been validating the certificate until now. Validating the SSL certificate is critical for thwarting man-in-the-middle (MITM) attacks.

In a MITM attack, the victim connects to the attacker, thinking it is the service they want to talk to (the IRC bouncer in this case). The attacker then forwards the connection to the service. Both connections might use SSL, but in the middle, the attacker can see the plaintext. They can simply eavesdrop, or modify the data flowing in both directions. SSL is supposed to prevent that, but if you don’t validate the certificate, then you don’t know who you’re talking to. I want to know I’m really talking to my IRC bouncer, so let’s figure out how to validate that certificate.

Adding LUKS hard disk encryption on LVM after the fact

I have an external hard disk enclosure with two disks. I used Logical Volume Manager to create a single logical volume that spanned them, and slowly filled it to about 60% capacity. Lately, I’ve been trying to be more conscious of using encryption, and this was one area where I hadn’t done so. At the time I felt like learning how to do LVM was enough, and LUKS could wait until later.

Introducing mvr: like mv, but clever

I wanted to move a large number of files from one directory to another, but the target directory already had many of the filenames already used. This is a common enough problem – digital cameras use DSC#, video downloaders often append numbers to get a unique filename, and so on. In both those examples, the sequence restarts if you empty the program’s work directory. So, you’ll end up with DSC0001.jpg every time you empty your camera’s memory card. If you’re trying to move such files into a single directory, you’ll get conflicts every time.

Instead of manually renaming the files before transferring them, I wrote a simple script to give each file a unique name in the destination directory.

Book review: 'Coding Freedom' by Gabriella Coleman

I’ve just finished reading Gabriella Coleman’s new book “Coding Freedom: The ethics and aesthetics of hacking” (2013, Princeton University Press) which culminates over a decade of field research, in-depth interviews, observation, and participation in the hacker scene globally. In this case, “hacker” refers to the free/open-source software (FOSS) hacker, and in particular the Debian project.

Hacking is where craft and craftiness converge.

Automating server build-out with Module::Build

At Pythian, we have one application that is composed of several components, the deployment of which needs to conform to our slightly peculiar server setup. Until recently, this required manually deploying each component. I did this a couple weeks ago, and it took me something like 40 hours to figure out and complete. As I went, I started reading up on Module::Build, trying to figure out how to automate as much as possible. It turns out that this core module gives us a surprisingly powerful tool for customized deployment. First, it will help to understand a few aspects of how our code is deployed.

Introducing File::Symlink::Atomic

In Tips & tricks from my 4 months at Pythian, I showed how to give a symlink a new target atomically. I wasn’t aware of any module to encapsulate that, so I quickly put together File::Symlink::Atomic. This module is useful because it eliminates the need to know how to do this safely - simply use File::Symlink::Atomic and you get a drop-in replacement for CORE::symlink. It creates a temporary symlink (using File::Temp to get a unique pathname) pointing to your new target, then moves it into place with a rename call.

Wherein I realize the bliss of writing init scripts with Daemon::Control

Init scripts are annoying little things – almost entirely boilerplate. Here’s how I learned to stop struggling, and love Daemon::Control to control my daemons.

The module really is as simple as the synopsis – you describe the daemon, have it write an init script (which actually just runs your Daemon::Control script) for you, then update-rc.d and you’re golden. It really is that simple.

Trimming whitespace in gedit with Perl

With gedit plugins, you can turn this simple text editor into a lightweight IDE. It’s fast, has good syntax highlighting, and can have code completion, shell integration, and many similar feature you might expect from an IDE. One feature it lacked was trimming whitespace from files. I searched for plugins to do this, and found several, but none of them quite met my expectations, because none were configurable. I typically want my files to end with one and only one newline. Of course, the solution is Perl.

Understanding load averages

Load averages are at once hugely simple and hideously complex. Understading what these numbers mean is important for correctly applying this simple indicator of system health. First, a load average is not CPU percentage. That is simply a snapshot of how often a process was found being executed on the CPU. The load average differs in that it includes all demand for CPU, not just what is currently running. A useful analogy A four-processor machine can be visualized as a four-lane freeway.

If Linux were popular

This essay was originally written in Portuguese, and subsequently translated to English. I rewrote the English version so it reads like an anglophone wrote it.

I understand that some people have problems switching from Windows to linux. Since all my friends keep telling me how great Windows is, I thought I’d try an experiment.