News for the ‘Technology’ Category

Book review: Site Reliability Engineering

Google has been working for about 2 years on a book about Site Reliability Engineering, the discipline and organization that keeps Google's large-scale systems running smoothly. "Site Reliability Engineering" was finally published last week. It spans some 500 pages, and offers a rare inside glimpse into how Google actually works. The authors are remarkably open, naming technologies and projects, and explaining how systems work. There may not be source code, but there's lots here that can be implemented outside Google. That makes it a great read for startups expecting to scale and small-to-medium tech companies that want to up their reliability game.


Diagnosing performance degradation under adverse circumstances

[This post is a few years old and was never published. Recently, I was reminded about memcached slab imbalance, which in turn reminded me of this post.]

At work, we encountered a sudden and precipitous performance regression on one particular page of a legacy application. It's a Perl web application, running under mod_perl, using ModPerl::RegistryLoader to compile scripts at server startup, and Apache::DBI to provide persistent database connections.

Our users suddenly began complaining about one particular page being "three times slower than normal." Later examination of the Apache logs showed a 20x(!!) slowdown.

Investigating this performance problem was interesting because we didn't have good access to required data, and our technology choices slowed us down or completely prevented us from collecting it. Although we solved the mystery, the experience had several important lessons.

CSRF vulnerability at provides low-cost virtual machines hosted in Canada. is the management interface, where customers can install OS images, access a console, etc.A system security breach indicator light. Photo by Jeff Keyzer (CC-BY-SA)

A cross-site request forgery vulnerability was discovered in this web application. If a customer could be tricked into visiting a crafted URL while logged in, an attacker could change the victim's password, gaining access to the management interface.

In turn, this grants root access on all the victim's VMs, the ability to wipe or reinstall VMs, and potentially allows the attacker to spend the victim's money on CloudAtCost products and services.

Changing the password does not end the session or email the user, so the victim will not immediately notice anything is wrong. Exploitation of CSRF is difficult to detect on the server, so CloudAtCost is unlikely to notice anything is wrong either.

There is no evidence the vulnerability is being exploited, but exploitation is trivial and the impact of exploitation is severe. Exploitation is simple: build a URL of the following form:$uri_encoded_password

Any method which gets the victim to load the crafted URL in their browser while logged in will cause their password to be changed, and the attacker can simply log in. Phishing emails are a common exploitation vector. A watering hole attack could also work: create a website that cloudatcost users would visit, or use one which already exists (such as and embed the crafted URL as an img resource, for example, and the attacker would achieve a similar effect. Other exploitation methods are certainly available, and practicable.


  • September 25, 2014: vulnerability discovered, and disclosed to CloudAtCost through a customer support ticket. Feedback is passed along, and the customer support ticket is closed.
  • September 25: vulnerability report is escalated via email. No reply.
  • October 2: vulnerability report is escalated via email, and a single point of contact at CloudAtCost is provided. Details are provided to that contact directly. No reply.
  • October 9: Direct contact is made with CloudAtCost and the Canadian Cyber Incident Response Centre (CCIRC), and full details are repeated. Follow-up is promised, but doesn't happen.
  • October 14: CCIRC reports that the vulnerability has been fixed. Testing shows that this is not the case. Clarification is requested from CloudAtCost. Deployment of the patch is scheduled for Nov 1.
  • November 3: The self-imposed date for deploying the patch (November 1) passes, and the application is still vulnerable. Clarification is requested from CloudAtCost. None is provided.
  • November 7: Information on the progress of deploying the patched application is requested. None is provided.
  • November 14: I spoke with a CloudAtCost representative on the phone, who said their team was having trouble getting a release out, and needed more time.
  • November 26: Information on the progress on deploying a fixed version of the web application was requested. No reply.
  • December 10: A hard deadline for public disclosure was set, and sent to CloudAtCost. The web application had been deployed in the past 3-4 days.
  • December 11: This disclosure is published.

Legal issues in computer security research

This Thursday, I gave a talk at AtlSecCon 2014. The weather threw a wrench in the organizers' plans, but they managed to pull off a solid conference. Unfortunately, the talks weren't recorded this year. The slides are posted on speakerdeck, and are embedded below the fold.

I also reprised this talk at NSLUG, and recorded audio, now posted on SoundCloud, and also embedded below the fold.

Finally: late last year, I wrote 3 posts exploring Canada's computer crime laws (1, 2, 3) which were initial versions of work that eventually became two papers I submitted this semester for a directed studies course. If you were interested in those posts, I've embedded the final PDFs below. The talk is a condensed version of that work.


Recovering from heartbleed

Heartbleed is a critical vulnerability in OpenSSL revealed yesterday. I'm not sure it could be more serious: it allows an attacker to connect to your server and use the TLS heartbeat extension to obtain 64k of server memory (and do it again to get another 64k and again and...) -- while leaving no traces in logs. That server memory might include primary key material (private keys), secondary key material (usernames and passwords), and collateral (memory addresses, canaries used to detect overflow, etc)

Upgrading encrypted Android devices

If you encrypt your Android device, the standard over-the-air (OTA) upgrades don't work, because /sdcard can't be mounted in recovery. (more…)

Exploring Canada's computer crime laws: Part 3

Since the exceptions in copyright law for encryption and security research don't apply if you're doing anything criminal, I next looked at the Criminal Code [PDF].

Exploring Canada's computer crime laws: Part 2

Since the exceptions in copyright law for encryption and security research don't apply if you're doing anything criminal, I next looked at the Criminal Code [PDF].

Exploring Canada's computer crime laws: Part 1

As someone with an interest in technology, security, and the legal issues surrounding them, I often watch relevant legal cases with interest. Typically, those cases come from the United States. The CFAA has been in the news frequently of late, and not always in a good light. I was pleased to see Zoe Lofgren's proposed changes, which try to make the law less draconian.

This is typical for Canada -- we often see more about American news on topics like this than Canadian. I realized that I really didn't know what the law in Canada said about so-called computer crimes, although I've often wondered. A while back, I took an afternoon to do some reading. I was not happy when that afternoon ended. This is part one of a three-part series on what I found.

Validating SSL certificates for IRC bouncers

IRC bouncers are sort of like a proxy. Your bouncer stays online, connected to IRC, all the time, and then you connect to the bouncer using a normal IRC client. I connect to my bouncer with an SSL-encrypted connection, but I hadn't been validating the certificate until now. Validating the SSL certificate is critical for thwarting man-in-the-middle (MITM) attacks.

In a MITM attack, the victim connects to the attacker, thinking it is the service they want to talk to (the IRC bouncer in this case). The attacker then forwards the connection to the service. Both connections might use SSL, but in the middle, the attacker can see the plaintext. They can simply eavesdrop, or modify the data flowing in both directions. SSL is supposed to prevent that, but if you don't validate the certificate, then you don't know who you're talking to. I want to know I'm really talking to my IRC bouncer, so let's figure out how to validate that certificate.

Introducing Hack::Natas

Last Monday, I presented my solutions to the Natas server-side security war games at NSLUG. Afterwards, I spent some time to clean up my code, and I've now published it to CPAN as Hack::Natas, which comes with modules and scripts to solve level 15 and 16 in an automated way, plus walkthroughs for all the levels up to 17 written in Markdown (those are almost the same as my blog posts, so you're not missing out by looking at only one or the other).

Presenting my Natas solutions at NSLUG

Last Monday, I presented my solutions to the Natas server-side security war games at my local linux users' group.

I recorded my talk, but it didn't turn out well. I was using Google Hangouts for the first time, and I forgot that it only records the windows you tell it to. In the video, embedded below the fold, there's a lot of talking about windows that were being projected, but which didn't get recorded. Still, the audio counts for something, and you can see what I'm talking about much of the time.

SSL configuration on nginx

This SSL configuration for nginx achieves an A on the SSL labs tool. It's what this server currently uses.

Server-side security war games: Part 16

This is the last level. We're challenged with an improved version of level 9 -- they've added additional "sanitation" to keep us out.

    if(preg_match('/[;|&`\'"]/',$key)) {
        print "Input contains an illegal character!";
    } else {
        passthru("grep -i \"$key\" dictionary.txt");


Server-side security war games: Part 15

We're nearly at the end! This is the 2nd-last level.

We know there is a users table, with columns "username" and "password". This time, the code just checks that the username exists. There's no way to print out the data we want. Instead, we'll have to do something cleverer.

Server-side security war games: Part 14

In level 14, we see a more traditional username & password form. Let's check the source code to see if there are holes we can slip through.

Server-side security war games: Part 13

This is level 13. Looks like they claim to only accept image files, in order to close the flaw we used previously. I bet we can get around that restriction just like we did when they disallowed certain characters in the search term. Let's examine the code.

Here's the new part of the code:

    if (! exif_imagetype($_FILES['uploadedfile']['tmp_name'])) {
        echo "File is not an image";


Server-side security war games: Part 12

In level 12, we're given a file upload form. Let's take a look at the code that processes input.

Server-side security war games: Part 11

This is level 11. Your clue is that "XOR encryption" is not encryption. Let's look in the cookies to find out they have XOR-ed, so we can mess with it.

Server-side security war games: Part 10

Welcome to level 10. "For security reasons, we now filter on certian characters" -- okay they've gotten wise to our little game. But let's check how good their countermeasures are.

Well, they don't allow us to use the semicolon or ampersand any longer. Well that's not a problem, I know other ways to manhandle that command into doing what I want.