Heartbleed is a critical vulnerability in OpenSSL revealed yesterday. I’m not sure it could be more serious: it allows an attacker to connect to your server and use the TLS heartbeat extension to obtain 64k of server memory (and do it again to get another 64k and again and…) – while leaving no traces in logs. That server memory might include primary key material (private keys), secondary key material (usernames and passwords), and collateral (memory addresses, canaries used to detect overflow, etc)
The researchers who discovered Heartbleed did a great job of outlining the severity of the vulnerability, as well as how to recover from it. For server operators, this is actually fairly straightforward: patch the OpenSSL vulnerability, install new keys, revoke the old ones, and be sure to restart every service that uses OpenSSL (or just reboot the whole server if in doubt). Unless your server was configured to use PFS, this still leaves all past traffic vulnerable to decryption, but even a leaked ticket key would compromise all the sessions it signed.
While these recovery steps are straightforward, they’re not easy to carry out for large organizations. I have just one server, and no overhead. I was able to recover in about an hour and a half, and most of that was waiting for the CA’s verification email process.
Secondary recovery
But what about recovering from potential compromise of secondary key material, and collateral?
Collateral, such as memory addresses, might be useful to an attacker, but are only useful for a short time. Upgrading to a patched version of OpenSSL is sufficient to make this data useless.
Secondary key material is of particular concern. Heartbleed made it possible for an attacker to obtain passwords from server memory, and/or obtain your private key, which they could then use to decrypt any traffic you thought was secure by virtue of using TLS. That includes, for example, passwords on the wire. So, as a website operator, you might want to assume that passwords have been compromised. Hopefully you have a way to force everyone to pick new passwords that you can deploy after you’ve completed your recovery procedures.
But as a user, how can you handle this proactively? Paranoid users might want to change all their passwords; less paranoid users might want to change a handful of high-value passwords. Regardless, how do you decide when to do that? There are probably many services that won’t be asking you to change your password – maybe because they didn’t think of it, or maybe because they decided on your behalf that that’s not needed. If you want to do it anyways, how do you know when to change your password? You want to make sure it is after they’ve completed their upgrades, but as soon as possible.
Do you change passwords now, which is as soon as possible, but might be while the server is still vulnerable? In this case, your password might be compromised either now or later by exploitation of heartbleed.
Change passwords every 90 days, or if compromise is suspected, except when changing password may result in compromise #heartbleed
— Adrian Sanabria (@sawaba@infosec.exchange) (@sawaba) April 8, 2014
Or do you wait, and do it later? I think this makes more sense. This risks leaving a compromised password in place, but gives the server operator a chance to patch their systems so that when you finally do change the password, it won’t be compromised (by heartbleed, at least). This isn’t a trivial matter: major websites like Yahoo! were vulnerable, and recovering from heartbleed on such large sites will be very complex, and thus time-consuming. You absolutely do not want to assume that big web companies are fully recovered yet – check out these scan results from a research team at the University of Michigan.
Detecting patched servers
So, how do you know when they’re done patching? Ideally, they’ll let you know. If you’re curious now, you can test their services yourself. I’ve seen several tools making the rounds:
- Qualys' SSL Labs now has a test for heartbleed that shouldn’t expose any sensitive data
- Adrian Hayter’s updated script
[Update: Predictably, there are bugs in the heartbleed detection scripts. I’ve updated the provided list accordingly.]
Please consider carefully whether you should be using these tools. Cybercrime legislation in both the US and Canada is pretty bad. You could land yourself in trouble by using these tools. Understanding your risk is the first step in determining if the risk is acceptable.
If you do use one, keep in mind that this only tells you whether the site in question has upgraded OpenSSL – not whether they have replaced private keys, which is a critical step in recovery.
You can check the “valid from” date on the certificate, which might indicate they’ve done something, but it is possible to get a new certificate without changing the private key, so even this isn’t sufficient. You’ll really just have to wait for the website to let you know when they’ve replaced private keys, and then take their word for it.
Remember: you should not change your passwords until the website has completed their recovery. Ideally, they will let you know when that happens.