Jacques Mattheij

Technology, Coding and Business

Letsencrypt, the Good, The bad and the Ugly

Letsencrypt is a pretty neat concept: free secure certificates for web servers, in order to increase the adoption of HTTPS across the web. The basic idea is that certificates should be free, that the barrier to install them should be as low as possible and that updating certificates should be automated. It protects this site and many 100’s of millions besides. The advantages are that in-flight data can no longer be easily snooped and that injection of data into pages is made either much harder or even impossible. From a security point of view it is a huge step forward.

The good

The project launched in April 2016 and has been a resounding success, meanwhile over 1 billion certificates have been issued. As with all such projects, after the initial launch the software was regularly updated in order to track the changing of the requirements but in essence the problem is a simple one: verify that the certificate request is valid by having the domain respond to a challenge and if it is to fetch a certificate from the letsencrypt.org servers and install it. So far so good.

For many institutions the ‘free’ part of the proposition was the secondary part. The main part - for instance for me - was that letsencrypt automated something that I would otherwise have to do once per year for the domains that are active: create and install a new certificate. A bit of a hassle, about 20 minutes per domain.

And this is where things are not looking all that good for Letsencrypt. Because that 20 minutes was more or less a fixed amount I knew that I was giving up a modicum of control for a small convenince, a rasonable trade-off.

The bad

The first time this bit me was when a while ago one of my servers suddenly refused to update its certificate. Somewhere someone has decided that Letsencrypt should auto-update even if absolutely nothing had been changed on the system that it was running on. And this automatic update completely and utterly destroyed the runtime environment on that computer, leading to its eventual re-imaging, something that should have never ever happened. I wrote the other day about why I absolutely loathe upgrading software, it’s like playing Russian roulette with 5 bullets and what you hope is a blank in a six shot revolver.

Unfortunately Letsencrypt is no exception to this: the upgrade process is bad enough, it tends to install a whole pile of cruft, whole python virtual environments are dragged in upon cert renewal, which is odd because after all *the situation hasn’t changed and if it worked last time it should still work this time*.

Then the challenge protocol was changed.

The ugly

I’m working on a new project. That new project requires a webserver on a new domain. My name server is running ‘mailinabox’ and has been happily humming away for the last two years. A new protocol for the letsencrypt challenges was released, announced in some obscure corner of the web and bit by bit the old method of issuing new certificates was deprecated. You would not have known this unless you interacted with Letsencrypt on a daily basis, but like most of these things their whole value lies in NOT having to interact with them all the time. They should just shut up, sit in a corner and do their bloody job.

Not so Letsencrypt. For the second time since I started using it Letsencrypt demanded to be center stage. Upon requesting a certificate for my brand-spanking-new domain I was greeted by some ridiculously obscure error message which then led me down a rabbit hole of endless websites and people - unsuccessfully - trying to solve the same kind of problem.

Eventually I managed to get the certificate issued, but I already know that I will have to do at least a day if not more of hard core system administration work in order to properly rectify the situaton. The machine that this is all running on will need a much more recent version of its OS. That means I’m going to have to migrate a whole bunch of data to another machine in order to be able to do this without downtime. Then I will have to temporarily re-route the DNS to make sure that other services not directly related to this project continue to function.

All of this is error prone and time consuming.

The effect of this is that now Letsencrypt, from a time saver and a convenience has turned itself - for the second time - into a very large net-negative. This is unacceptable for a project that tries to make something easier and where convenience, not financial considerations are a big driver in adoption.

I understand that Letsencrypt needs to stay with the times. But the Letsencrypt folks should also understand that in the greater scheme of things certificates are not the most important thing in life and that causing your end users this level of grief is in the longer term not in the projects’ best interest.