SitesLinksOttawaLifePhotosTravelToolsJournalBlog
See More Stuff
Saturday, November 26, 2016

LetsEncrypt in AWS

Update: see here for LetsEncrypt on Amazon Linux 2

Previously I was using StartSSL in AWS, but consensus seems to have moved to LetsEncrypt for free SSL.

I used Larry Land as a guide. Note that CertBot doesn't come with instructions for Lighttpd, but you basically just automate the manual process performed with StartSSL. Also note that full features of Certbot aren't supported on Amazon Linux.

[ec2-user]$ ./certbot-auto
FATAL: Amazon Linux support is very experimental at present...
if you would like to work on improving it, please ensure you have backups
and then run this script again with the --debug flag!

Following this advice (coderwall, superuser), I decided to use it in a simpler mode.

cd /etc/lighttpd/ssl
sudo wget https://dl.eff.org/certbot-auto
sudo chmod a+x certbot-auto

Here's what we're going to do:

Since certbot isn't fully supported on Amazon Linux, we have to use the certonly command to obtain the certificate. This will allow us to interactively select the plugin and options used to obtain the certificate.

We have to use the --debug command or it will refuse to run.

We'll avoid some of the interactive part by using the --webroot command to specify the "webroot" plugin. This works with the webroot directory of any webserver software.

sudo /etc/lighttpd/ssl/certbot-auto 
 certonly 
 --debug
 --webroot 
 -w /var/www/lighttpd/holtstrom -d holtstrom.com -d www.holtstrom.com 
 -w /var/www/lighttpd/evilgoblin -d evilgoblin.com -d www.evilgoblin.com
 -w /var/www/lighttpd/frothing -d frothing.com -d www.frothing.com
 -w /var/www/lighttpd/tribalworker -d tribalworker.com -d www.tribalworker.com

This command will put /.well-known/acme-challenge in each -w path, (i.e. /var/www/lighttpd/holtstrom/.well-known/acme-challenge) to prove domain ownership. You webserver must be willing to serve this hidden files.

The .well-known folder is persistent, but its contents are not.

Here is the command in one line:

sudo /etc/lighttpd/ssl/certbot-auto certonly --debug --webroot -w /var/www/lighttpd/holtstrom -d holtstrom.com -d www.holtstrom.com -w /var/www/lighttpd/evilgoblin -d evilgoblin.com -d www.evilgoblin.com -w /var/www/lighttpd/frothing -d frothing.com -d www.frothing.com -w /var/www/lighttpd/tribalworker -d tribalworker.com -d www.tribalworker.com

Here is the output:

Bootstrapping dependencies via Amazon Linux...
yum is /usr/bin/yum
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main/latest                                                                                                                    | 2.1 kB     00:00
amzn-updates/latest                                                                                                                 | 2.3 kB     00:00
Package 1:openssl-1.0.1k-15.96.amzn1.x86_64 already installed and latest version
Package ca-certificates-2015.2.6-65.0.1.16.amzn1.noarch already installed and latest version
Package python27-2.7.12-2.120.amzn1.x86_64 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package augeas-libs.x86_64 0:1.0.0-5.7.amzn1 will be installed
---> Package dialog.x86_64 0:1.1-9.20080819.1.5.amzn1 will be installed
---> Package gcc.noarch 0:4.8.3-3.20.amzn1 will be installed
--> Processing Dependency: gcc48 = 4.8.3 for package: gcc-4.8.3-3.20.amzn1.noarch
---> Package libffi-devel.x86_64 0:3.0.13-16.5.amzn1 will be installed
---> Package openssl-devel.x86_64 1:1.0.1k-15.96.amzn1 will be installed
--> Processing Dependency: zlib-devel(x86-64) for package: 1:openssl-devel-1.0.1k-15.96.amzn1.x86_64
--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.1k-15.96.amzn1.x86_64
---> Package python27-devel.x86_64 0:2.7.12-2.120.amzn1 will be installed
---> Package python27-pip.noarch 0:6.1.1-1.23.amzn1 will be installed
---> Package python27-tools.x86_64 0:2.7.12-2.120.amzn1 will be installed
---> Package python27-virtualenv.noarch 0:12.0.7-1.13.amzn1 will be installed
---> Package system-rpm-config.noarch 0:9.0.3-42.27.amzn1 will be installed
--> Running transaction check
---> Package gcc48.x86_64 0:4.8.3-9.111.amzn1 will be installed
--> Processing Dependency: cpp48(x86-64) = 4.8.3-9.111.amzn1 for package: gcc48-4.8.3-9.111.amzn1.x86_64
--> Processing Dependency: libgomp(x86-64) >= 4.8.3-9.111.amzn1 for package: gcc48-4.8.3-9.111.amzn1.x86_64
--> Processing Dependency: glibc-devel(x86-64) >= 2.2.90-12 for package: gcc48-4.8.3-9.111.amzn1.x86_64
--> Processing Dependency: libmpc.so.3()(64bit) for package: gcc48-4.8.3-9.111.amzn1.x86_64
--> Processing Dependency: libgomp.so.1()(64bit) for package: gcc48-4.8.3-9.111.amzn1.x86_64
--> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc48-4.8.3-9.111.amzn1.x86_64
---> Package krb5-devel.x86_64 0:1.13.2-12.40.amzn1 will be installed
--> Processing Dependency: libverto-devel for package: krb5-devel-1.13.2-12.40.amzn1.x86_64
--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.13.2-12.40.amzn1.x86_64
--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.13.2-12.40.amzn1.x86_64
--> Processing Dependency: libselinux-devel for package: krb5-devel-1.13.2-12.40.amzn1.x86_64
---> Package zlib-devel.x86_64 0:1.2.8-7.18.amzn1 will be installed
--> Running transaction check
---> Package cpp48.x86_64 0:4.8.3-9.111.amzn1 will be installed
---> Package glibc-devel.x86_64 0:2.17-106.168.amzn1 will be installed
--> Processing Dependency: glibc-headers = 2.17-106.168.amzn1 for package: glibc-devel-2.17-106.168.amzn1.x86_64
--> Processing Dependency: glibc-headers for package: glibc-devel-2.17-106.168.amzn1.x86_64
---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.12.amzn1 will be installed
---> Package libcom_err-devel.x86_64 0:1.42.12-4.40.amzn1 will be installed
---> Package libgomp.x86_64 0:4.8.3-9.111.amzn1 will be installed
---> Package libmpc.x86_64 0:1.0.1-3.3.amzn1 will be installed
---> Package libselinux-devel.x86_64 0:2.1.10-3.22.amzn1 will be installed
--> Processing Dependency: libsepol-devel >= 2.1.5-1 for package: libselinux-devel-2.1.10-3.22.amzn1.x86_64
--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.1.10-3.22.amzn1.x86_64
---> Package libverto-devel.x86_64 0:0.2.5-4.9.amzn1 will be installed
---> Package mpfr.x86_64 0:3.1.1-4.14.amzn1 will be installed
--> Running transaction check
---> Package glibc-headers.x86_64 0:2.17-106.168.amzn1 will be installed
--> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.17-106.168.amzn1.x86_64
--> Processing Dependency: kernel-headers for package: glibc-headers-2.17-106.168.amzn1.x86_64
---> Package libsepol-devel.x86_64 0:2.1.7-3.12.amzn1 will be installed
--> Running transaction check
---> Package kernel-headers.x86_64 0:4.4.30-32.54.amzn1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===========================================================================================================================================================
 Package                                 Arch                       Version                                         Repository                        Size
===========================================================================================================================================================
Installing:
 augeas-libs                             x86_64                     1.0.0-5.7.amzn1                                 amzn-main                        345 k
 dialog                                  x86_64                     1.1-9.20080819.1.5.amzn1                        amzn-main                        205 k
 gcc                                     noarch                     4.8.3-3.20.amzn1                                amzn-main                        3.9 k
 libffi-devel                            x86_64                     3.0.13-16.5.amzn1                               amzn-main                         23 k
 openssl-devel                           x86_64                     1:1.0.1k-15.96.amzn1                            amzn-updates                     1.5 M
 python27-devel                          x86_64                     2.7.12-2.120.amzn1                              amzn-main                        524 k
 python27-pip                            noarch                     6.1.1-1.23.amzn1                                amzn-main                        1.9 M
 python27-tools                          x86_64                     2.7.12-2.120.amzn1                              amzn-main                        709 k
 python27-virtualenv                     noarch                     12.0.7-1.13.amzn1                               amzn-main                        2.0 M
 system-rpm-config                       noarch                     9.0.3-42.27.amzn1                               amzn-main                         63 k
Installing for dependencies:
 cpp48                                   x86_64                     4.8.3-9.111.amzn1                               amzn-main                        6.7 M
 gcc48                                   x86_64                     4.8.3-9.111.amzn1                               amzn-main                         17 M
 glibc-devel                             x86_64                     2.17-106.168.amzn1                              amzn-main                        1.1 M
 glibc-headers                           x86_64                     2.17-106.168.amzn1                              amzn-main                        737 k
 kernel-headers                          x86_64                     4.4.30-32.54.amzn1                              amzn-updates                     1.0 M
 keyutils-libs-devel                     x86_64                     1.5.8-3.12.amzn1                                amzn-main                         37 k
 krb5-devel                              x86_64                     1.13.2-12.40.amzn1                              amzn-main                        700 k
 libcom_err-devel                        x86_64                     1.42.12-4.40.amzn1                              amzn-main                         35 k
 libgomp                                 x86_64                     4.8.3-9.111.amzn1                               amzn-main                        175 k
 libmpc                                  x86_64                     1.0.1-3.3.amzn1                                 amzn-main                         53 k
 libselinux-devel                        x86_64                     2.1.10-3.22.amzn1                               amzn-main                        157 k
 libsepol-devel                          x86_64                     2.1.7-3.12.amzn1                                amzn-main                         70 k
 libverto-devel                          x86_64                     0.2.5-4.9.amzn1                                 amzn-main                         11 k
 mpfr                                    x86_64                     3.1.1-4.14.amzn1                                amzn-main                        237 k
 zlib-devel                              x86_64                     1.2.8-7.18.amzn1                                amzn-main                         53 k

Transaction Summary
===========================================================================================================================================================
Install  10 Packages (+15 Dependent packages)

Total download size: 36 M
Installed size: 69 M
Is this ok [y/d/N]: y
Downloading packages:
(1/25): augeas-libs-1.0.0-5.7.amzn1.x86_64.rpm                                                                                      | 345 kB     00:00
(2/25): cpp48-4.8.3-9.111.amzn1.x86_64.rpm                                                                                          | 6.7 MB     00:00
(3/25): dialog-1.1-9.20080819.1.5.amzn1.x86_64.rpm                                                                                  | 205 kB     00:00
(4/25): gcc-4.8.3-3.20.amzn1.noarch.rpm                                                                                             | 3.9 kB     00:00
(5/25): gcc48-4.8.3-9.111.amzn1.x86_64.rpm                                                                                          |  17 MB     00:00
(6/25): glibc-devel-2.17-106.168.amzn1.x86_64.rpm                                                                                   | 1.1 MB     00:00
(7/25): glibc-headers-2.17-106.168.amzn1.x86_64.rpm                                                                                 | 737 kB     00:00
(8/25): kernel-headers-4.4.30-32.54.amzn1.x86_64.rpm                                                                                | 1.0 MB     00:00
(9/25): keyutils-libs-devel-1.5.8-3.12.amzn1.x86_64.rpm                                                                             |  37 kB     00:00
(10/25): krb5-devel-1.13.2-12.40.amzn1.x86_64.rpm                                                                                   | 700 kB     00:00
(11/25): libcom_err-devel-1.42.12-4.40.amzn1.x86_64.rpm                                                                             |  35 kB     00:00
(12/25): libffi-devel-3.0.13-16.5.amzn1.x86_64.rpm                                                                                  |  23 kB     00:00
(13/25): libgomp-4.8.3-9.111.amzn1.x86_64.rpm                                                                                       | 175 kB     00:00
(14/25): libmpc-1.0.1-3.3.amzn1.x86_64.rpm                                                                                          |  53 kB     00:00
(15/25): libselinux-devel-2.1.10-3.22.amzn1.x86_64.rpm                                                                              | 157 kB     00:00
(16/25): libsepol-devel-2.1.7-3.12.amzn1.x86_64.rpm                                                                                 |  70 kB     00:00
(17/25): libverto-devel-0.2.5-4.9.amzn1.x86_64.rpm                                                                                  |  11 kB     00:00
(18/25): mpfr-3.1.1-4.14.amzn1.x86_64.rpm                                                                                           | 237 kB     00:00
(19/25): openssl-devel-1.0.1k-15.96.amzn1.x86_64.rpm                                                                                | 1.5 MB     00:00
(20/25): python27-devel-2.7.12-2.120.amzn1.x86_64.rpm                                                                               | 524 kB     00:00
(21/25): python27-pip-6.1.1-1.23.amzn1.noarch.rpm                                                                                   | 1.9 MB     00:00
(22/25): python27-tools-2.7.12-2.120.amzn1.x86_64.rpm                                                                               | 709 kB     00:00
(23/25): python27-virtualenv-12.0.7-1.13.amzn1.noarch.rpm                                                                           | 2.0 MB     00:00
(24/25): system-rpm-config-9.0.3-42.27.amzn1.noarch.rpm                                                                             |  63 kB     00:00
(25/25): zlib-devel-1.2.8-7.18.amzn1.x86_64.rpm                                                                                     |  53 kB     00:00
-----------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                       17 MB/s |  36 MB  00:00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : mpfr-3.1.1-4.14.amzn1.x86_64                                                                                                           1/25
  Installing : libmpc-1.0.1-3.3.amzn1.x86_64                                                                                                          2/25
  Installing : cpp48-4.8.3-9.111.amzn1.x86_64                                                                                                         3/25
  Installing : libcom_err-devel-1.42.12-4.40.amzn1.x86_64                                                                                             4/25
  Installing : libverto-devel-0.2.5-4.9.amzn1.x86_64                                                                                                  5/25
  Installing : libsepol-devel-2.1.7-3.12.amzn1.x86_64                                                                                                 6/25
  Installing : libselinux-devel-2.1.10-3.22.amzn1.x86_64                                                                                              7/25
  Installing : zlib-devel-1.2.8-7.18.amzn1.x86_64                                                                                                     8/25
  Installing : keyutils-libs-devel-1.5.8-3.12.amzn1.x86_64                                                                                            9/25
  Installing : krb5-devel-1.13.2-12.40.amzn1.x86_64                                                                                                  10/25
  Installing : python27-devel-2.7.12-2.120.amzn1.x86_64                                                                                              11/25
  Installing : libgomp-4.8.3-9.111.amzn1.x86_64                                                                                                      12/25
  Installing : kernel-headers-4.4.30-32.54.amzn1.x86_64                                                                                              13/25
  Installing : glibc-headers-2.17-106.168.amzn1.x86_64                                                                                               14/25
  Installing : glibc-devel-2.17-106.168.amzn1.x86_64                                                                                                 15/25
  Installing : gcc48-4.8.3-9.111.amzn1.x86_64                                                                                                        16/25
  Installing : python27-pip-6.1.1-1.23.amzn1.noarch                                                                                                  17/25
  Installing : python27-virtualenv-12.0.7-1.13.amzn1.noarch                                                                                          18/25
  Installing : gcc-4.8.3-3.20.amzn1.noarch                                                                                                           19/25
  Installing : 1:openssl-devel-1.0.1k-15.96.amzn1.x86_64                                                                                             20/25
  Installing : system-rpm-config-9.0.3-42.27.amzn1.noarch                                                                                            21/25
  Installing : libffi-devel-3.0.13-16.5.amzn1.x86_64                                                                                                 22/25
  Installing : dialog-1.1-9.20080819.1.5.amzn1.x86_64                                                                                                23/25
  Installing : python27-tools-2.7.12-2.120.amzn1.x86_64                                                                                              24/25
  Installing : augeas-libs-1.0.0-5.7.amzn1.x86_64                                                                                                    25/25
  Verifying  : python27-pip-6.1.1-1.23.amzn1.noarch                                                                                                   1/25
  Verifying  : augeas-libs-1.0.0-5.7.amzn1.x86_64                                                                                                     2/25
  Verifying  : kernel-headers-4.4.30-32.54.amzn1.x86_64                                                                                               3/25
  Verifying  : python27-tools-2.7.12-2.120.amzn1.x86_64                                                                                               4/25
  Verifying  : libgomp-4.8.3-9.111.amzn1.x86_64                                                                                                       5/25
  Verifying  : python27-virtualenv-12.0.7-1.13.amzn1.noarch                                                                                           6/25
  Verifying  : mpfr-3.1.1-4.14.amzn1.x86_64                                                                                                           7/25
  Verifying  : python27-devel-2.7.12-2.120.amzn1.x86_64                                                                                               8/25
  Verifying  : libselinux-devel-2.1.10-3.22.amzn1.x86_64                                                                                              9/25
  Verifying  : keyutils-libs-devel-1.5.8-3.12.amzn1.x86_64                                                                                           10/25
  Verifying  : zlib-devel-1.2.8-7.18.amzn1.x86_64                                                                                                    11/25
  Verifying  : dialog-1.1-9.20080819.1.5.amzn1.x86_64                                                                                                12/25
  Verifying  : libffi-devel-3.0.13-16.5.amzn1.x86_64                                                                                                 13/25
  Verifying  : glibc-headers-2.17-106.168.amzn1.x86_64                                                                                               14/25
  Verifying  : 1:openssl-devel-1.0.1k-15.96.amzn1.x86_64                                                                                             15/25
  Verifying  : gcc48-4.8.3-9.111.amzn1.x86_64                                                                                                        16/25
  Verifying  : cpp48-4.8.3-9.111.amzn1.x86_64                                                                                                        17/25
  Verifying  : libmpc-1.0.1-3.3.amzn1.x86_64                                                                                                         18/25
  Verifying  : system-rpm-config-9.0.3-42.27.amzn1.noarch                                                                                            19/25
  Verifying  : glibc-devel-2.17-106.168.amzn1.x86_64                                                                                                 20/25
  Verifying  : libsepol-devel-2.1.7-3.12.amzn1.x86_64                                                                                                21/25
  Verifying  : libverto-devel-0.2.5-4.9.amzn1.x86_64                                                                                                 22/25
  Verifying  : krb5-devel-1.13.2-12.40.amzn1.x86_64                                                                                                  23/25
  Verifying  : libcom_err-devel-1.42.12-4.40.amzn1.x86_64                                                                                            24/25
  Verifying  : gcc-4.8.3-3.20.amzn1.noarch                                                                                                           25/25

Installed:
  augeas-libs.x86_64 0:1.0.0-5.7.amzn1               dialog.x86_64 0:1.1-9.20080819.1.5.amzn1         gcc.noarch 0:4.8.3-3.20.amzn1
  libffi-devel.x86_64 0:3.0.13-16.5.amzn1            openssl-devel.x86_64 1:1.0.1k-15.96.amzn1        python27-devel.x86_64 0:2.7.12-2.120.amzn1
  python27-pip.noarch 0:6.1.1-1.23.amzn1             python27-tools.x86_64 0:2.7.12-2.120.amzn1       python27-virtualenv.noarch 0:12.0.7-1.13.amzn1
  system-rpm-config.noarch 0:9.0.3-42.27.amzn1

Dependency Installed:
  cpp48.x86_64 0:4.8.3-9.111.amzn1                 gcc48.x86_64 0:4.8.3-9.111.amzn1                    glibc-devel.x86_64 0:2.17-106.168.amzn1
  glibc-headers.x86_64 0:2.17-106.168.amzn1        kernel-headers.x86_64 0:4.4.30-32.54.amzn1          keyutils-libs-devel.x86_64 0:1.5.8-3.12.amzn1
  krb5-devel.x86_64 0:1.13.2-12.40.amzn1           libcom_err-devel.x86_64 0:1.42.12-4.40.amzn1        libgomp.x86_64 0:4.8.3-9.111.amzn1
  libmpc.x86_64 0:1.0.1-3.3.amzn1                  libselinux-devel.x86_64 0:2.1.10-3.22.amzn1         libsepol-devel.x86_64 0:2.1.7-3.12.amzn1
  libverto-devel.x86_64 0:0.2.5-4.9.amzn1          mpfr.x86_64 0:3.1.1-4.14.amzn1                      zlib-devel.x86_64 0:1.2.8-7.18.amzn1

Complete!
Creating virtual environment...
/etc/lighttpd/ssl/certbot-auto: line 583: virtualenv: command not found

Note: I don't think it was necessary to enable EPEL and have disabled it below, so you might want to skip this step.

GitHub suggests that we need to enable EPEL.

[ec2-user]$ sudo yum repolist
Loaded plugins: priorities, update-motd, upgrade-helper
repo id                                                                     repo name                                                                status
!amzn-main/latest                                                           amzn-main-Base                                                           5,612
!amzn-updates/latest                                                        amzn-updates-Base                                                          476
repolist: 6,088
[ec2-user]$ sudo yum-config-manager --enable epel
-- output not shown --
[ec2-user]$ sudo yum repolist
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main/latest                                                                                                                    | 2.1 kB     00:00
amzn-updates/latest                                                                                                                 | 2.3 kB     00:00
epel/x86_64/metalink                                                                                                                |  11 kB     00:00
epel/x86_64                                                                                                                         | 4.3 kB     00:00
epel/x86_64/group_gz                                                                                                                | 150 kB     00:00
epel/x86_64/updateinfo                                                                                                              | 729 kB     00:00
epel/x86_64/primary_db                                                                                                              | 5.8 MB     00:00
976 packages excluded due to repository priority protections
repo id                                                    repo name                                                                             status
amzn-main/latest                                           amzn-main-Base                                                                             5,612
amzn-updates/latest                                        amzn-updates-Base                                                                            476
epel/x86_64                                                Extra Packages for Enterprise Linux 6 - x86_64                                        11,151+976
repolist: 17,239

Let's try again:

Bootstrapping dependencies via Amazon Linux...
yum is /usr/bin/yum
Loaded plugins: priorities, update-motd, upgrade-helper
976 packages excluded due to repository priority protections
Package gcc-4.8.3-3.20.amzn1.noarch already installed and latest version
Package dialog-1.1-9.20080819.1.5.amzn1.x86_64 already installed and latest version
Package augeas-libs-1.0.0-5.7.amzn1.x86_64 already installed and latest version
Package 1:openssl-1.0.1k-15.96.amzn1.x86_64 already installed and latest version
Package 1:openssl-devel-1.0.1k-15.96.amzn1.x86_64 already installed and latest version
Package libffi-devel-3.0.13-16.5.amzn1.x86_64 already installed and latest version
Package system-rpm-config-9.0.3-42.27.amzn1.noarch already installed and latest version
Package ca-certificates-2015.2.6-65.0.1.16.amzn1.noarch already installed and latest version
Package python27-2.7.12-2.120.amzn1.x86_64 already installed and latest version
Package python27-devel-2.7.12-2.120.amzn1.x86_64 already installed and latest version
Package python27-virtualenv-12.0.7-1.13.amzn1.noarch already installed and latest version
Package python27-tools-2.7.12-2.120.amzn1.x86_64 already installed and latest version
Package python27-pip-6.1.1-1.23.amzn1.noarch already installed and latest version
Nothing to do
Creating virtual environment...
/etc/lighttpd/ssl/certbot-auto: line 583: virtualenv: command not found

Same failure. Let's try to find the virtualenv package.

[ec2-user]$ repoquery -l python27-virtualenv
/usr/bin/virtualenv
/usr/bin/virtualenv-2.7
/usr/lib/python2.7/dist-packages/virtualenv-12.0.7.egg-info
...
[ec2-user]$ sudo ls -la /usr/bin/v*
-rwxr-xr-x 1 root root  113440 Feb 25  2016 /usr/bin/vdir
-rwxr-xr-x 1 root root    9858 Aug 18 23:53 /usr/bin/verifytree
-rwxr-xr-x 1 root root 2441384 Jul  6 18:26 /usr/bin/vim
lrwxrwxrwx 1 root root       3 Jul 15 16:24 /usr/bin/vimdiff -> vim
-rwxr-xr-x 1 root root    2084 Jul  6 18:26 /usr/bin/vimtutor
-rwxr-xr-x 1 root root     322 Aug 10 21:57 /usr/bin/virtualenv-2.7
-rwxr-xr-x 1 root root   23352 Mar 17  2015 /usr/bin/vmstat

It looks like it's a simple matter of the non-versioned symlink not being in place. Let's add the symlink and try again.

cd /usr/bin
sudo ln -s /usr/bin/virtualenv-2.7 /usr/bin/virtualenv

This time it works. Since nothing new was installed, I don't think it was necessary to enable EPEL.

Bootstrapping dependencies via Amazon Linux...
yum is /usr/bin/yum
To use Certbot, packages from the EPEL repository need to be installed.
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main/latest                                                                                                                    | 2.1 kB     00:00
amzn-updates/latest                                                                                                                 | 2.3 kB     00:00
976 packages excluded due to repository priority protections
Package epel-release-6-8.9.amzn1.noarch already installed and latest version
Nothing to do
Loaded plugins: priorities, update-motd, upgrade-helper
976 packages excluded due to repository priority protections
Package gcc-4.8.3-3.20.amzn1.noarch already installed and latest version
Package dialog-1.1-9.20080819.1.5.amzn1.x86_64 already installed and latest version
Package augeas-libs-1.0.0-5.7.amzn1.x86_64 already installed and latest version
Package 1:openssl-1.0.1k-15.96.amzn1.x86_64 already installed and latest version
Package 1:openssl-devel-1.0.1k-15.96.amzn1.x86_64 already installed and latest version
Package libffi-devel-3.0.13-16.5.amzn1.x86_64 already installed and latest version
Package system-rpm-config-9.0.3-42.27.amzn1.noarch already installed and latest version
Package ca-certificates-2015.2.6-65.0.1.16.amzn1.noarch already installed and latest version
Package python27-2.7.12-2.120.amzn1.x86_64 already installed and latest version
Package python27-devel-2.7.12-2.120.amzn1.x86_64 already installed and latest version
Package python27-virtualenv-12.0.7-1.13.amzn1.noarch already installed and latest version
Package python27-tools-2.7.12-2.120.amzn1.x86_64 already installed and latest version
Package python27-pip-6.1.1-1.23.amzn1.noarch already installed and latest version
Nothing to do
Creating virtual environment...
Installing Python packages...
Installation succeeded.
Version: 1.1-20080819

It shows a text GUI that asks for an email address, asks for you to agree to the terms, and explains that the log will be saved to /var/log/letsencrypt/letsencrypt.log

It fails.

...
   Detail: Invalid response from
   http://www.tribalworker.com/.well-known/acme-challenge/P1hc5Fo2UXFbywNnvOL75Emq4wNHQdI1LChL6dqKa9E:
...
   Detail: Invalid response from
   http://www.holtstrom.com/.well-known/acme-challenge/Jbi_u2egXJs8c5KI5ucc8vX9CNO1Uz8V3bXQAGD4dXU:
...
   Detail: Invalid response from
   http://www.frothing.com/.well-known/acme-challenge/iGNFCAzVY4ZcjBW227p5WU4f2k5-OmvzGixDSD7MN2s:
...
   Detail: Invalid response from
   http://www.evilgoblin.com/.well-known/acme-challenge/Z3m2toYTnQEnW4Gq6OjP2EDhwjTU6QwPcBdrdB-OloA:
...
   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address.
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.

Try again without www

sudo /etc/lighttpd/ssl/certbot-auto certonly --debug --webroot -w /var/www/lighttpd/holtstrom -d holtstrom.com -w /var/www/lighttpd/evilgoblin -d evilgoblin.com -w /var/www/lighttpd/frothing -d frothing.com -w /var/www/lighttpd/tribalworker -d tribalworker.com 

That works!

...
 Generating key (2048 bits)
 /etc/letsencrypt/keys/0000_key-certbot.pem                           │
 Creating CSR: /etc/letsencrypt/csr/0000_csr-certbot.pem
...
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/holtstrom.com/fullchain.pem. Your cert will
   expire on 2017-02-25. To obtain a new or tweaked version of this
   certificate in the future, simply run certbot-auto again. To
   non-interactively renew *all* of your certificates, run
   "certbot-auto renew"

Okay, a few things. First, I want to disable EPEL, because I don't think it was necessary.

[ec2-user]$ sudo yum-config-manager --disable epel
-- output not shown --
[ec2-user]$ sudo yum repolist
Loaded plugins: priorities, update-motd, upgrade-helper
repo id                                                                     repo name                                                                status
!amzn-main/latest                                                           amzn-main-Base                                                           5,612
!amzn-updates/latest                                                        amzn-updates-Base                                                          476
repolist: 6,088

Next, the reason the command with www failed is that I have rewrite rules so that attempts to reach my domains by www redirect to the no-prefix version via the error-handler-404.

else $HTTP["host"] =~ "^www\.holtstrom\.com$" {
  server.document-root = "/var/www/lighttpd/_redirects/holtstrom"
}

So the problem was that I didn't specify the proper document roots in my certbot command.

sudo /etc/lighttpd/ssl/certbot-auto 
 certonly 
 --debug
 --webroot 
 -w /var/www/lighttpd/holtstrom -d holtstrom.com
 -w /var/www/lighttpd/_redirects/holtstrom -d www.holtstrom.com 
 -w /var/www/lighttpd/evilgoblin -d evilgoblin.com
 -w /var/www/lighttpd/_redirects/evilgoblin -d www.evilgoblin.com
 -w /var/www/lighttpd/frothing -d frothing.com 
 -w /var/www/lighttpd/_redirects/frothing -d www.frothing.com
 -w /var/www/lighttpd/tribalworker -d tribalworker.com 
 -w /var/www/lighttpd/_redirects/tribalworker -d www.tribalworker.com

Also, I'd like to automate the remaining interactive components. But before we try that, let's have a look at the files it produced.

[ec2-user]$ sudo ls /etc/letsencrypt
accounts  archive  csr  keys  live  renewal
[ec2-user]$ sudo ls /etc/letsencrypt/accounts
acme-v01.api.letsencrypt.org
[ec2-user]$ sudo ls /etc/letsencrypt/archive
holtstrom.com
[ec2-user]$ sudo ls /etc/letsencrypt/csr
0000_csr-certbot.pem
[ec2-user]$ sudo ls /etc/letsencrypt/keys
0000_key-certbot.pem
[ec2-user]$ sudo ls /etc/letsencrypt/live
holtstrom.com
[ec2-user]$ sudo ls /etc/letsencrypt/renewal
holtstrom.com.conf

According to the certbot user guide we could have used the --noninteractive flag, the --agree-tos flag and the --email flag to avoid the interactive part. Also it looks like we want to use the --expand switch to add the www domains to our cert.

sudo /etc/lighttpd/ssl/certbot-auto 
 certonly 
 --debug
 --noninteractive
 --agree-tos 
 --email you@you.com
 --expand 
 --webroot
 -w /var/www/lighttpd/holtstrom -d holtstrom.com
 -w /var/www/lighttpd/_redirects/holtstrom -d www.holtstrom.com 
 -w /var/www/lighttpd/evilgoblin -d evilgoblin.com
 -w /var/www/lighttpd/_redirects/evilgoblin -d www.evilgoblin.com
 -w /var/www/lighttpd/frothing -d frothing.com 
 -w /var/www/lighttpd/_redirects/frothing -d www.frothing.com
 -w /var/www/lighttpd/tribalworker -d tribalworker.com 
 -w /var/www/lighttpd/_redirects/tribalworker -d www.tribalworker.com

It looks like that worked.

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for holtstrom.com
http-01 challenge for www.holtstrom.com
http-01 challenge for evilgoblin.com
http-01 challenge for www.evilgoblin.com
http-01 challenge for frothing.com
http-01 challenge for www.frothing.com
http-01 challenge for tribalworker.com
http-01 challenge for www.tribalworker.com
Using the webroot path /var/www/lighttpd/_redirects/tribalworker for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0001_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0001_csr-certbot.pem

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/holtstrom.com/fullchain.pem. Your cert will
   expire on 2017-02-27. To obtain a new or tweaked version of this
   certificate in the future, simply run certbot-auto again. To
   non-interactively renew *all* of your certificates, run
   "certbot-auto renew"

Ok, so how do we get that into lighttpd?

Assuming your /etc/lighttpd/lighttpd.conf looks like this:

$SERVER["socket"] == ":443" {
  ssl.engine  = "enable"
  ssl.pemfile = "/etc/lighttpd/ssl/ssl.pem"
  ssl.ca-file = "/etc/lighttpd/ssl/intermediate.pem"
}

Let's back those up and replace their contents

# backup existing
cd /etc/lighttpd/ssl
sudo mkdir old3
sudo cp ssl.pem old3
sudo cp intermediate.pem old3
ls old3

# combine the key and cert into one file
sudo cat /etc/letsencrypt/live/holtstrom.com/privkey.pem /etc/letsencrypt/live/holtstrom.com/cert.pem > /tmp/ssl.pem
sudo mv /tmp/ssl.pem /etc/lighttpd/ssl/ssl.pem
sudo chmod 400 ssl.pem
sudo chown root:root ssl.pem

# update the chain
sudo cp /etc/letsencrypt/live/holtstrom.com/chain.pem /etc/lighttpd/ssl/intermediate.pem

# confirm permissions are safe
sudo ls -la /etc/lighttpd/ssl
-r-------- 1 root root  1647 Nov 29 12:37 intermediate.pem
-r-------- 1 root root  3672 Nov 29 12:36 ssl.pem

# restart
sudo service lighttpd restart

Looking good. Next steps are to setup auto-renewal, and ensure that all hosted content is using SSL everywhere (i.e. callback urls, and included js/css/etc).

From the certbot user guide it's not clear to me if renew can be used in a renew-but-don't-install mode. If not, we have to continue to use the certonly command for our renewals. Let's try a dry run.

sudo /etc/lighttpd/ssl/certbot-auto renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/holtstrom.com.conf
-------------------------------------------------------------------------------
Cert not due for renewal, but simulating renewal for dry run
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for holtstrom.com
http-01 challenge for evilgoblin.com
http-01 challenge for frothing.com
http-01 challenge for tribalworker.com
http-01 challenge for www.evilgoblin.com
http-01 challenge for www.frothing.com
http-01 challenge for www.holtstrom.com
http-01 challenge for www.tribalworker.com
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0002_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0002_csr-certbot.pem
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/holtstrom.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)

Let's see what that did.

[ec2-user]$ sudo ls -la /etc/letsencrypt/live/holtstrom.com
lrwxrwxrwx 1 root root   37 Nov 29 12:19 cert.pem -> ../../archive/holtstrom.com/cert2.pem
lrwxrwxrwx 1 root root   38 Nov 29 12:19 chain.pem -> ../../archive/holtstrom.com/chain2.pem
lrwxrwxrwx 1 root root   42 Nov 29 12:19 fullchain.pem -> ../../archive/holtstrom.com/fullchain2.pem
lrwxrwxrwx 1 root root   40 Nov 29 12:19 privkey.pem -> ../../archive/holtstrom.com/privkey2.pem

[ec2-user]$ sudo ls -la /etc/letsencrypt/archive/holtstrom.com
-rw-r--r-- 1 root root 1858 Nov 27 03:49 cert1.pem
-rw-r--r-- 1 root root 1968 Nov 29 12:19 cert2.pem
-rw-r--r-- 1 root root 1647 Nov 27 03:49 chain1.pem
-rw-r--r-- 1 root root 1647 Nov 29 12:19 chain2.pem
-rw-r--r-- 1 root root 3505 Nov 27 03:49 fullchain1.pem
-rw-r--r-- 1 root root 3615 Nov 29 12:19 fullchain2.pem
-rw-r--r-- 1 root root 1704 Nov 27 03:49 privkey1.pem
-rw-r--r-- 1 root root 1704 Nov 29 12:19 privkey2.pem

Okay. So #1 was the first one with the incomplete domain list. And #2 was the final result from the --extend command. And it looks like renew would just create #3. This is reinforecd by looking in the conf file which shows installer = None.

vi /etc/letsencrypt/renewal/holtstrom.com.conf

# renew_before_expiry = 30 days
version = 0.9.3
cert = /etc/letsencrypt/live/holtstrom.com/cert.pem
privkey = /etc/letsencrypt/live/holtstrom.com/privkey.pem
chain = /etc/letsencrypt/live/holtstrom.com/chain.pem
fullchain = /etc/letsencrypt/live/holtstrom.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = webroot
installer = None
account = [redacted]
webroot_path = /var/www/lighttpd/holtstrom, /var/www/lighttpd/_redirects/holtstrom, /var/www/lighttpd/evilgoblin, /var/www/lighttpd/_redirects/evilgoblin, /var/www/lighttpd/frothing, /var/www/lighttpd/_redirects/frothing, /var/www/lighttpd/tribalworker, /var/www/lighttpd/_redirects/tribalworker
[[webroot_map]]
evilgoblin.com = /var/www/lighttpd/evilgoblin
www.frothing.com = /var/www/lighttpd/_redirects/frothing
frothing.com = /var/www/lighttpd/frothing
www.holtstrom.com = /var/www/lighttpd/_redirects/holtstrom
holtstrom.com = /var/www/lighttpd/holtstrom
tribalworker.com = /var/www/lighttpd/tribalworker
www.evilgoblin.com = /var/www/lighttpd/_redirects/evilgoblin
www.tribalworker.com = /var/www/lighttpd/_redirects/tribalworker

Okay, so all we need is a cron job that runs the renew command and when successfull updates lighttpd and we can use the --post-hook for that (which is only run when the certificate is due for renewal). I think the --post-hook runs even if the renewal fails. But that's okay because our script will be non-destructive.

Here's the command we will eventually run:

sudo /etc/lighttpd/ssl/certbot-auto 
 renew 
 --debug
 --quiet
 --post-hook "/etc/lighttpd/ssl/certbot-deploy"

Now we just need to write the certbot-deploy script and schedule the cron job.

sudo vi /etc/lighttpd/ssl/certbot-deploy
#!/bin/sh

echo certbot-deploy $(date)
cat /etc/letsencrypt/live/holtstrom.com/privkey.pem /etc/letsencrypt/live/holtstrom.com/cert.pem > /tmp/ssl.pem
mv /tmp/ssl.pem /etc/lighttpd/ssl/ssl.pem
chmod 400 /etc/lighttpd/ssl/ssl.pem
chown root:root /etc/lighttpd/ssl/ssl.pem

cp /etc/letsencrypt/live/holtstrom.com/chain.pem /etc/lighttpd/ssl/intermediate.pem

/etc/init.d/lighttpd restart
sudo chmod 755 /etc/lighttpd/ssl/certbot-deploy

Now let's test it.

sudo /etc/lighttpd/ssl/certbot-deploy

Looks ok.

sudo ls -la /etc/lighttpd/ssl/

Looks ok.

We want some logs, so let's write a caller for the renew.

sudo vi /etc/lighttpd/ssl/certbot-renew
#!/bin/sh

echo certbot-renew $(date)
/etc/lighttpd/ssl/certbot-auto renew --debug --quiet --post-hook "/etc/lighttpd/ssl/certbot-deploy"
sudo chmod 755 /etc/lighttpd/ssl/certbot-renew

Here's how to setup a cron job to call the renewal nightly. This example runs on the 38th minute of the 3rd hour each day, i.e. 3:38am (server-time). LetsEncrypt asks you to choose a random minute so they don't get load spikes. The output of the cronjob will be appended to /etc/lighttpd/ssl/chron.log

Note: we must be sure to use root's crontab. The sudo below opens root's crontab. Thsi way sudo is not necessary to run the commands in this context.

sudo crontab -e

This opens something like a vi editor. Just enter the following line and save and quit.

38 3 * * * /etc/lighttpd/ssl/certbot-renew >> /etc/lighttpd/ssl/chron.log 2>&1

And let's make sure that file is read/write by all.

sudo vi /etc/lighttpd/ssl/chron.log
sudo chmod 666 /etc/lighttpd/ssl/chron.log

That't it. Now we wait and see if it works.

Update

Sadly, it seems that by the time you need the automatic Let's Encrypt update, your standard yum updates have broken the setup. I've fixed this the last two times with the following procedure

First observe the failure:

sudo vi /etc/lighttpd/ssl/chron.log
G
[esc]:q

Then observe that the mandatory symlink has been deleted:

[ec2-user]$ sudo ls -la /usr/bin/v*
-rwxr-xr-x 1 root root  113440 Feb 25  2016 /usr/bin/vdir
-rwxr-xr-x 1 root root    9858 Aug 18 23:53 /usr/bin/verifytree
-rwxr-xr-x 1 root root 2441384 Jul  6 18:26 /usr/bin/vim
lrwxrwxrwx 1 root root       3 Jul 15 16:24 /usr/bin/vimdiff -> vim
-rwxr-xr-x 1 root root    2084 Jul  6 18:26 /usr/bin/vimtutor
-rwxr-xr-x 1 root root     322 Aug 10 21:57 /usr/bin/virtualenv-2.7
-rwxr-xr-x 1 root root   23352 Mar 17  2015 /usr/bin/vmstat

Then fix it:

cd /usr/bin
sudo ln -s /usr/bin/virtualenv-2.7 /usr/bin/virtualenv
aws