Tuesday, May 14, 2019

PHP MySQL (MariaDB) in AWS Micro with Amazon Linux 2


This shows how to migrate your webserver from Amazon Linux to Amazon Linux 2. The initial Amazon Linux setup guides are here and here, but they're very old and I've included the necessary bits for first-timers below.

If you don't already have an AWS account

You can start with a free account. To sign up, you just provide: email, password, billing info, enter a capcha, and they auto-dial your home phone and get you to enter a pin.

After you receive your confirmation email, you need to sign in to the AWS Management Console

But what if someone keylogs your AWS password? You're exposed to billing-by-usage liability. Get multi factor authentication at aws.amazon.com/mfa. You can purchase a device from onlinenoram.gemalto.com.

Instance & Region

The initial free offering was a t1.micro, and there was only one eastern america region (Nothern Virginia). When you launch an EC2 instance you have to choose an availability zone within a region.

The current free offering is a t2.micro. But you have to consider the unlimited option and the t3 alternatives. For reference, I ran a LAMP stack with peaks of 200 users per day with the t1.micro with no noticeable lag. For this application, the on-demand total cost is USD $14.40/mo for the t1.micro instance and $1.22 for the EBS.

The key difference between t1.micro and t2.micro is the way the CPU bursting happens. It is not possible to predict or control the performance of t1.micro. With the T2 instance types, you can accumulate points that can be redeemed within 24 hours to get predictable performance. During a burst, when you run out of credits it falls back to the Baseline CPU Performance. (cloudacademy.com)
T2 standard was misunderstood due to its CPU throttling over baseline. Amazon introduced T2 unlimited as a way to overcome the CPU throttling with a pay for credit mechanism. The new introduction – T3, is a T2 unlimited with some subtle variations. (cloudsqueeze.ai)
t1.micro:  0.6 GiB, 1 vCPUs,                    poor network,     $0.0200/hour 
t2.micro:  1.0 GiB, 1 vCPUs (for 2h 24m burst), moderate network, $0.0116/hour 
t3.micro:  1.0 GiB, 2 vCPUs (for 2h 24m burst), moderate network, $0.0104/hour 
t3a.micro: 1.0 GiB, 2 vCPUs (for 2h 24m burst), moderate network, $0.0094/hour

t2.micro: 1 vCPUs, Baseline per vCPU = 10%
t3.micro: 2 vCPUs, Baseline per vCPU = 10%

Note that "1 vCPUs for a 2h 24m burst" is the same as "10% of 1 vCPU in a 24 hour period", and that t3.micro is effectively 20% of a vCPU. There is a good explanation at roberttisdale.com. On a t2.micro, you're constantly earning enough credits to drive the CPU at 10%. If your server load is a constant 10%, then you never earn any extra and never burst. If your customer demand is a constant 11% then your server is capped at 10% and your users have to wait.

The micro instance provides different levels of CPU resources at different times (up to 2 ECUs). By comparison, the m1.small instance type provides 1 ECU at all times. The following figure illustrates the difference. (archive.org)

They don't explicitly say, but the figure suggests that the t1.micro background level should be ~20% of an ECU in order to not have peaks throttled. So it's not clear if a t2 is weaker than a t1, but a t3 is certainly stronger than t1.

CPU credits on a running instance do not expire. For T3 and T3a, the CPU credit balance persists for seven days after an instance stops. For T2, the CPU credit balance does not persist between instance stops and starts. (amazon.com)
In standard mode, if the instance is running low on accrued credits, performance is gradually lowered to the baseline performance level. In unlimited mode, the instance can run at higher CPU utilization for a flat additional rate per vCPU-hour. T3 and T3a instances are launched as unlimited by default. T2 instances are launched as standard by default. (amazon.com)
T3 uses Intel Xeon processors and provides 30% price performance over T2. T3a uses AMD EPYC processors and provides 10% price performance over T3. (amazon.com)

concurrencylabs.com has a very extensive article explaining the price, latency and service difference between different regions.

May 3rd, 2019

On-Demand Linux:

US East (Ohio)       t3a.micro $0.0094/hr = $82/yr
                      t3.micro $0.0104/hr = $91/yr
US East (N.Virginia) t3a.micro $0.0094/hr = $82/yr
                      t3.micro $0.0104/hr = $91/yr
Canada (Central)     t3a.micro (not available)
                      t3.micro $0.0116/hr = $102/yr

Reserved Linux, standard 3-year term, partial upfront:

US East (Ohio)       t3a.micro $49.00 + $1.39/mo = $33/yr
                      t3.micro $55.00 + $1.53/mo = $37/yr
US East (N.Virginia) t3a.micro $49.00 + $1.39/mo = $33/yr 
                      t3.micro $55.00 + $1.53/mo = $37/yr
Canada (Central)     t3a.micro (not available)
                      t3.micro $61.00 + $1.68/mo = $40/yr

Note that I don't consider the EBS region difference cost because as described above it is only 8% of my total cost. Also, T3a is the better deal but I'm willing to pay the extra $3/yr to be hosted in Canada.

Therefore for a personal LAMP stack in eastern Canada that doesn't generate income you should use T3 in standard mode in Canada (Central) aka Montreal.


If you're new to AWS, you'll also need to get familiar with these concepts.

Amazon EBS volumes are off-instance storage that persists independently from the life of an instance. ... can be attached to a running Amazon EC2 instance and exposed as a device within the instance. (aws.amazon.com/ebs)
An Elastic IP address is associated with your account not a particular instance, and you control that address until you choose to explicitly release it. Unlike traditional static IP addresses, however, Elastic IP addresses allow you to mask instance or Availability Zone failures by pro grammatically remapping your public IP addresses to any instance in your account. (aws.amazon.com/ec2)

This means that you don't want to "terminate" because that kills your EBS. Instead you start/stop your EC2. But each time it gets a new IP, so you need to assign it an elastic IP which you can then point to from DNS.

New t3.micro Instance

At first I tried "64-bit (Arm)", but that was only available in a1.medium and larger. Also, you have to select the region in the top-bar before using the "Launch Instance" wizard.

AWS Management Console
In the top-bar, select "Canada (Central)" in the Region drop-down.
In the top-bar, select "Services" > "EC2"
In the side-bar, select "EC2 Dashboard"
Click the "Launch Instance" button

Amazon Linux 2 AMI (HVM), SSD Volume Type
64-bit (x86)

Next: Configure Instance Details

 Number of instances: (default) 1
 Purchasing option: (default) [unchecked] Request Spot instances
 Network: (default) VPC
 Subnet: (default) No preference
 Auto-assign Public IP: (default) Use subnet setting (Enable)
 Placement group: (default) [unchecked] Add instance to placement group
 Capacity Reservaton: (default) Open
 IAM role: (default) None
 CPU options: (default) [unchecked] Specify CPU options
 Shutdown behavior: (default) Stop  
 Enable termination protection: [checked] Protect against accidental termination
 Monitoring: (default) [unchecked] Enable CloudWatch detailed monitoring
 EBS-optimized instance: (mandatory) [checked] Launch as EBS-optimized instance
 Tenancy: (default) Shared  
 T2/T3 Unlimited: [unchecked] Enable

 Note that termination protection just prevents you from accidentally trashing 
 your instance when you just meant to power it off. And that by not allowing
 unlimited, you are not exposed to increased costs due to increased CPU.

Next: Add Storage

 Size (GiB): (default) 8
 Volume Type: (default) General Purpose SSD (gp2)
 Delete on Termination: (default) [checked]

Next: Add Tags


Next: Configure Security Group

 Name: VPC-WebServerSecurityGroup
 Description: VPC-WebServerSecurityGroup

 SSH   TCP  22
 HTTP  TCP  80, ::/0
 HTTPS TCP 443, ::/0

 Note: we need 22 since that's in the default SSHD config,
 but we'll restrict to IP and later will change the port.

Review and Launch
Choose an existing key pair // if you don't have one, see below
Launch Instances

If you've never created an EC2 instance before, you'll need to create a keypair like this:

 # continue
 - create a new key pair
 - name: WebServerKey
 # create and download your key pair 
 // this is WebServerKey.pem that you will use later with putty

After that, navigate to the EC2 dashboard where you can see the instance starting up.

Left-click the blank instance name and give it something meaningful.

In the Description tab below your new instance, take note of its Public IP.

Basic SSH

Clone your existing putty config but use the new IP and port 22. Connect to your instance with putty.

If you've never connected to an EC2 instance before and you just created WebServerKey.pem above, then you'll need to do the following.

If you're on windows, download putty.

The WebServer.pem key won't work with putty, you have to convert it. Download puttygen.

 conversions > import key > WebServerKey.pem
 confirm passphrase: STRONG_PASSWORD_AGAIN
 save private key > WebServerKey.ppk

You now have an encrypted .ppk version of your plaintext .pem key. I'd suggest deleting, encrypting, or storing the .pem on a thumb drive.

Use putty to connect:

session.saved_sessions: amazon
session.port: 22
session.type: SSH
window.lines_of_scrollback: 2000
window.colours.use_system_colours: checked
connection.data.auto-login_username: ec2-user
connection.ssh.auth.private_key_file_for_authentication: WebServerKey.ppk

Session > Save
then double-click: amazon
- accept unknown thumbrint (only happens once)
- enter your .ppk passhprase

You're in! Now let's make it bullet proof.

Harden SSH

sudo yum -y update

That will probably pickup a bunch of updates.

Next make a copy of the config we're about to copy. We'll download it as a local backup later.

mkdir /tmp/org
sudo cp /etc/ssh/sshd_config /tmp/org
sudo chmod 644 /tmp/org/sshd_config

Now we'll harden the SSHD config.

sudo vi /etc/ssh/sshd_config

Ensure the following are in place.

# change the port to the custom SSH port from your security group
# this makes it a little bit harder for people to attack you as they
# now have to scan all ports to discover which is your SSH port
Port 12345

# Explicitly require strong protocol 2 (which is the default)
Protocol 2

# change this to no, we never want root access over SSH
PermitRootLogin no

# explicitly disable weak authentication systems
RhostsRSAAuthentication no
HostbasedAuthentication no
IgnoreUserKnownHosts yes
IgnoreRhosts yes

# EC2 uses keys for remote access
PasswordAuthentication no
PermitEmptyPasswords no

# Explicitly disable Kerberos Authentication
KerberosAuthentication no

# Explicitly disable GSSAPI Authentication
GSSAPIAuthentication no

# explicitly disable x11 forwarding, we will never connect with a gui
X11Forwarding no

Reboot. This will pickup the SSHD changes and anything from the yum update.

sudo shutdown -r now

In the AWS Console remove the temporary port 22 entry from your security group so that you just have something like this.

 HTTP             TCP    80
 HTTP             TCP    80 ::/0
 HTTPS            TCP   443
 HTTPS            TCP   443 ::/0
 Custom TCP Rule  TCP 12345

Re-connect with putty now using the custom port. Note that the instance will have a new public IP.

No Lighttpd

Note that in the previous posts I used the Lighttpd webserver but packages from Amazon Linux 1 aren't available on Amazon Linux 2.

Here's what happens if you try the old packages.

Don't do this:

$ sudo yum -y install lighttpd lighttpd-fastcgi
No package lighttpd available.
No package lighttpd-fastcgi available.

$ sudo yum -y install mysql mysql-server
No package mysql-server available.
Package mariadb.x86_64 1:5.5.60-1.amzn2 will be installed

$ sudo yum -y install php-cli php-mysql php-mbstring php-xml
Package php-mysql is obsoleted by php-mysqlnd, trying to install php-mysqlnd-5.4.16-45.amzn2.0.6.x86_64 instead

Install LAMP

First, have a look at the default users and groups on your system.

sudo cat /etc/passwd
sudo cat /etc/group

I make a local backup of these for my records.

Tutorial: Install a LAMP Web Server on Amazon Linux 2

Apache web server
PHP 7.2
MariaDB (a community-developed fork of MySQL)
Amazon Linux 2 

The above doesn't include SSL (Tutorial: Configure Apache Web Server on Amazon Linux 2 to Use SSL/TLS) or the [php-mbstring php-xml] extensions that I require.

Here's the whole package that I use:

sudo amazon-linux-extras install -y lamp-mariadb10.2-php7.2 php7.2
sudo yum install -y httpd mariadb-server
sudo yum install -y mod_ssl
sudo yum install -y php-mbstring php-xml


Installing php-pdo, php-mysqlnd, php-fpm, php-cli, php-json, mariadb
 Package                         Arch                    Version               
 mariadb                         x86_64                  3:10.2.10-2.amzn2.0.3 
 php-cli                         x86_64                  7.2.16-1.amzn2.0.1    
 php-fpm                         x86_64                  7.2.16-1.amzn2.0.1    
 php-json                        x86_64                  7.2.16-1.amzn2.0.1    
 php-mysqlnd                     x86_64                  7.2.16-1.amzn2.0.1    
 php-pdo                         x86_64                  7.2.16-1.amzn2.0.1    
Installing for dependencies:
 mariadb-common                  x86_64                  3:10.2.10-2.amzn2.0.3 
 mariadb-config                  x86_64                  3:10.2.10-2.amzn2.0.3 
 php-common                      x86_64                  7.2.16-1.amzn2.0.1    
Updating for dependencies:
 mariadb-libs                    x86_64                  3:10.2.10-2.amzn2.0.3 

  Installing : php-json-7.2.16-1.amzn2.0.1.x86_64          
  Installing : php-common-7.2.16-1.amzn2.0.1.x86_64        
  Installing : php-pdo-7.2.16-1.amzn2.0.1.x86_64           
  Installing : 3:mariadb-config-10.2.10-2.amzn2.0.3.x86_64 
  Installing : 3:mariadb-common-10.2.10-2.amzn2.0.3.x86_64 
  Updating   : 3:mariadb-libs-10.2.10-2.amzn2.0.3.x86_64   
  Installing : 3:mariadb-10.2.10-2.amzn2.0.3.x86_64        
  Installing : php-mysqlnd-7.2.16-1.amzn2.0.1.x86_64       
  Installing : php-cli-7.2.16-1.amzn2.0.1.x86_64           
  Installing : php-fpm-7.2.16-1.amzn2.0.1.x86_64           

  mariadb.x86_64 3:10.2.10-2.amzn2.0.3     
  php-cli.x86_64 0:7.2.16-1.amzn2.0.1  
  php-fpm.x86_64 0:7.2.16-1.amzn2.0.1  
  php-json.x86_64 0:7.2.16-1.amzn2.0.1
  php-mysqlnd.x86_64 0:7.2.16-1.amzn2.0.1  
  php-pdo.x86_64 0:7.2.16-1.amzn2.0.1

Dependency Installed:
  mariadb-common.x86_64 3:10.2.10-2.amzn2.0.3          
  mariadb-config.x86_64 3:10.2.10-2.amzn2.0.3          
  php-common.x86_64 0:7.2.16-1.amzn2.0.1

Dependency Updated:
  mariadb-libs.x86_64 3:10.2.10-2.amzn2.0.3

 Package                                      Arch                Version               
 httpd                                        x86_64              2.4.39-1.amzn2.0.1    
 mariadb-server                               x86_64              3:10.2.10-2.amzn2.0.3 
Installing for dependencies:
 apr                                          x86_64              1.6.3-5.amzn2.0.2     
 apr-util                                     x86_64              1.6.1-5.amzn2.0.2     
 apr-util-bdb                                 x86_64              1.6.1-5.amzn2.0.2     
 bison                                        x86_64              3.0.4-6.amzn2.0.2     
 generic-logos-httpd                          noarch              18.0.0-4.amzn2        
 httpd-filesystem                             noarch              2.4.39-1.amzn2.0.1    
 httpd-tools                                  x86_64              2.4.39-1.amzn2.0.1    
 jemalloc                                     x86_64              3.6.0-1.amzn2.0.1     
 m4                                           x86_64              1.4.16-10.amzn2.0.2   
 mailcap                                      noarch              2.1.41-2.amzn2        
 mariadb-backup                               x86_64              3:10.2.10-2.amzn2.0.3 
 mariadb-cracklib-password-check              x86_64              3:10.2.10-2.amzn2.0.3 
 mariadb-errmsg                               x86_64              3:10.2.10-2.amzn2.0.3 
 mariadb-gssapi-server                        x86_64              3:10.2.10-2.amzn2.0.3 
 mariadb-rocksdb-engine                       x86_64              3:10.2.10-2.amzn2.0.3 
 mariadb-server-utils                         x86_64              3:10.2.10-2.amzn2.0.3 
 mariadb-tokudb-engine                        x86_64              3:10.2.10-2.amzn2.0.3 
 mod_http2                                    x86_64              1.14.1-1.amzn2        
 perl-Compress-Raw-Bzip2                      x86_64              2.061-3.amzn2.0.2     
 perl-Compress-Raw-Zlib                       x86_64              1:2.061-4.amzn2.0.2   
 perl-DBD-MySQL                               x86_64              4.023-6.amzn2         
 perl-DBI                                     x86_64              1.627-4.amzn2.0.2     
 perl-Data-Dumper                             x86_64              2.145-3.amzn2.0.2     
 perl-IO-Compress                             noarch              2.061-2.amzn2         
 perl-Net-Daemon                              noarch              0.48-5.amzn2          
 perl-PlRPC                                   noarch              0.2020-14.amzn2       

  Installing : apr-1.6.3-5.amzn2.0.2.x86_64                                
  Installing : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                       
  Installing : apr-util-1.6.1-5.amzn2.0.2.x86_64                           
  Installing : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                   
  Installing : httpd-tools-2.4.39-1.amzn2.0.1.x86_64                       
  Installing : jemalloc-3.6.0-1.amzn2.0.1.x86_64                           
  Installing : m4-1.4.16-10.amzn2.0.2.x86_64                               
  Installing : bison-3.0.4-6.amzn2.0.2.x86_64                              
  Installing : perl-Net-Daemon-0.48-5.amzn2.noarch                         
  Installing : 3:mariadb-errmsg-10.2.10-2.amzn2.0.3.x86_64                 
  Installing : httpd-filesystem-2.4.39-1.amzn2.0.1.noarch                  
  Installing : perl-Compress-Raw-Bzip2-2.061-3.amzn2.0.2.x86_64            
  Installing : generic-logos-httpd-18.0.0-4.amzn2.noarch                   
  Installing : mailcap-2.1.41-2.amzn2.noarch                               
  Installing : mod_http2-1.14.1-1.amzn2.x86_64                             
  Installing : httpd-2.4.39-1.amzn2.0.1.x86_64                             
  Installing : 1:perl-Compress-Raw-Zlib-2.061-4.amzn2.0.2.x86_64           
  Installing : perl-IO-Compress-2.061-2.amzn2.noarch                       
  Installing : perl-PlRPC-0.2020-14.amzn2.noarch                           
  Installing : perl-DBI-1.627-4.amzn2.0.2.x86_64                           
  Installing : perl-DBD-MySQL-4.023-6.amzn2.x86_64                         
  Installing : 3:mariadb-backup-10.2.10-2.amzn2.0.3.x86_64                 
  Installing : 3:mariadb-tokudb-engine-10.2.10-2.amzn2.0.3.x86_64          
  Installing : 3:mariadb-rocksdb-engine-10.2.10-2.amzn2.0.3.x86_64         
  Installing : 3:mariadb-cracklib-password-check-10.2.10-2.amzn2.0.3.x86_64
  Installing : 3:mariadb-gssapi-server-10.2.10-2.amzn2.0.3.x86_64          
  Installing : 3:mariadb-server-10.2.10-2.amzn2.0.3.x86_64                 
  Installing : 3:mariadb-server-utils-10.2.10-2.amzn2.0.3.x86_64           

  httpd.x86_64 0:2.4.39-1.amzn2.0.1                                       
  mariadb-server.x86_64 3:10.2.10-2.amzn2.0.3

Dependency Installed:
  apr.x86_64 0:1.6.3-5.amzn2.0.2                                          
  apr-util.x86_64 0:1.6.1-5.amzn2.0.2
  apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2                                  
  bison.x86_64 0:3.0.4-6.amzn2.0.2
  generic-logos-httpd.noarch 0:18.0.0-4.amzn2                             
  httpd-filesystem.noarch 0:2.4.39-1.amzn2.0.1
  httpd-tools.x86_64 0:2.4.39-1.amzn2.0.1                                 
  jemalloc.x86_64 0:3.6.0-1.amzn2.0.1
  m4.x86_64 0:1.4.16-10.amzn2.0.2                                         
  mailcap.noarch 0:2.1.41-2.amzn2
  mariadb-backup.x86_64 3:10.2.10-2.amzn2.0.3                             
  mariadb-cracklib-password-check.x86_64 3:10.2.10-2.amzn2.0.3
  mariadb-errmsg.x86_64 3:10.2.10-2.amzn2.0.3                             
  mariadb-gssapi-server.x86_64 3:10.2.10-2.amzn2.0.3
  mariadb-rocksdb-engine.x86_64 3:10.2.10-2.amzn2.0.3                     
  mariadb-server-utils.x86_64 3:10.2.10-2.amzn2.0.3
  mariadb-tokudb-engine.x86_64 3:10.2.10-2.amzn2.0.3                      
  mod_http2.x86_64 0:1.14.1-1.amzn2
  perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.amzn2.0.2                      
  perl-Compress-Raw-Zlib.x86_64 1:2.061-4.amzn2.0.2
  perl-DBD-MySQL.x86_64 0:4.023-6.amzn2                                   
  perl-DBI.x86_64 0:1.627-4.amzn2.0.2
  perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2                             
  perl-IO-Compress.noarch 0:2.061-2.amzn2
  perl-Net-Daemon.noarch 0:0.48-5.amzn2                                   
  perl-PlRPC.noarch 0:0.2020-14.amzn2

 Package                           Arch                           Version               
 mod_ssl                           x86_64                         1:2.4.39-1.amzn2.0.1  
Installing for dependencies:
 libtalloc                         x86_64                         2.1.13-1.amzn2        
 sscg                              x86_64                         2.3.3-2.amzn2.0.1     

  Installing : libtalloc-2.1.13-1.amzn2.x86_64     
  Installing : sscg-2.3.3-2.amzn2.0.1.x86_64       
  Installing : 1:mod_ssl-2.4.39-1.amzn2.0.1.x86_64 

  mod_ssl.x86_64 1:2.4.39-1.amzn2.0.1

Dependency Installed:
  libtalloc.x86_64 0:2.1.13-1.amzn2                                             
  sscg.x86_64 0:2.3.3-2.amzn2.0.1

 Package                        Arch                     Version            
 php-mbstring                   x86_64                   7.2.16-1.amzn2.0.1 
 php-xml                        x86_64                   7.2.16-1.amzn2.0.1 
Installing for dependencies:
 libxslt                        x86_64                   1.1.28-5.amzn2.0.2 
 oniguruma                      x86_64                   5.9.6-1.amzn2      

  Installing : oniguruma-5.9.6-1.amzn2.x86_64           
  Installing : libxslt-1.1.28-5.amzn2.0.2.x86_64        
  Installing : php-xml-7.2.16-1.amzn2.0.1.x86_64        
  Installing : php-mbstring-7.2.16-1.amzn2.0.1.x86_64   

  php-mbstring.x86_64 0:7.2.16-1.amzn2.0.1                                       
  php-xml.x86_64 0:7.2.16-1.amzn2.0.1

Dependency Installed:
  libxslt.x86_64 0:1.1.28-5.amzn2.0.2                                           
  oniguruma.x86_64 0:5.9.6-1.amzn2

Now look again to see what new users and groups have been added.

sudo cat /etc/passwd
sudo cat /etc/group

New entires:


Also, get a local backup of all the config files we're about to modify.

sudo cp /etc/httpd/conf/httpd.conf /tmp/org
sudo cp /etc/httpd/conf.d/ssl.conf /tmp/org
sudo cp /etc/httpd/conf.modules.d/00-mpm.conf /tmp/org
sudo cp /etc/my.cnf /tmp/org
sudo cp /etc/php.ini /tmp/org

Then use something like WinSCP to copy these to local.

WinSCP 4.3.5
Installation package
winscp435setup.exe > agree with defaults

Port: 12345
User: ec2-user
Password: [blank]
Private key file: WebServerKey.ppk
File protocol: SFTP (don't allow scp fallback)
> save > login > [enter password when prompted]

After the copy delete the temp files.

sudo rm -rf /tmp/org

Setup and harden Apache

Start the Apache web server.

sudo systemctl start httpd

Note, in the following, apache may fail to start because of bad config. In that case you will get an error like "Job for httpd.service failed because the control process exited with error code." To see the actual error code, you issues the status command.

systemctl status httpd.service

Configure the Apache web server to start at each system boot.

sudo systemctl enable httpd

Add ec2-user to the apache group.

sudo usermod -a -G apache ec2-user

Close your terminal and log back in to pickup the change then issues these commands so that ec2-user and future members of the apache group can modify apache files.

sudo chown -R ec2-user:apache /var/www
sudo chmod 2775 /var/www && find /var/www -type d -exec sudo chmod 2775 {} \;
find /var/www -type f -exec sudo chmod 0664 {} \;

In a browser go to the IP of your EC2 instance, i.e. to view the Apache test page. Use the browser to inspect the response and see that etag and version info are exposed.

Response Headers:
 Accept-Ranges: bytes
 Connection: Upgrade, Keep-Alive
 Content-Length: 3630
 Content-Type: text/html; charset=UTF-8
 Date: Sun, 05 May 2019 16:50:47 GMT
 ETag: "e2e-585b840220000"
 Keep-Alive: timeout=5, max=100
 Last-Modified: Thu, 04 Apr 2019 18:08:00 GMT
 Server: Apache/2.4.39 () OpenSSL/1.0.2k-fips
 Upgrade: h2,h2c

We don't want that. Also note the HTTP/2 error in your logs:

sudo cat /var/log/httpd/error_log
[http2:warn] [pid 3358] AH10034: The mpm module (prefork.c) is not supported by 
mod_http2. The mpm determines how things are processed in your server. HTTP/2 has 
more demands in this regard and the currently selected mpm will just not do. This is an 
advisory warning. Your server will continue to work, but the HTTP/2 protocol will be 

You can additionally verify that your test page doesn't support HTTP/2 by entering your public IP at http2.pro.

Also observe that your current Server MPM is prefork:

httpd -V
Server version: Apache/2.4.39 ()
Server MPM:     prefork
  threaded:     no

We'll harden apache and resolve these issues as follows.

For reasoning see geekflare, vaulted.io, stackoverflow.com.

Note that I've added AllowOverrideList to completely disable htaccess.

Note that since the remaining Options command doesn't include Indexes, that means directory listing is forbidden.

Note that ProxyErrorOverride was added to resolve what appears to be a bug in the default config of apache/php. See howtoforge.com and stackoverflow.com for details. Here is what happens in the default config:

example.com/fake.php ->
 Browser output: "File not found."
 Error.log: [proxy_fcgi:error] [client] AH01071: Got error 'Primary script unknown\n'
example.com/fake.html -> default 404 page
example.com/fake/fake.php -> default 404 page

The problem here is that if you specify a custom error document it won't be processed for 404 of *.php in the root folder. And this resolved via ProxyErrorOverride.

sudo vi /etc/httpd/conf/httpd.conf

Delete the existing <Directory "/var/www/html"> node.
Delete the existing <Directory "/var/www/cgi-bin"> node.
And make the following edits:

<Directory />
    AllowOverride none
    AllowOverrideList none
    Require all denied

    Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure
    Header always append X-Frame-Options SAMEORIGIN
    Header set X-XSS-Protection "1; mode=block"

    Header unset Etag
    FileETag None
    RewriteEngine On
    RewriteCond %{SERVER_PROTOCOL} ^HTTP/0\.9$
    RewriteRule ^ - [F]
    RewriteCond %{SERVER_PROTOCOL} ^HTTP/1\.0$
    RewriteRule ^ - [F]
<Directory "/var/www">
    Options FollowSymLinks
    Require all granted
    ProxyErrorOverride on
<Location "/">
    Deny from all
ServerTokens Prod
ServerSignature Off
TraceEnable off
Timeout 60
rmdir /var/www/cgi-bin
sudo vi /etc/httpd/conf.modules.d/00-mpm.conf

Change this:

LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
#LoadModule mpm_event_module modules/mod_mpm_event.so

To this:

#LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
LoadModule mpm_event_module modules/mod_mpm_event.so

Restart apache.

sudo systemctl restart httpd

Observe that your Server MPM is now event/threaded:

httpd -V
Server version: Apache/2.4.39 ()
Server MPM:     event
  threaded:     yes (fixed thread count)

Observe that your error log now shows a successful mpm notice

sudo cat /var/log/httpd/error_log
[mpm_event:notice] [pid 30266:tid 139623658481856] AH00489: 
 Apache/2.4.39 () OpenSSL/1.0.2k-fips configured 
 -- resuming normal operations

Observe that your public IP at http2.pro now indicates HTTP/2 is supported.

Reload your public IP in the browser, and observe that the etag and version are removed. And that X-Frame and X-XSS are added. You may need to CTRL+F5.

Response Headers:
 Accept-Ranges: bytes
 Connection: Upgrade, Keep-Alive
 Content-Length: 3630
 Content-Type: text/html; charset=UTF-8
 Date: Sun, 05 May 2019 16:59:46 GMT
 Keep-Alive: timeout=5, max=100
 Last-Modified: Thu, 04 Apr 2019 18:08:00 GMT
 Server: Apache
 Upgrade: h2,h2c
 X-Frame-Options: SAMEORIGIN
 X-XSS-Protection: 1; mode=block

To convince yourself that HTTP/0.9 and HTTP/1.0 are blocked, you can add a HTTP/1.1 line and save and restart apache and refresh your test page. You'll get the forbidden response instead.

To convince yourself that only OPTIONS GET HEAD POST are permitted, remove GET from LimitExcept and save and restart apache and refresh your test page. You'll get the forbidden response instead.

To convince yourself that directory listing is disabled, add a folder and file and attempt to visit the folder. You'll get the forbidden response instead.

Confirm that apache doesn't run as root. The ps command should display one process running as root (which allows apache to listen on port 80) and the rest as the apache user.

ps -ef |grep http

Confirm that everything except the web content files are owned by root.

sudo ls -la /etc/httpd
sudo ls -la /var/log/httpd
sudo ls -la /usr/lib64/httpd

We must define domain mappings via virtualhosts.conf. But the first one defined will also act as the default/primary server for unspecified server names, so lets adapt the default html to explicitly indicate those as invalid.

mv /var/www/html /var/www/_invalid
vi /var/www/_invalid/index.html

For this to work you must change:

sudo vi /etc/httpd/conf/httpd.conf
DocumentRoot "/var/www/html"


DocumentRoot "/var/www/_invalid"

Note that all *.conf in the /conf.d/ are picked up at the end of the /etc/httpd/conf/httpd.conf via:

IncludeOptional conf.d/*.conf

So, create the following:

sudo vi /etc/httpd/conf.d/virtualhost.conf
<VirtualHost *:80>
  ServerName invalid
  DocumentRoot /var/www/_invalid
<VirtualHost *:80>
  ServerName example.com
  DocumentRoot /var/www/example.com
<VirtualHost *:80>
  ServerName another.com
  DocumentRoot /var/www/another.com

For this to work the DocumentRoot paths must exist.

mkdir /var/www/example.com
vi /var/www/example.com/index.html

mkdir /var/www/another.com
vi /var/www/another.com/index.html

If your local box is Windows, you can edit your hosts so that a test domain maps to the IP of your AWS box.

C:\Windows\System32\drivers\etc\hosts example.com www.example.com another.com www.another.com fake.com
sudo systemctl restart httpd

You should get the following results in your browser:          contents of /var/www/_invalid/index.html
fake.com         contents of /var/www/_invalid/index.html
example.com      contents of /var/www/example.com/index.html
www.example.com  error 404
another.com      contents of /var/www/another.com/index.html 
www.another.com  error 404

I want to always strip away the www, but we'll do that after we've setup SSL.


In this setup I'm using one cert from LetsEncrypt with SubjectAltNames for each domain.

I think that the install of mod_ssl created a key and self-signed cert at the default paths from /etc/httpd/conf.d/ssl.conf

sudo ls -la /etc/pki/tls/certs/localhost.crt
-rw-r--r-- root root /etc/pki/tls/certs/localhost.crt

sudo ls -la /etc/pki/tls/private/localhost.key
-rw------- root root /etc/pki/tls/private/localhost.key

Migrate the cert and key from the existing server. Replace the contents of the existing .crt and .key file on your new server.

sudo vi /etc/pki/tls/certs/localhost.crt

sudo vi /etc/pki/tls/private/localhost.key
 -----END PRIVATE KEY-----

The default config allows a wide range of crypto. There is no reason to support anything but the modern algorithms that work with your certificate.

Note that the OpenSSL terms don't match the TLS specification. You can find a map at openssl.org. And the definition of the SSLCipherSuite directive is at apache.org.

sudo vi /etc/httpd/conf.d/ssl.conf
<VirtualHost _default_:443>
  SSLEngine on
  SSLProtocol -ALL +TLSv1.2
  SSLCertificateFile /etc/pki/tls/certs/localhost.crt
  SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

Now append to your virtual hosts file. I would like to just add the ServerName and DocumentRoot. In fact this works because it gets the SSL settings from the _default_ entry in ssl.conf, but your phpinfo $_SERVER['SERVER_PORT'] will report 80 and $_SERVER['HTTPS'] will be undefined because the SSL directives weren't explicitly in your VirtualHost. See a similar issue on serverfault.com. For this reason, I've duplicated the full SSL config in each VirtualHost.

sudo vi /etc/httpd/conf.d/virtualhost.conf
<VirtualHost *:443>
  ServerName example.com
  DocumentRoot /var/www/example.com
  SSLEngine on
  SSLProtocol -ALL +TLSv1.2
  SSLCertificateFile /etc/pki/tls/certs/localhost.crt
  SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
sudo systemctl restart httpd

Now you should be able to access both of the following urls. We'll set this up with LetsEncrypt auto-renewal later.



For SSL sites, I only want urls of the form https://example.com/index.html, which means redirecting www and http. For non-SSL sites, I only need to redirect the www. I liked the suggestion of simonecarletti.com but to use that outside a virtualhost, you need all your destinations to be SSL.

Here's a scenario where you have example.com as SSL and another.com without SSL. Also, I've added custom 404 handlers for both.

sudo vi /etc/httpd/conf.d/virtualhost.conf
<VirtualHost *:80>
  ServerName invalid
  DocumentRoot /var/www/_invalid
  ErrorDocument 404 /index.html

<VirtualHost *:80>
  ServerName another.com
  ServerAlias www.another.com
  DocumentRoot /var/www/another.com
  ErrorDocument 404 /index.html

  RewriteEngine On
  RewriteCond %{HTTP_HOST} ^www\. [NC]
  RewriteRule ^/(.*) http://another.com/$1 [R=301,L,NE]

<VirtualHost *:80>
  ServerName example.com
  ServerAlias www.example.com
  DocumentRoot /var/www/example.com

  RewriteEngine On
  RewriteRule ^/(.*) https://example.com/$1 [R=301,L,NE]

<VirtualHost *:443>
  ServerName example.com
  ServerAlias www.example.com
  DocumentRoot /var/www/example.com
  ErrorDocument 404 /index.html

  SSLEngine on
  SSLProtocol -ALL +TLSv1.2
  SSLCertificateFile /etc/pki/tls/certs/localhost.crt
  SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

  RewriteEngine On
  RewriteCond %{HTTP_HOST} ^www\. [NC]
  RewriteRule ^/(.*) https://example.com/$1 [R=301,L,NE]
sudo systemctl restart httpd

Now you should have the following behaviour: -> index.html -> error page
fake.com -> index.html
fake.com/blah -> error page

another.com -> index.html
www.another.com -> another.com -> index.html
another.com/blah -> error page

http://example.com -> https://example.com
https://www.example.com -> https://example.com
http://www.example.com/index.html?one=two#three -> https://example.com/index.html?one=two#three

PHP Config

sudo vi /etc/php.ini
# It looks like this is the default now, so this may be unnecessary.
cgi.fix_pathinfo = 1

# I use short tags
short_open_tag = On

# Simpler logs
error_reporting = E_ALL & ~E_NOTICE & ~E_DEPRECATED & ~E_STRICT

# decrease max post size so you don't waste time on bogus payloads
# this also limits the size of attack packets and reduces the risk of overflow
post_max_size = 1M

# http://tympanus.net/codrops/2009/08/31/solving-php-mysql-utf-8-issues/
# UTF8
mbstring.language = Neutral
mbstring.internal_encoding = UTF-8
mbstring.encoding_translation = Off
mbstring.http_input = auto
mbstring.http_output = UTF-8
mbstring.detect_order = auto
mbstring.substitute_character = none
default_charset = UTF-8

Restart php.

sudo systemctl restart php-fpm

Create a test php page.

vi /var/www/example.com/test.php
hello <? echo "world"; ?> <?php echo " again"; ?>

You should be able to see this at http://example.com/test.php
It should say: hello world again

Create a phpinfo page, view it, and save it to pdf for future reference.

vi /var/www/example.com/phpinfo.php
<? phpinfo(); ?>

Afterwards delete it because it contains sensitive information.

rm /var/www/example.com/phpinfo.php

Setup and harden DB

Configure the db server to start at each system boot.

sudo systemctl enable mariadb

Secure the database server. It's default config is for development and test.

sudo systemctl start mariadb
sudo mysql_secure_installation
[enter]   # existing root mysql password is blank
Y         # yes set a root mysql password
password  # choose a password
password  # enter it again
Y         # remove anonymous users
Y         # disallow root login remotely
Y         # remove the test db
Y         # reload privilege tables now

The default config listens to the network.

# this shows that it is listening
sudo netstat -tap | grep mysql
sudo vi /etc/my.cnf
# ensure you already have this line

# append this line to the [mysqld] block to disable TCP/IP listening

# append these to the [mysqld] block to set utf8 as your default

# I forget why I append this line to the [mysqld] block

Restart MySQL for the settings to take effect and look at netstat again to see that it's no longer listening.

sudo systemctl restart mariadb
sudo netstat -tap | grep mysql

Here is the full list of port uses. Note that nothing is on port 22 (the standard SSH port).

sudo lsof -i -P

Note that even with all that utf8 config, you still need to explicitly select utf8 in your php mysqli constructor like this $db->set_charset('utf8'); if you want $db->character_set_name() to return utf8 instead of latin1.

We will now create all the databases that we wish to migrate. If you don't remember which databases you created, you can use show databases; in mysql on your live box.

mysql -u root -p 

SELECT User, Host, Password FROM mysql.user;
--- no passwords in the returned list should be blank

SELECT * FROM mysql.db;
--- empty result means the anonymous access to test tables has been deleted

CREATE USER 'db1user'@'localhost' IDENTIFIED BY 'db1password';
GRANT ALL PRIVILEGES ON db1name.* TO 'db1user'@'localhost';

CREATE USER 'db2user'@'localhost' IDENTIFIED BY 'db2password';
GRANT ALL PRIVILEGES ON db2name.* TO 'db2user'@'localhost';


Next you must migrate all your database contents with commands like the following.

Old box:

mysqldump --user=db1user --password=db1password --skip-lock-tables --databases db1name > /tmp/db1name.sql

New box:

mysql -D db1name -u db1user -p
source /tmp/db1name.sql

Be sure to delete the SQL from the /tmp folder once you're done.

Migrate Content

Next you must migrate all your webserver files.

Old box:


New box:


Once your files are uploaded, you'll need to create any symlinks that exist on your source server.

Old box:

sudo find /var/www/lighttpd -type f | sort > /tmp/files1.txt
sudo find /var/www/lighttpd -type l | sort > /tmp/links1.txt

Create necessary symlinks on the new box.

New box:

sudo find /var/www -type f | sort > /tmp/files2.txt
sudo find /var/www -type l | sort > /tmp/links2.txt

Diff the outputs to confirm the migration was identical.

Backup Initial Config

Get a local backup of the modified files.

mkdir /tmp/org
sudo cp /etc/ssh/sshd_config /tmp/org
sudo chmod 644 /tmp/org/sshd_config
sudo cp /etc/httpd/conf/httpd.conf /tmp/org
sudo cp /etc/httpd/conf.d/ssl.conf /tmp/org
sudo cp /etc/httpd/conf.modules.d/00-mpm.conf /tmp/org
sudo cp /etc/my.cnf /tmp/org
sudo cp /etc/php.ini /tmp/org

Then use WinSCP to copy these to local and delete the temp files.

sudo rm -rf /tmp/org

Go Live

Restart you new box to ensure all the settings take effect. Edit your hosts file to test the new server before swapping your elastic IP. You can move an elastic IP across regions, so I created a new one in the new region, assigned it, then updated my dns records at my domain registrar to point to the new IP.

Once the DNS has fully propagated you are now live and can shutdown and archive the old box.

Local Backup

You really should setup AMI backups, but it's not unreasonable to want a local copy of the data as well. I always have a local copy of the source code, but I want a way to fetch nightly backups of the database. You can achieve that by hosting an encrypted dump of the db at a randomly named folder and get your local machine to fetch it each night.

Choose a web location for the backup.

mkdir /var/www/example.com/randomFolderName/

Create the backup script.

sudo mkdir /backup
sudo chown ec2-user:ec2-user /backup
vi /backup/go.sh

# call this script with no arguments to create an encrypted dump of all your databases
# call this script with a file name as the only argument to decrypt that file
# here's how to setup a cron job to call this script nightly
# this example runs on the 5th minute fo the 4th hour each day, i.e. 4:05am
# watch-out, that's the time of the server, which may not be your local time
# the output of the cronjob will be appended to /backup/chron.log
# crontab -e
# 5 4 * * * /backup/go.sh 2>&1 >> /backup/chron.log

if [ "$1" = "" ]; then

  today=`date +%Y_%m_%d`
  if [ -e $today -o -e $today.dat ]; then
    echo "$today already exists"
  echo `date`
  mkdir $today

  mysqldump --user=user1 --password=password1 --skip-lock-tables --databases database1 > ./$today/database1_$today.sql
  mysqldump --user=user2 --password=password2 --skip-lock-tables --databases database2 > ./$today/database2_$today.sql

  tar czf - $today | openssl des3 -salt -k password | dd of=$today.dat
  rm -rf $today
  rm -f /var/www/example.com/randomFolderName/*
  mv $today.dat /var/www/example.com/randomFolderName/
  echo "finished"

elif [ -f "$1" ]; then

  dd if="$1" | openssl des3 -d -k password | tar xzf -


  echo "$1 doesn't exist"

chmod 700 /backup/go.sh

Create the cronjob.

crontab -e
5 4 * * * /backup/go.sh 2>&1 >> /backup/chron.log

Windows Scheduler

To get a windows box to automatically wake up each night and download your backup, you can use the windows scheduler and the background intelligent transfer service (bits).

First create a local bat file that will fetch your backup from the web.

bitsadmin /TRANSFER jobname /DOWNLOAD http://site1.com/randomFolderName/%DATE:~6,4%_%DATE:~3,2%_%DATE:~0,2%.dat C:\Backups\%DATE:~6,4%_%DATE:~3,2%_%DATE:~0,2%.dat

Next, schedule a task to wake the computer and run the script. I'm assuming that your computer is configured to automatically go back to sleep after an idle period.

Start > Control Panel > Administrative Tools > Task Scheduler

[right sidebar] > Create Task

 Name: Fetch Database
 [checked] Run with highest privileges

  Start: 2am (some time shortly after your web backup becomes available)

  Action: Start a program
  Program/script: C:\Backups\fetch.bat (or whatever you named the above script) 

 [checked] Wake the computer to run this task

Your task now appears in the "Active Tasks" list in the Task Scheduler.


It's a good idea to setup some AWS alarms to let you know when your systems are operating outside their expected range. Basic Monitoring metrics (at five-minute frequency) for Amazon EC2 instances and EBS volumes are free of charge.

In the AWS Console, on your new instance:

[right-click] EC2 Instance > CloudWatch Monitoring > Add/Edit Alarms > Create Alarm

[checked] send a notification to: your contact info
Whenever: Average of CPU Utilization
Is: >= 40 Percent
For at least 1 consecutive period of 5 minutes

Create Alarm > Close

I've only exceeded that CPU range when there was a bug in my code and php was stuck in a loop.

[right-click] EC2 Instance > CloudWatch Monitoring > Add/Edit Alarms > Create Alarm

[checked] send a notification to: your contact info
Whenever: Average of Network Out
Is: >= 150000 Bytes
For at least 1 consecutive period of 6 hours

Create Alarm > Close

I've only exceeded that Network range when re-imaging a box or when I was under attack.

It's fairly easy to adjust the alarms to your system after it's been running for a few days as the alarms console shows you a graph of each and the red line after which the alarm would fire.

You probably also want a billing alarm.

[top-bar] Services > Billing > Billing preferences 
 > [check] Receive Billing Alerts > Save preferences
 > Manage billing alerts
 (note that billing alerts appear in the N.Virginia region, 
 regardless of where you have your instances)
[side-bar] Billing > Create Alarm > EstimatedCharges > ...

AMI Backups

The AMI is your best backup option. This is a snapshot of your disk and your instance details (micro, etc). From the AMI you can launch a new box and move over your IP in a few minutes. Ideally you'll create an AMI from your instance and take snapshots of your instance every 24 hours and keep the last week and perhaps a few older copies. We'll want to configure the creation of these snapshots to happen automatically.

We'll setup nightly Amazon EBS Snapshots of our instance. They can later be used as the basis for an AMI.

Something has to issue the nightly command, and that something must contain an unprotected copy of the credential that allows the snapshot to occur. First we'll create a constrained credential to reduce the risk of its exposure, then we'll piggyback on our existing local database backup script to kickoff the amazon snapshot. Your server is probably more likely to be attacked then your devbox.

IAM Credential

In the AWS Console, select IAM. If you haven't used this yet, you'll have zero groups, users and roles. First we'll create a group that can create snapshots and then we'll create a user and assign them that group. The backup script will connect as this user.

Create New Group
 Group Name: snapshot
 Policy Generator
   Effect: Allow
   AWS Service: Amazon EC2
   Actions: Create Snapshot
            Delete Snapshot
            Describe Snapshots
   ARN: *
   Add Statement
Create Group

Create New Users  
 User Name 1: snapshot
 [checked] generate and access key for each user 
 Download Credentials
 Close Window

  Groups > Add User to Groups > snapshot > Add to Groups


We'll use the following command from the AWS API Reference.

ec2-create-snapshot volume_id -d "Nightly Backup"

To find your volume_id, go the the AWS Console, select EC2, select Volumes from the sidebar, scroll right to the Attachment Information column which will show your WebServer instance, then scroll left and record the Volume ID.

First we have to setup the tools.

Download and unzip the latest tools. I used these. The latest link will be posted here. No install is required.

You have to have Java installed.

Run the following commands for a dos console to create your first snapshot.

SET JAVA_HOME=C:\Program Files (x86)\Java\jre1.8.0_191

SET EC2_HOME=C:\Amazon\ec2-api-tools-

SET EC2_URL=https://ec2.ca-central-1.amazonaws.com

ec2-create-snapshot vol-11111111 -d "Nightly Backup"

Run the following command to show existing snapshots.


You can filter that to only show the snapshots of a particular volume.

ec2-describe-snapshots --filter "volume-id=vol-11111111"

You can further restrict that to only show the snapshots of a particular volume that are tagged as "Nightly Backup", thus avoiding any ones you created manually.

ec2-describe-snapshots --filter "volume-id=vol-11111111" --filter "description=Nightly Backup"

Here's a windows script to create a snapshot and delete old nightly backup snapshots from a particular volume. You can call this from the script that you already setup to run nightly.

@echo off

SET JAVA_HOME=C:\Program Files (x86)\Java\jre1.8.0_191

SET EC2_HOME=C:\Amazon\ec2-api-tools-

SET EC2_URL=https://ec2.ca-central-1.amazonaws.com

SET EC2_VOLUME=vol-22222222

REM  This command lists all snapshots:
REM  ec2-describe-snapshots
REM  SNAPSHOT        snap-11111111   vol-22222222    completed       2014-04-12T20:29:58+0000        100%    333333333333    8       Nightly Backup
REM  SNAPSHOT        snap-11111111   vol-22222222    completed       2014-04-12T20:29:58+0000        100%    333333333333    8       Nightly Backup
REM  SNAPSHOT        snap-11111111   vol-22222222    completed       2014-04-12T20:29:58+0000        100%    333333333333    8       Created by CreateImage(i-44444444) for ami-55555555 from vol-22222222
REM  SNAPSHOT        snap-11111111   vol-33333333    completed       2014-04-12T20:29:58+0000        100%    333333333333    8       Nightly Backup
REM  This command lists only snapshots:
REM  - from the given volume
REM  - with the nightly backup tag
REM  - sorted from oldest to newest, http://ss64.com/nt/sort.html
REM  - note that whitespace above is actually a tab character so it counts as one space
REM  ec2-describe-snapshots --filter "volume-id=vol-22222222" --filter "description=Nightly Backup" | sort /R /+49

echo List interesting snapshots:
call ec2-describe-snapshots --filter "volume-id=%EC2_VOLUME%" --filter "description=Nightly Backup" | sort /R /+49

REM  This loop finds the selected snapshots that are older than 7 days:
REM  usebackq - use `` to delimit the command to be executed so that it can contain ""
REM  skip=7   - skip the first 7 rows, so we keep a week's worth of backups
REM  tokens=2 - select the 2nd column, delimited by spaces
REM  note: that the | must be escaped as ^|

echo Delete old snapshots:
FOR /F "usebackq skip=7 tokens=2" %%G IN (`ec2-describe-snapshots --filter "volume-id=%EC2_VOLUME%" --filter "description=Nightly Backup" ^| sort /R /+49`) DO (
  echo Delete %%G
  call ec2-delete-snapshot %%G

REM  Create the snapshot after we delete old snapshots so that our list won't contain any 
REM  pending entries that would mess up our assumptions about the poistion of the date at /+49

echo Create a snapshot:
call ec2-create-snapshot %EC2_VOLUME% -d "Nightly Backup"

Let's Encrypt

My original setup with lighttpd is here, and I've just done the minimum to get that working on Amazon Linux2 with Apache. There is probably an easier official way at this point, but I didn't experiment.

On the source box:

cd /tmp
sudo tar -zcvf letsencrypt.tar.gz /etc/letsencrypt
sudo tar -zcvf certbot.tar.gz /etc/lighttpd/ssl

On the destination box:

cd /tmp

sudo tar -xvzf letsencrypt.tar.gz
rm letsencrypt.tar.gz
sudo mv /tmp/etc/letsencrypt /etc

sudo tar -xvzf certbot.tar.gz
rm certbot.tar.gz
sudo mv /tmp/etc/lighttpd/ssl /etc/httpd/certbot
sudo rm -f /etc/httpd/certbot/intermediate.pem
sudo rm -f /etc/httpd/certbot/ssl.pem

On the source box:

rm -f /tmp/letsencrypt.tar.gz
rm -f /tmp/certbot.tar.gz 

On the destination box:

sudo vi /etc/httpd/certbot/certbot-renew

echo certbot-renew $(date)
/etc/httpd/certbot/certbot-auto renew --debug --quiet --post-hook "/etc/httpd/certbot/certbot-deploy"
sudo vi /etc/httpd/certbot/certbot-deploy

echo certbot-deploy $(date)

cp /etc/letsencrypt/live/holtstrom.com/cert.pem /etc/pki/tls/certs/localhost.crt
cp /etc/letsencrypt/live/holtstrom.com/privkey.pem /etc/pki/tls/private/localhost.key
cp /etc/letsencrypt/live/holtstrom.com/chain.pem /etc/pki/tls/certs/server-chain.crt

systemctl restart httpd
sudo vi /etc/httpd/conf.d/ssl.conf
# uncomment this line
SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt
sudo vi /etc/httpd/conf.d/virtualhost.conf
# add this line to each ssl block
SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt
# test the deploy
sudo /etc/httpd/certbot/certbot-deploy
# observe good permission and ownership
sudo ls -la /etc/pki/tls/certs/localhost.crt
-rw-r--r-- 1 root root

sudo ls -la /etc/pki/tls/private/localhost.key
-rw------- 1 root root 

sudo ls -la /etc/pki/tls/certs/server-chain.crt
-rw-r--r-- 1 root root 
# test the renew
sudo /etc/httpd/certbot/certbot-renew

# it failed, so try to replace with updated script
sudo rm -f /etc/httpd/certbot/certbot-auto
cd /etc/httpd/certbot
sudo wget https://dl.eff.org/certbot-auto
sudo chmod a+x certbot-auto

# try again
sudo /etc/httpd/certbot/certbot-renew

# still failure
# found advice here

# hack it
sudo vi /etc/httpd/certbot/certbot-auto

# replace this
elif [ -f /etc/issue ] && grep -iq "Amazon Linux" /etc/issue ; then

# with this
elif grep -i "Amazon Linux" /etc/issue > /dev/null 2>&1 || \
    grep 'cpe:.*:amazon_linux:2' /etc/os-release > /dev/null 2>&1; then

# try again
sudo /etc/httpd/certbot/certbot-renew

# successful output doesn't look like much, you can remove the --quiet
# to see that it doesn't attempt renewal because the certs aren't due yet
certbot-renew Wed May 15 03:14:58 UTC 2019
yum is /bin/yum
yum is hashed (/bin/yum)
Package 1:openssl-1.0.2k-16.amzn2.1.1.x86_64 already installed and latest version
Package ca-certificates-2018.2.22-70.0.amzn2.noarch already installed and latest version
Package python-devel-2.7.14-58.amzn2.0.4.x86_64 already installed and latest version
Package 1:mod_ssl-2.4.39-1.amzn2.0.1.x86_64 already installed and latest version

# Later, when there was work to do, I see it failed
# I have this in the log:
Attempting to renew cert (example.com) from /etc/letsencrypt/renewal/example.com.conf produced an unexpected error: [Errno 2] No such file or directory: '/var/www/lighttpd/example.com'.

# needed to fix the paths in this file
sudo vi /etc/letsencrypt/renewal/example.com.conf
# setup the chronjob
sudo crontab -e

38 3 * * * /etc/httpd/certbot/certbot-renew >> /etc/httpd/certbot/chron.log 2>&1

See Also

Summary of a secure LAMP stack


# restart apache
sudo systemctl restart httpd

# find out why apache failed to start
systemctl status httpd.service

# search apache for all request from that weren't forbidden
sudo grep /var/log/httpd/access_log | grep -v " 403 " | grep -v " 301 "

# given that alarm Threshold Crossed (07/01/20 01:01:00)
# open apache log and go to that time
sudo vi /var/log/httpd/access_log

# restart php
sudo systemctl restart php-fpm

# php error logs
sudo cat /var/log/php-fpm/www-error.log

# see your full list of servers
sudo netstat -plnt

# watch the db log
sudo tail -n 200 -f /var/log/mariadb/mariadb.log

# see what's actually going over the wire
sudo tcpdump port 80 -A | strings
{ "url": "https:\/\/holtstrom.com\/michael\/", "base": "\/michael", "domain": "holtstrom.com", "loggedin": false, "avatar": "", "render": "nothing", "doTracking": true, "trackingID": "UA-36983794-1", "description": "", "tmdbUrl": "http:\/\/www.themoviedb.org\/", "tmdbPoster": "http:\/\/image.tmdb.org\/t\/p\/w342" }