Wednesday, February 6, 2013

Machine Binding in the Amazon Cloud

You might want to store a password, say for a keystore, on a machine and let software on that machine retrieve the password, but have that password resist attack by someone who can read the disk. I.e. we don't want to store the password in cleartext. This is called machine binding. It allows unattended login (UAL). A hardware solution (that I've never used) is TPM. Most software solutions create a machine fingerprint (say from the MAC address and the hard disk serial number) and use that to encrypt the password. But in a virtualized environment like AWS most of these features change during reboots, and those that don't are relatively static and shared by new images. The soluiton only provides security if an attacker who can read the disk at rest (like from a snapshot) can't also gather the machine fingerprint information used to encrypt your secrets.

TPM

There is some chatter about this on the amazon forum, but it appears there is not yet any support for this in AWS.

Meta Data

Unlike a baremetal system, EC2 instances provide access to additional metadata. Although you can only access instance metadata for an instance from the instance itself, the data is not protected by cryptographic methods. Instance metadata includes: security-groups, instance-id, ami-id.

User Data

pivotapp describes EC2 user data. When you launch an EC2 instance via the wizard on aws.amazon.com, the Advanced Instance Options dialog contains accepts User Data as a textarea or as a file. Alternatively when you create an as-create-launch-config via the CLI (for example for use by an auto-scaling-group) you may specify user data as a string value or as a file.

--user-data VALUE
    The data to make available to the instance.  The data must be specified
    as a string on the command line.  To upload data from a file see the
    '--user-data-file' option.

--user-data-file VALUE
    The name of the file containing user data for the instance.

User data is generically available to the launched instance. Amazon Linux AMI uses cloud-init for processing and executing supplied user-data. If the supplied user-data starts with "#!" it is executed as a script by the root user late the boot sequence.

The amazon docs explain that the cloud-init package is an open source application used to bootstrap Linux images. The version of cloud-init used in the Amazon Linux AMI enables you to specify actions that should happen to your instance at boot time. User data may be: gzip, mime, base64, or plain text that starts with "#!" or "#include" or "#cloud-config" or "#cloud-boothook".

"#cloud-boothook" data is stored in a file under /var/lib/cloud and executed immediately. It is the earliest "hook" available. It doesn't contain a mechanism to run only once. If run-once is desired it must implement that itself. (tests below show that is false. boothook is only run once) The boothook data is provided with the instance id in the environment variable "INSTANCE_ID".

"#!" scripts will be executed at an "rc.local" like point in the boot sequence on the first boot of a launched instance. For EBS backed instances that can be rebooted, it will not be executed on later boots.

User data can be modified.

EC2 > yourInstance > Actions > View/Change User Data > yourData > Yes, Change

User data can also be modified via the cli ec2-modify-instance-attribute --user-data

But user data "#!" scripts are only executed on first launch. You can change the behaviour of cloud-init by modifying its files after first boot.

The most promising of these from the prospective of passing in a machine secret is the coud-boothook.

Experiment

Starting with a machine that was originally launched without any user data.

$ sudo cat /var/lib/cloud/data/user-data.txt
[nothing displayed]

AWS > EC2 > myInstance > Actions > Stop > Yes, Stop

AWS > EC2 > myInstance > Actions > View/Change User Data > hello > Yes, Change

AWS > EC2 > myInstance > Actions > Start > Yes, Start

$ sudo cat /var/lib/cloud/data/user-data.txt
hello

The above indicates that the raw data is made available by file access at each reboot.

AWS > EC2 > myInstance > Actions > Stop > Yes, Stop

AWS > EC2 > myInstance > Actions > View/Change User Data

#cloud-boothook
#!/bin/sh
echo "hello" > /tmp/test.txt

> Yes, Change

AWS > EC2 > myInstance > Actions > Start > Yes, Start

$ sudo cat /var/lib/cloud/data/user-data.txt
#cloud-boothook
#!/bin/sh
echo "hello" > /tmp/test.txt

$ sudo cat /tmp/test.txt
cat: /tmp/test.txt: No such file or directory

AWS > EC2 > Launch > ... > User Data

#cloud-boothook
#!/bin/sh
echo "hello" > /tmp/test.txt

> ... Launch

$ sudo cat /var/lib/cloud/data/user-data.txt
#cloud-boothook
#!/bin/sh
echo "hello" > /tmp/test.txt

$ sudo cat /tmp/test.txt
hello

$ sudo ls -la /tmp/test.txt
-rw-r--r-- 1 root root 6 Feb  6 19:43 /tmp/test.txt

The above indicates that modified user data is copied to the instance at each reboot but it is only operated on during first boot.

$ sudo rm /tmp/test.txt
$ sudo shutdown -r now
$ sudo ls -la /tmp/test.txt
ls: cannot access /tmp/test.txt: No such file or directory

The above indicates that a boothook registerd a first launch doesn't execute at reboot.

AWS > EC2 > Launch > ... > User Data > zzxxcc > ... > Launch

$ env | grep zzxxcc
[nothing displayed]

$ sudo grep -I -r "zzxxcc" /var/lib/cloud/
/var/lib/cloud/data/user-data.txt:zzxxcc
/var/lib/cloud/data/cache/obj.pkl:S'zzxxcc'
/var/lib/cloud/data/user-data.txt.i:zzxxcc

So it seems that the best you can do is treat user data as a text file overwritten at the start of each boot sequence. To prevent an from attacker reading the user data secret you would have to remove it from disk and put it in volatile memory either as a shared memory blob or as a process available to other processes on the same box.

aws
{ "loggedin": false, "owner": false, "avatar": "", "render": "nothing", "trackingID": "UA-36983794-1", "description": "", "page": { "blogIds": [ 399 ] }, "domain": "holtstrom.com", "base": "\/michael", "url": "https:\/\/holtstrom.com\/michael\/", "frameworkFiles": "https:\/\/holtstrom.com\/michael\/_framework\/_files.4\/", "commonFiles": "https:\/\/holtstrom.com\/michael\/_common\/_files.3\/", "mediaFiles": "https:\/\/holtstrom.com\/michael\/media\/_files.3\/", "tmdbUrl": "http:\/\/www.themoviedb.org\/", "tmdbPoster": "http:\/\/image.tmdb.org\/t\/p\/w342" }