NetBSD-Users archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Amazon EC2 build script (was: Re: EC2 images)



Dear all,

I wrote a set of shell scripts to help build and upload NetBSD images (see attached). I am presently using them to update the 5.1 images.

I am well aware that they are ugly :) Feel free to send me patches/ideas to improve them, and/or commit them to localsrc.

The set_ec2_env.sh.example script is a skeleton to set up environment for EC2 use. Edit it to fit your EC2 account.

What you can do:
- build_ec2_img.sh: compile and generate a NetBSD image, suitable for upload to EC2. Feel free to adapt it to your needs, I wrote a function for each step (building src, patching installed files, makefs, etc).

Please keep the various ec2 scripts, and only modify them when you know what you are doing. EC2 AMIs require some automation to fetch ssh keys, for example.

Usage:
# ./build_ec2_img.sh <path-to-NetBSD-src> <tmp>

- create_ec2_ami.sh:
  - list NetBSD AMIs by region (when called without arguments).
  - Upload the image and kernel built previously, and create the new AMI.

Usage:
$ ./create_ec2_ami.sh # Outputs NetBSD's AMI information
$ ./create_ec2_ami.sh <tmp> # Upload and create a NetBSD AMI

=== Shortcomings and possible improvements ===

- build_ec2_img.sh must be executed as root. Using mtree specs with makefs can avoid that, but as I am using ./build.sh install=$EC2DIR with specific sets, I did not find a way around this.

- scripts are ugly, and make extensive use of awk.

- i386 is not supported, but should be trivial to implement.

- there may be some unhandled corner cases with AMI creation, especially when communicating via EC2 API. Amazon did not design their tools in a shell friendly way. Errors can be reported either by command's return value, or out-of-band (in the output).

- the upload/creation of the AMI can take some time. It checks the SSH host key from the console output, and since it's buffered on Amazon's side upload will wait for them before starting AMI creation.

--
Jean-Yves Migeon
jeanyves.migeon%free.fr@localhost
#!/bin/sh

# This script is used to generate a correct NetBSD AMI image, suitable
# for uploading.

# The script must be run as root (eventually, we will not require
# this when fileutils are ready)

# XXX does not support i386 and PAE. Fix build + summary paths.

# User used to build the NetBSD's src tree
BUILD_USER="nobody"
BUILD_FLAGS="-j14 -U"

usage() {
        echo "Usage: ${0##*/} src [tmp]"
        echo "  src: path to NetBSD's src directory"
        echo "  tmp: temporary directory where EC2 image and kernel are built."
        exit 1

}

build_amd64() {
        cd $SRCDIR
        su "$BUILD_USER"                    \
        ./build.sh -O ../obj  -T ../tools   \
                   -D ../dest -R ../release \
                   $BUILD_FLAGS release
}

install_amd64() {
        cd $SRCDIR
        ./build.sh -O ../obj  -T ../tools    \
                   -D ../dest -R ../release  \
                   -V INSTALLSETS="base comp etc man tests" \
                   $BUILD_FLAGS install=${EC2DIR}

        cp "../obj/sys/arch/amd64/compile/XEN3_DOMU/netbsd" $KERNEL
}

patch_install() {
        cd $EC2DIR

        TMPFILE=$(mktemp /tmp/netbsd-ec2.XXXXXX)
        trap "rm -f $TMPFILE; exit 1" INT EXIT QUIT

        sed 's/rc_configured=NO/rc_configured=YES/g' \
            etc/rc.conf > $TMPFILE
        cp "$TMPFILE" etc/rc.conf

        # required to allow connection through EC2 SSH keys
        cat >> etc/rc.conf << 'EOF'

ec2_init=YES        # for setup of SSH key pair(s)
sshd=YES            # for remote shell access to instance
dhcpcd_flags="-t 0" # Wait for the DHCP server forever
EOF

        sed 's/^#PermitRootLogin.*/PermitRootLogin without-password/g' \
                etc/ssh/sshd_config > $TMPFILE
        cp "$TMPFILE" etc/ssh/sshd_config

        # DO NOT MODIFY THE FOLLOWING FILE WITHOUT PRIOR CONSENT.
        # Its output is parsed by custom scripts to handle SSH host
        # key fingerprints.
        cat > etc/rc.d/ec2_init << 'EOF'
#!/bin/sh
#
# PROVIDE: ec2_init
# REQUIRE: NETWORKING sshd
# BEFORE:  LOGIN

$_rc_subr_loaded . /etc/rc.subr

name="ec2_init"
rcvar=${name}
start_cmd="ec2_init"
stop_cmd=":"

METADATA_URL="http://169.254.169.254/latest/meta-data/";
SSH_KEY_URL="public-keys/0/openssh-key"
HOSTNAME_URL="hostname"

SSH_KEY_FILE="/root/.ssh/authorized_keys"

ec2_init()
{
        (
        umask 022
        # fetch the key pair from Amazon Web Services
        EC2_SSH_KEY=$(ftp -o - "${METADATA_URL}${SSH_KEY_URL}")

        if [ -n "$EC2_SSH_KEY" ]; then
                # A key pair is associated with this instance, add it
                # to root 'authorized_keys' file
                mkdir -p $(dirname "$SSH_KEY_FILE")
                touch "$SSH_KEY_FILE"
                cd $(dirname "$SSH_KEY_FILE")

                grep -q "$EC2_SSH_KEY" "$SSH_KEY_FILE"
                if [ $? -ne 0 ]; then
                        echo "ec2: Setting EC2 SSH key pair: ${EC2_SSH_KEY##* }"
                        echo "$EC2_SSH_KEY" >> "$SSH_KEY_FILE"
                fi
        fi

        # set hostname
        HOSTNAME=$(ftp -o - "${METADATA_URL}${HOSTNAME_URL}")
        echo "ec2: Setting EC2 hostname: ${HOSTNAME}"
        echo "$HOSTNAME" > /etc/myname
        hostname "$HOSTNAME"

        # Output the SSH host keys
        echo "ec2: ###########################################################"
        echo "ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----"
        for FILE in /etc/ssh/ssh_host_*_key.pub; do
                echo -n "ec2: "
                ssh-keygen -l -f "$FILE"
        done
        echo "ec2: -----END SSH HOST KEY FINGERPRINTS-----"
        echo "ec2: ###########################################################"
        )
}

load_rc_config $name
run_rc_command "$1"
EOF
        chmod 555 etc/rc.d/ec2_init

        # Some required files and directories
        # Add proc and kern directories
        mkdir grub kern proc
        # EC2 network configuration, via DHCP
        echo "dhcp" > etc/ifconfig.xennet0
        # Basic fstab entries
        cat > etc/fstab << 'EOF'
/dev/xbd1a /        ffs    rw 1 1
/dev/xbd0a /grub    ext2fs rw 2 2
kernfs     /kern    kernfs rw
ptyfs      /dev/pts ptyfs  rw
procfs     /proc    procfs rw
EOF

        cat > etc/motd << 'EOF'
Welcome to NetBSD - Amazon EC2 image!

This system is currently running a snapshot of a stable branch of the NetBSD
operating system, adapted for running on the Amazon EC2 infrastructure.

The environment is very similar to one provided within a typical Xen domU
installation. It contains a small, autonomous environment (including a
compiler toolchain) that you can run to build-up your own system.

The file-system is lightly populated so you have plenty of space to play with.
Should you need a src or pkgsrc tree, please use the "bootstrap" script found
under /usr to download them:

                /usr/bootstrap.sh [src|pkgsrc]

You are encouraged to test this image as thoroughly as possible.  Should you
encounter any problem, please report it back to the development team using the
send-pr(1) utility (requires a working MTA).  If yours is not properly set up,
use the web interface at: http://www.NetBSD.org/support/send-pr.html

Thank you for helping us test and improve NetBSD's quality!
EOF

        cat > usr/bootstrap.sh << 'EOF'
#! /bin/sh

URLROOT="http://ftp.netbsd.org/pub/";

usage() {
        echo "Usage: ${0##*/} [src|pkgsrc]"
        exit
}

if [ $# -ne 1 ]; then
        usage
fi

cd /tmp

case $1 in
src)
        echo "Downloading -current src..."
        ftp -a "$URLROOT/NetBSD/NetBSD-current/tar_files/src.tar.gz"
        progress -f src.tar.gz tar -xzpf - -C /usr/
        rm -f src.tar.gz
        ;;
pkgsrc)
        echo "Downloading latest pkgsrc stable release..."
        ftp -a "$URLROOT/pkgsrc/stable/pkgsrc.tar.gz"
        progress -f pkgsrc.tar.gz tar -xzpf - -C /usr/
        rm -f pkgsrc.tar.gz
        ;;
*)
        usage
        ;;
esac
EOF
        chmod 755 usr/bootstrap.sh
}

generate_img() {
        makefs -t ffs -B le -s 5g -M 5g -N $EC2DIR/etc/ \
            -o version=2 -f600000 $IMGFILE $EC2DIR
        gzip -9n $IMGFILE
}

summary() {
        echo "Creation of NetBSD Amazon EC2 image successful."
        echo "Kernel DOMU: $KERNEL"
        echo "Root image (compressed): $IMGFILE.gz"
}

if [ $(id -u) -ne 0 ]; then
        echo "This script must be run as root."
        exit 2
fi

case $# in
1)
        SRCDIR="$1"
        BUILDDIR="/tmp"
        ;;
2)
        SRCDIR="$1"
        BUILDDIR="$2"
        ;;
*)
        usage
        ;;
esac

set -e
ulimit -p 1024
ulimit -n 1024

EC2DIR="$BUILDDIR/ec2"
KERNEL="$BUILDDIR/netbsd"
IMGFILE="$BUILDDIR/NetBSD-AMI.img"

build_amd64
install_amd64
patch_install
generate_img
summary
#!/bin/sh

export EC2_PRIVATE_KEY=$HOME/.ec2/pk-XXXXX.pem
export EC2_CERT=$HOME/.ec2/cert-XXXX.pem
export EC2_SSH_KEY=$HOME/.ec2/id_rsa.ec2
export EC2_SSH_PUBKEY=$EC2_SSH_KEY.pub
export EC2_ACCOUNT_NUM=XXXX-XXXX-XXXX
export EC2_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXX
export EC2_SECRET_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
export EC2_SSH_KEYNAME=xxxxxxxxx
#!/bin/sh

# This script is used to:
# - query for current state of NetBSD AMIs
# - update an image disk and create an AMI remotely

# In update mode, it performs the following steps:
# - create volumes for Grub (1GiB, ext2), and NetBSD's root (5GiB, ffs)

# - upload the kernel and populate the Grub boot volume
# - upload the image and create NetBSD's root volume
# - snapshots the grub and / disks
# - output identifiers and some info, then exit.

export LC_ALL=C

ARCH="x86_64"
VERSION="5.1"
DESCRIPTION="built: $(date)"
# path to file that sets EC2 credentials (SSH keys, user's x509 certificate)
EC2CREDS="$HOME/set_ec2_env.sh"

# load the Amazon EC2 credentials
. "$EC2CREDS"

usage() {
        echo "Usage: ${0##*/} [region] [build-dir]"
        echo "Without parameter, returns per-region information about " \
                "NetBSD's AMIs"
        echo "  region: the EC2 region we want to create an AMI for"
        echo "  build-dir: path to the directory where EC2 files were built"
        exit 1
}

if [ $# -eq 0 ]; then
        # Assume we just want information about NetBSD AMIs
        # The name should contain NetBSD, at least...
        echo "REGION            ID              NAME            ARCH    ACCESS"
        for REGION in $(ec2-describe-regions | awk '{print $2}'); do
                ec2-describe-images --region $REGION -F name=*NetBSD* | (
                        while read TYPE ID NAME OWNER AVAIL PUBLIC ARCH VOID; do
                                # Skip all except IMAGEs
                                if [ "$TYPE" != "IMAGE" ]; then continue; fi

                                echo "$REGION   $ID     $NAME   $ARCH   $OWNER  
$PUBLIC"
                        done
                )
        done
        exit
fi

if [ $# -eq 2 ]; then
        REGION=$1
        BUILDDIR=$2
else
        usage
fi

KERNEL="$BUILDDIR/netbsd"
IMGFILE="$BUILDDIR/NetBSD-AMI.img.gz"

# We are in creation mode. Very simple preliminary checks
if [ ! -r $KERNEL ]; then
        echo "Kernel file '$KERNEL' does not exist!"
        exit 1
fi
if [ ! -r $IMGFILE ]; then
        echo "Image files '$IMGFILE' does not exist!"
        exit 1
fi

# Start a micro instance for the region we want.
# Any Amazon Linux instance should do.
AMIID=$(ec2-describe-images -o amazon --region $REGION \
            -F name=amzn-ami-* -F architecture=$ARCH |\
            awk 'NR == 1 {print $2}')

if [ "$AMIID" = "" ]; then
        # No Amazon Linux AMI for this region. Bail out.
        echo "There is no Amazon Linux AMI available for '$REGION'."
        echo "Consult http://wiki.netbsd.org/amazon_ec2/build_your_own_ami/";
        echo "to see how you can build one from scratch."
        exit 1
fi

# Run an instance
INSTANCE=$(ec2-run-instances $AMIID -k $EC2_SSH_KEYNAME \
        -t t1.micro --region $REGION)
INSTANCEID=$(echo $INSTANCE | awk '{print $6}')
ZONEID=$(echo $INSTANCE | awk '{print $13}')

echo "Creating: $ZONEID $AMIID  $INSTANCEID"

# Create the 1GiB, ext2fs volume. Will contain kernel and grub.conf files
VOL1ID=$(ec2-create-volume -s 1 --region $REGION -z $ZONEID | awk '{print $2}')
echo "Creating: volume $VOL1ID (1GiB), Grub configuration and kernel"

# Create the 5GiB, ffs volume. Will contain the root file-system.
VOL2ID=$(ec2-create-volume -s 5 --region $REGION -z $ZONEID | awk '{print $2}')
echo "Creating: volume $VOL2ID (5GiB), the root file-system"

# Wait for the instance.
COUNT=0
while true; do
        ADDRESS=$(ec2-describe-instances $INSTANCEID --region $REGION |\
            awk '/running/ {print $4}')
        if [ "$ADDRESS" != "" ]; then
                echo "$INSTANCEID running, address: $ADDRESS"
                break
        fi

        COUNT=$(($COUNT + 1))
        if [ $COUNT -gt 5 ]; then
                echo "Instance was not reported as 'started' after $COUNT " \
                        "tries. Leaving."
                exit 1
        fi

        echo "Waiting for instance $INSTANCEID..."
        sleep 30
done

# Now grab the correct SSH host key. This can take a while, the console
# output is buffered on EC2's side.
# By convention, the EC2 initialization script should prefix all its output
# lines by "ec2: "
TMPFILE1=$(mktemp /tmp/ec2-ssh-host-keys.XXXXXX)
TMPFILE2=$(mktemp /tmp/ec2-ssh-host-keys2.XXXXXX)
LOGFILE="/tmp/system.log.$INSTANCEID"
trap "rm -f $TMPFILE1 $TMPFILE2; exit 1" INT EXIT QUIT
COUNT=0
while true; do
        ec2-get-console-output $INSTANCEID --region $REGION |\
                sed -n "s/^ec2: //gp" > $TMPFILE1

                if grep -q "^-----END SSH HOST KEY FINGERPRINTS-----" \
                    $TMPFILE1; then
                        echo "EC2 instance reported:"
                        cat $TMPFILE1
                        break
                fi

        COUNT=$(($COUNT + 1))
        if [ $COUNT -gt 5 ]; then
                ec2-get-console-output $INSTANCEID --region $REGION > $LOGFILE
                echo "Failed contacting instance ($COUNT tries). The system "
                echo "log is available under:"
                echo "  $LOGFILE"
                echo "Connect to AWS Console to terminate and clean instance."
                exit 1
        fi

        echo "Waiting for console output of EC2 init script..."
        sleep 90
done

# Compare the fingerprints between EC2 console output and direct connection.
FP1=$(awk '$NF ~ /(RSA)/ {print $2}' $TMPFILE1)

ssh-keyscan -T60 -t rsa $ADDRESS 2>/dev/null > $TMPFILE2
FP2=$(ssh-keygen -l -f $TMPFILE2 | awk '{print $2}')

if [ "$FP1" != "$FP2" ]; then
        echo "SSH server fingerprint mismatch:"
        echo "EC2 Console reported:  '$FP1'"
        echo "Host keyscan reported: '$FP2'"
        echo "You are most likely trying to connect to an instance you do"
        echo "not own, or are subject to MITM. Continue? [yn]"
        read YESNO
        if [ "$YESNO" != "y" ]; then
                echo "Leaving..."
                exit 1
        fi
fi

ssh-keygen -R $ADDRESS
cat $TMPFILE2 >> ~/.ssh/known_hosts

# Attach 1GiB volume
echo "Attaching $VOL1ID to $INSTANCEID"
ec2-attach-volume $VOL1ID --region $REGION -i $INSTANCEID -d "/dev/sdf" \
        > /dev/null

# Attach 5GiB volume
echo "Attaching $VOL2ID to $INSTANCEID"
ec2-attach-volume $VOL2ID --region $REGION -i $INSTANCEID -d "/dev/sdg" \
        > /dev/null

# Wait for the 5GiB and 1GiB volume to be attached
COUNT=0
while true; do
        PRESENT=$(ssh -i $EC2_SSH_KEY ec2-user@$ADDRESS \
                "/bin/dmesg | /bin/grep -c 'xvd[fg]: unknown'")
        if [ "$PRESENT" -eq 2 ]; then
                break
        fi

        COUNT=$(($COUNT + 1))
        if [ $COUNT -gt 5 ]; then
                echo "Volume did not attach after $COUNT tries."
                echo "Connect to AWS Console to terminate and clean volumes."
                exit 1
        fi

        echo "Waiting for volumes $VOL1ID,$VOL2ID to be attached..."
        sleep 15
done

# Quick hack, we do not care about sudo tty safety checks. Avoids a lot
# of piping magic through sudo/ssh/fifo
ssh -t -i $EC2_SSH_KEY ec2-user@$ADDRESS \
        "sudo sed -i '/.*requiretty.*/d' /etc/sudoers"

# Format the 1GiB boot partition, mount, copy grub.conf + NetBSD kernel
echo "Creating the Grub boot partition"
ssh -i $EC2_SSH_KEY ec2-user@$ADDRESS \
        "sudo mkfs.ext3 /dev/xvdf; sudo mount /dev/xvdf /mnt;" \
        "sudo mkdir -p /mnt/boot/grub/"

ssh -i $EC2_SSH_KEY ec2-user@$ADDRESS \
        "sudo dd of=/mnt/boot/grub/menu.lst" << EOF
default=0
timeout=0
hiddenmenu

title NetBSD AMI
root (hd0)
kernel /boot/netbsd root=xbd1
EOF

echo "Copying $KERNEL to $INSTANCEID"
progress -f "$KERNEL" ssh -i $EC2_SSH_KEY ec2-user@$ADDRESS \
        "sudo dd of=/mnt/boot/netbsd"

# Overwrite the 5GiB volume with the prepared image
echo "Copying $IMGFILE to $INSTANCEID"
progress -f "$IMGFILE" ssh -i $EC2_SSH_KEY ec2-user@$ADDRESS \
        "gunzip | sudo dd of=/dev/xvdg"

# Everything done. sync and umount.
ssh -i $EC2_SSH_KEY ec2-user@$ADDRESS "sudo sync; sudo umount /mnt"

# Snapshot the volumes.
SNAP1ID=$(ec2-create-snapshot $VOL1ID --region $REGION -d "Boot volume" |\
        awk '{print $2}')
SNAP2ID=$(ec2-create-snapshot $VOL2ID --region $REGION -d "Root volume" |\
        awk '{print $2}')

# Wait for the snapshots to be complete
COUNT=0
while true; do
        COMPLETE=$(ec2-describe-snapshots --region $REGION $SNAP1ID $SNAP2ID |\
            grep -c 'completed')
        if [ "$COMPLETE" -eq 2 ]; then
                echo "Created snapshots:"
                echo "$SNAP1ID (1GiB -- boot volume)"
                echo "$SNAP2ID (5GiB -- root file-system)"
                break
        fi

        COUNT=$(($COUNT + 1))
        if [ $COUNT -gt 5 ]; then
                echo "One of $SNAP1ID,$SNAP2ID was not reported as " \
                    "completed after $COUNT tries. Leaving."
                exit 1
        fi

        echo "Waiting for snapshots $SNAP1ID,$SNAP2ID to be completed..."
        sleep 60
done

# Destroy the instance
ec2-terminate-instances $INSTANCEID --region $REGION

# We need the proper AKI to build the AMI. Our partitioning being:
# - 1GiB volume, contains grub.conf/menu.lst + boot kernel
# - 5GiB volume, the root partition
# we need an AKI where the volume is itself the boot partition, ie. "hd0" in
# Amazon terminology
AKIID=$(ec2-describe-images -o amazon --region $REGION \
                -F image-type=kernel \
                -F manifest-location=*pv-grub-hd0-* -F architecture=$ARCH |\
                awk 'NR == 1 {print $2}')

# Get the old NetBSD AMI ID
ONBAMIID=$(ec2-describe-images -o self --region $REGION \
            -F name=*NetBSD* -F architecture=$ARCH |\
            awk 'NR == 1 {print $2}')

# Register the new AMI
NEWAMIID=$(ec2-register -a $ARCH --kernel $AKIID --region $REGION \
    -b "/dev/sda1=$SNAP1ID" -b "/dev/sda2=$SNAP2ID" -n "NetBSD-$VERSION" \
    -d "$DESCRIPTION" | awk '{print $2}')

# Summary:
echo
echo "A new AMI has been created: $NEWAMIID"
echo "You are advised to check that it is bootable by running the"
echo "following command:"
echo
echo "  ec2-run-instances $NEWAMIID -t t1.micro --region $REGION"
echo
echo "then make the AMI public if it is bootable:"
echo "  ec2-modify-image-attribute $NEWAMIID --region $REGION -l -a all"
echo
echo "If the new image boots correctly, you can proceed to the deletion"
echo "of the build volumes:"
echo
for VOL in $VOL1ID $VOL2ID; do
        echo "  ec2-delete-volume $VOL --region $REGION"
done

if [ "$ONBAMIID" != "" ]; then
        # Obtain the snapshot IDs used to create the old AMI
        SNAPIDS=$(ec2-describe-images $ONBAMIID --region $REGION |\
                awk '/BLOCKDEVICEMAPPING/ {print $3}')
        echo
        echo "Do not forget to delete the old NetBSD AMI and its snapshots:"
        echo
        echo "  ec2-deregister $ONBAMIID --region $REGION"
        for SNAPID in $SNAPIDS; do
                echo "  ec2-delete-snapshot $SNAPID --region $REGION"
        done
fi


Home | Main Index | Thread Index | Old Index