Building a CentOS 4 AMI: Part 3 of 2

First, I described how to prepare a system to be used to create the AMI.

Next, I outlined the process of creating a CentOS 4.7 AMI.

Finally, I'm going to show how to upload, register and start the instance. I shouldn't have to say, though I will, there are numerous variations on how to perform all of these tasks and getting your system and application in the cloud.

I'm my last installment I created the image itself on my server. After doing so, it needs to be broken up into smaller chunks to be uploaded and create a manifest. This is done with one command ec2-bundle-image.

ec2-bundle-image \
-k $EC2_PRIVATE_KEY \
--cert $EC2_CERT \
--user 123412341234 \
-i centos-4.7-i386.img \
-r i386 \
-B "ami=sda1,root=/dev/sda1,swap=sda3"

NOTE: If you do not have the EC2_PRIVATE_KEY or EC2_CERT shell variables then you didn't follow my instructions in my first post.

There are some obvious options here that are necessary. -B for block device maps and -r specifying the architecture but there are a few others that may be useful. --kernel to specify the kernel ID to boot with and --ramdisk in case one is needed.

NOTE: I should mention, at this point that if the system you're working on has a system clock that is too far off with Amazon's then you may receive the message

Client.InvalidSecurity: Request has expired

Best bet is to get your system's NTP service running, such as ntp(8), for details on synchronizing your systems clock.

Uploading the chucks...

You'll need your access key and secret key at this point for the -a and -s parameters below. Amazon S3 stores your files in what they call buckets. Every file in the bucket must have a unique name. I'm not 100% on this but the bucket name needs to be alphanumeric and can also use periods. Beyond that, I'm not sure. For more information on the full S3 API and it's capabilities and constraints, check out the Amazon S3 documentation pages.

OK. We're uploading. The chunkified image and manifest must be sent to a bucket (-b) in the S3 storage cloud.

ec2-upload-bundle \
-b my.first.centos.4.7 \
-m /tmp/centos-4.7-i386.img.manifest.xml \
-a "sssssshhhhhh" \
-s "itsre/allyr/eally/secret"

In this case, the full path to the files will have the form

my.first.centos.4.7/centos-4.7-i386.img.manifest.xml

Simple. Oh, if the files (they're not an AMI yet) aren't in use and you want to delete them check out the ec2-delete-bundle. There is help available online as well by using the --help command line option.

The files are in the storage cloud but the EC2 service needs to know about it. To do so, it needs the path to the manifest file. Registering is simple

ec2-register \
-K $EC2_PRIVATE_KEY \
-C $EC2_CERT \
my.first.centos.4.7/centos-4.7-i386.img.manifest.xml

Conversely, when you get bored with your image and want to eventually remove it from S3 (so you don't get charged every month) there is a ec2-unregister command. Again, like most or all EC2 commands in the toolkit, you may use the --help command line option.

A listing of all AMI's are available using the command below or by using Elasticfox, the web browser extension for Firefox that provides a GUI for

ec2-describe-images
IMAGE ami-89abcdef my.first.centos.4.7/centos-
4.7-i386.img.manifest.xml 495219933132 available private

The security group used will be your "default" policy. Since we're expecting to connect to the instance via SSH we ought to enable port 22/tcp.

ec2-authorize default -P tcp -p 22 -s 0.0.0.0/0

NOTE: Keep this command in mind. Depending on what you intend to do with your server in the cloud, additional ports may need to be opened.

Finally, we can start the AMI. By default, the m1.small instance type will be used providing the instance with a 1 core processor and roughly 1.7 GB of memory. You can start as many as you like.

ec2-run-instances \
-K $EC2_PRIVATE_KEY \
-C $EC2_CERT \
ami-89abcdef

Use the ec2-describe-instances to check the instance and once it's running the output will contain the IP address assigned to it. Earlier, we opened port 22 so starting a SSH connection now should work just fine. Log in and check out your server in the cloud.

If you find the m1.small lacking in memory or horsepower, other instance types with more cores and memory are available. I'd recommend starting with m1.small and if it isn't working, shut down the instance and bring it back up with a different type such as m1.large or c1.medium. There are described on the AWS web site.

All commands are well documented using the command line --help as well as on the AWS web site

That's all I have for now.

Building a CentOS 4 AMI: Part 2 of 2

In my last post, I outlined the preparation of a virtual system to be used to create the AMI. Everything done here is on the virtual machine and references the exact settings in the previous post.

NOTE: If you have not signed up for an AWS account and agreed to the EC2 and S3 terms of agreement you won't get too far here. Your own private key, certificate, and AWS user id is required to complete all of these steps.

Here, I will describe the actual image creation, uploading and then starting the AMI in the cloud. The steps I outline are exerpts from a script I use to automate all the steps. If you copy all the steps below to your own script, it ought to work.

Like all systems, whether virtual or real, requires a disk or disk image. I've opted to create a 2 GB image. I recognize that this is not the only way and you can accomplish this through other processes and means.

# Making the image and formatting the file system
dd if=/dev/zero of=centos-4.7-i386.img bs=1M count=2000
mke2fs -F -j centos-4.7-i386.img

# Mounting the image file
mount -o loop centos-4.7-i386.img /mnt

NOTE: You may create a larger image to suit your needs and it may be more economical than downloading things after the instance is started but remember that Amazon charges not just for uploads but for monthly storage. Creating an image larger than what you need may cost you.

Next step is creating some necessary directories and device files in the image. These processes are basic though provide a foundation for the software installation and running system later. More information can be had from any one of the many build your own distribution web sites.

# Creating the necessary directories
mkdir /mnt/dev
mkdir /mnt/proc
mkdir /mnt/etc

# Creating some minimal device files
for i in console null zero
do
/sbin/MAKEDEV -d /mnt/dev -x $i
done

More system basics here. The fstab file needs to be created and the image's proc file system is also necessary for the loading of software.

# Create fstab
cat <<EOFSTAB > /mnt/etc/fstab
dev/sda1 / ext3 defaults 1 1
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
/dev/sda2 /mnt ext3 defaults 1 2
/dev/sda3 swap swap defaults 0 0
EOFSTAB

# Mount the image proc now
mount -t proc none /mnt/proc

I'm using yum to perform the installation. It's a great tool for automatically resolving software dependencies and installing what's needed without the need for searching countless Google hit pages, cruising RPMfind.net or hunting for your distribution installation images.

You can expand on the yum.conf file as needed to customize your image. What I've included here are the basics but you may add the configuration for any other bit of software.

# Custom yum.conf for our image
cat <<EOCONF > /tmp/yumec2.conf
[main]
cachedir=/var/cache/yum
debuglevel=2
logfile=/var/log/yum.log
exlude=*-debuginfo
gpgcheck=0
obsoletes=1
reposdir=/dev/null

[base]
name=CentOS-4.7 - Base
mirrorlist=http://mirrorlist.centos.org/?release=4.7&arch=i386&repos=os
protect=1

[update]
name=CentOS-4.7 - Update
mirrorlist=http://mirrorlist.centos.org/?release=4.7&arch=i386&repos=updates
protect=1
EOCONF

NOTE: I would recommend for those who plan on using or tweaking their image to download all the packages you need from the distribution and create your own internal mirror site. Mirrors come and go and should your required version 1.2.3-5-EL4 get replaced and cause problems with package wyzzwg-4.3.2-1-EL4 then you may need to invest in Tylenol .

Finally, let's install some software. This may take a while depending on your internet connection. So, start now and go get some coffee or do your Christmas shopping... just not online. Go to the store. They're those big brick, stone or stucco buildings you see in real life alongside highways.

# Avoid a yum lock file error
mkdir -p /mnt/var/lock/rpm

# Install the base
yum -c /tmp/yumec2.conf --installroot=/mnt -y groupinstall Base

# Cleanup
yum -c /tmp/yumec2.conf --installroot=/mnt -y clean packages

Done? No errors? Das ist gut.

We're in the home stretch. We need to solve our access problems. I, as the rest of the world, should prefer nothing less than SSH for accessing systems. If you haven't generated a public-private key pair yet, please do so. This will store the key paid in /root/.ssh.

ssh-keygen -t dsa -C '' -N ''

The next few steps I've refined so our instance whenever started is ready for me to login. After all, this is like a freshly installed system and we can't expect it to know our password nor do we want some password stored out there in the S3-land.

# Make sure TLS is disabled
mv /mnt/lib/tls /mnt/lib/tls-disabled

# When this instance boots, the keys to the instance need to be there
if [ ! - d /mnt/root/.ssh ] ; then
mkdir -p /mnt/root/.ssh
chmod 700 /mnt/root/.ssh
fi

# Copy public key to instance
cp /root/.ssh/id_dsa.pub /mnt/root/.ssh/authorized_keys2
chmod 644 /mnt/root/.ssh/authorized_keys2

NOTE: No private keys installed. This is ultimately important. Like your password which does not exist in the /etc/shadow file on the image your private key should be kept out of the cloud.

The SSH daemon needs some tweaking to keep us from being shut out.

cat <<EOCONF >> /mnt/etc/ssh/sshd_config
UseDNS no
PermitRootLogin without-password
EOCONF

Our network configuration. Kinda pointless if we don't let the AMI have a network connection.

cat <<EOCONF > /mnt/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=localhost.localdomain
EOCONF
cat <<EOCONF > /mnt/etc/sysconfig/network-scripts/ifcfg-eth0
ONBOOT=yes
DEVICE=eth0
BOOTPROTO=dhcp
EOCONF

Let's not forget to unmount things. Hate to spend all this time creating a nice, neat image just to corrupt it later.

# Unmount image
sync
umount /mnt/proc
umount /mnt

Putting the ribbon around the image now. Here you can go crazy telling the instance on boot to use wget or curl to download your custom scripts, data, or whatever installing, running or just having it spin in circles. It's all completely customizable just by adding the scripts to the /etc/init.d director and appropriate links to the /etc/rc3.d directory. I believe run level 3 is the default.

NOTE: If you're going to put custom boot scripts in that leverage commands such as wget or curl then make sure they exist in the image. If they do not, just add some more yum command similar to what is above and you'll be all set.

Okay. This post ran a little long so my 2 post steps to building your AMI is going to spill over into a 3rd. Should give you time to customize your image and creates dreams of conquering the world.

Building a CentOS 4 AMI: Part 1 of 2

Building a new AMI for Amazon's Web Services takes as much time prepping your target system as does creating the AMI itself. There are system requirements, some, aren't taken care of depending on the distribution you're using.

Case in point, I'm building a CentOS 4 AMI. Why? Because CentOS / Red Hat 5 it all over and there are some applications that require older versions. ISOdx, the project I've been on for the past 6 years, grew up on Red Hat 3 and now runs on Red Hat and CentOS 4. I'd love to invest the time in getting the RPM's up to snuff so it'd run on v5 but I have bigger fish to fry.

What I settled on was CentOS 4.7. It's the most recent and release in the v4 release. After downloading the DVD copy of the distribution, I built a VMware server and installed almost everything, after all, you never know what you're going to need. There was some additional software and updates required. Here are some I've identified.

From http://aws.amazon.com/

• ec2-ami-tools.noarch.rpm
• ec2-api-tools.zip
  

From http://www.java.com/

• jre-6u7-linux-i586-rpm.bin

Located using http://RPMfind.net/

• tar-1.15.1-1.i386.rpm

From http://dev.centos.org/centos/4/testing/i386/RPMS

• ruby-1.8.5-1.el4.centos.i386.rpm
• ruby-libs-1.8.5-1.el4.centos.i386.rpm

I won't bore you with the downloading and installing. That ought to be straight forward enough. It will be evident from some paths where I unpacked things on my system. It may not be the perfect place but it works.

Next, I had to tweak my shell environment.

export EC2_HOME=/root/ec2-api-tools-1.3-24159
export JAVA_HOME=/usr/java/jre1.6.0_07

Yes. I'm logged in as root. Don't panic. It's a virtual machine that I may trash and I've been doing this since 1988. Yes, 1988. Older than some who may read this so you youngsters just settle down. Where's my spectacles?

Needless to say, you'll need a AWS account as well as the private key and cert. Having them on the system is very helpful and I stored them in /root/.amazon plus setup the following shell variables.

export EC2_PRIVATE_KEY=/root/.amazon/pk.pem
export EC2_CERT=/root/.amazon/cert.pem

And, finally we'll add to the path the bin directory for the EC2 tools.

export PATH=$PATH:$EC2_HOME/bin

Next post, which I hope occurs before the next turn of the century, I'll explain the a simple process of building the actual CentOS 4 AMI, uploading it, and eventually starting the thing.