How to find the right Ubuntu AMI with tools

A while ago, I gave instructions on how you could find the right Ubuntu AMI. I promised to write about how you could accomplished programmaticly.

When we publish Ubuntu AMIs, we simultaneously publish machine consumable data to https://cloud-images.ubuntu.com/query/. The data there contains information so that you can:

• Find the latest ami of a given type (hvm/ebs/instance-store), arch, and region.
• Download the pristine image files

I think the format of the data is generally discernible, but there is some more information on the Ubuntu Wiki.

I've put an example client together. Here is some example usage:

• Launch the latest released image in us-east-1

$ euca-run-instances --instance-type t1.micro --key mykey $(ubuntu-ami)

• Open the Amazon EC2 console to launch the latest Oneiric daily build

You can now directly link to launching an image in Amazon EC2 console, combine that with this tool to open your browser to the right page.
$ ami=$(ubuntu-ami us-west-1 oneiric daily i386)
$ gnome-open https://console.aws.amazon.com/ec2/home?region=us-west-1#launchAmi=${ami}

• Download and extract the latest tarball for lucid
Here, 'pubname' is the recommended "publish name" of this AMI, which happens to correspond to the basename of the name on EC2, and "url" is a fully qualified url to http://cloud-images.ubuntu.com .
$ wget $(ubuntu-ami -f "%{url} -O %{pubname}.tar.gz")
$ uec-publish-tarball *.tar.gz my-ubuntu-images

I don't think I'll get this into 11.10, but I'd like to have something with this function into 12.04, and support launching AMIs directly through it for ease of use. I'd love to hear input on what you'd like a "ubuntu-run-instance" command to look like and do.

Updated AWS tools PPA for Ubuntu

I thought I would post a quick entry to spread the word about a ppa I've been maintaining with up to date versions of some of the AWS tools. It is named simply awstools. You can find it here.

Right now the ppa has the following packages:

• ec2-api-tools : Amazon's EC2 command line tools
• ec2-ami-tools : Amazon's EC2 AMI tools (rebundling and uploading images)
• iamcli : Identity Access Management (IAM) Command Line Toolkit
• rdscli : Command Line Toolkit for the Amazon Relational Database Service

To add this repository its as easy as:

$ sudo apt-add-repository ppa:awstools-dev/awstools
$ sudo apt-get update


Then, to install the newest available version of ec2-api-tools, do:

$ sudo apt-get install ec2-api-tools


I hope hope that is helpful.

How to find the right Ubuntu AMI on EC2

For anyone getting started on EC2, the first obstacle they're faced with is selecting an AMI (Amazon Machine Image). If your trying to find an Ubuntu image either via the Amazon Console or via the output of ec2-describe-images, you're likely to be overwhelmed. The success of Ubuntu as a platform and Ubuntu's commitment to refreshing AMIs means that there are literally thousands of of images on Amazon EC2 with "ubuntu"in their name. That, combined with and the lack of Ubuntu on the "Quick Start" menu makes this a non-trivial task.

The purpose of this post is to document how you can easily, quickly and safely find the Official Ubuntu AMIs on EC2 via the Amazon EC2 console or via your web browser.

Some General Ubuntu Information

You already may be aware of these items, but I want to point them out for those who are just getting started with Ubuntu or EC2.

• Ubuntu releases every 6 months. Each release has a version number and a codename. The most important thing to note here is that every 2 years a LTS (Long Term Support) release is made. If you want stability and support for 5 years, select an LTS release. If you want the newest packages, select the most recent release. See the wikipedia entry for more information.
  
• At the time of this writing, there are 5 "regions" in Amazon EC2. Each region represents a geographical location. Each region has its own AMI ids. Inside each region there are 2 architectures (x86_64, i386) and 2 "root store" types (EBS or instance). That means that for each build Ubuntu releases, we generate 20 ami ids.

Easiest: Find AMIs From Your Web Browser

You can choose your interface for selecting images. Go to either:

• http://cloud.ubuntu.com/ami
At the bottom of this page, you can select the region, release, arch or root-store. You're only shown the most recent releases here. When you've made your selection, you can copy and paste the ami number, or just click on it to go right to the EC2 console launch page for that AMI. or
• https://cloud-images.ubuntu.com/server/releases/
• Select Your release by number or code-name
• Select 'release/': We keep historical builds around for debugging, but the 'release/' directory will always be the latest.
• Select your AMI from the table and click to launch in the console or copy and paste a command line.

Search through the Amazon EC2 Console

The EC2 Console is a graphical way to sort through AMIs and select one to launch. To Launch an Official Ubuntu Image here, follow the steps below.

• Select the region you want in the top left, under 'Navigation'
Example: "Us East (Virginia)"
• Click "AMIs"
Do not click "Launch Instance", see note below
• for 'Viewing', select "All Images"
• Limit the results to Ubuntu Stable Release images by typing ubuntu-images/
You should expand the 'AMI Name' field as wide as possible (maybe shrink the others).
• Limit the results to a specific release by appending '.*'.
For example: ubuntu-images/.*10.04
• Limit the results to a given arch by appending '.*i386' or '.*amd64'
Note: If you want to run a m1.small or c1.medium, you need 'i386'. If you want to run a t1.micro, you will need to select an 'ebs' image.
• Sort your results by AMI Name and make selection
By sorting by AMI name, you can more easily see the newest AMI for a given set. Each AMI ends with a number in the format YYYYMMDD (year,month,day). You want the most recent one.
• Verify the Owner is 099720109477!
  Any user can register an AMI under any name. Nothing prevents a malicious user from registering an AMI that would match the search above. So, in order to be safe, you need to verify that the owner of the ami is '099720109477'.
  If "Owner" is not a column for you, click "Show/Hide" at the top right and select "Owner" to be shown.
  
• Click on the AMI name, then Click 'Launch'

Notes

• HTTPS Access
Of the options above, right now https://cloud-images.ubuntu.com/server/releases/ is the only one that provides data over https. This may be important to you if you are concerned about potential "Man in the Middle" attacks when finding a AMI id. I've requested Ahmed [kim0 in irc] to support https access to https://cloud.ubuntu.com/ami .
• Web Console 'Launch Instance' dialog
I saw no way in the 'Launch Instance' dialog to see the Owner ID. Because if this, I suggest not using that dialog to find "Community AMIs". There is simply no way you can reliably know who the owner of the image is from within the console. For advanced users, I will blog sometime soon on a way to find AMIs programmatically [Hint].

Getting a larger root volume on a cluster compute instance

At this point, all of the Ubuntu EBS root images are 8GB. This is nice and small. I previously covered how you could increase the size of those disks.

On Cluster Compute instances, its a bit more difficult. The cluster compute instances have their root filesystem in a partition inside of the disk attached. Thats all well and good, and most likely a partitioned disk is more familiar to you than one that is not partitioned. The cluster compute images have grub2 installed in the MBR of that disk.

The problem with the partition on the disk is that you can no longer simply launch the instance with a larger root volume and then 'resize2fs /dev/sda1'. This is because the kernel won't re-read the partition table of a disk until all of its partitions are unmounted. For the disk that holds your root partition, that means you basically have to reboot after a change.

To avoid that waste of precious time, we've included a utility called 'growpart' inside of the initramfs on the Ubuntu images. It is invoked by the 'cloud-initramfs-growroot' package. This code runs before the root fileystem is busy, so the request for the kernel to re-read the partition table will work without requiring a reboot. To try it out, do:


# us-east-1 ami-1cad5275 hvm/ubuntu-natty-11.04-amd64-server-20110426
$ ec2-run-instances --region us-east-1 --instance-type cc1.4xlarge \
--block-device-mapping /dev/sda1=:20 ami-1cad5275


When you get to the instance, you'll have a 20G filesystem on /. And, if you're interested enough to look in the console output, you'll see something like:

GROWROOT: CHANGED: partition=1 start=16065 old: size=16755795 end=16771860 new: size=41913585,end=41929650