Thursday, February 3, 2011

Migrating to pv-grub kernels for kernel upgrades

After the release of the Ubuntu 10.04 LTS images that use the pv-grub kernel, there were some questions on what to do if you're running from an older AMI and want to take advantage of what pv-grub offers.

Heres a short list of how you might be affected:

  • If you are running an EBS root instance launched from or rebundled from an Official Ubuntu image of 10.04 LTS (Lucid Lynx) released 20110201.1 or later, 10.10 (Maverick Meerkat), Natty Narwhal, or later, then you do not need to do anything. You already are using pv-grub, and can simply apply software updates and reboot to get new kernel updates. You can stop reading now.
  • If you are running an instance-store instance that is not using a pv-grub kernel, there is nothing you can do. There is simply no way to change the kernel of an instance store instance.
  • If you are running an EBS-root based instance rebundled from a Ubuntu 9.10 (Karmic Koala) or older, then there is currently no supported path to getting kernel upgrades. There were no officially released EBS-root based Ubuntu images of 9.10, and with Karmic's end of life coming in April, there is not likely to be support for this new feature.
  • If you are running an EBS-root instance launched from or rebundled from an official Ubuntu release of 10.04, read on.

Updating a 10.04 based image basically entails 2 steps, setting up /boot/grub/menu.lst, and then modifying your instance to have a pv-grub kernel.


Step 1: installing grub-legacy-ec2.

If you launched or rebundled your instance from an Ubuntu 10.04 numbered 20101020 or earlier, you need to do this step. If you started from a release of 20101228 you can skip this step.

  • Apply software updates.

    Depending on how out of date you are, this might take a while.


    sudo apt-get update && sudo apt-get dist-upgrade

  • Install grub-legacy-ec2

    The 'grub-legacy-ec2' package is what the images use to manage /boot/grub/menu.lst. If you had used Ubuntu prior to the default selection of grub2, you will be familiar with how it works. grub-legacy-ec2 is basically just the menu.lst managing portion of the Ubuntu grub 0.97 package with some EC2 specifics thrown in.

    To get a functional /boot/grub/menu.lst, all you have to do is:


    sudo apt-get install grub-legacy-ec2


Step 2: modifying the instance to use pv-grub kernels

Now, your images should have a functional /boot/grub/menu.lst, and grub-legacy-ec2 should be properly installed such that future kernels will get automatically added and selected on reboot. However, you have to change your instance to boot using pv-grub rather than the old kernel aki that you originally started with.
  • Shut down the instance

    The best way to do this is probably to just issue '/sbin/poweroff' inside the instance. Alternatively, you could use the ec2 api tools, or do so from the AWS console.

    % sudo /sbin/poweroff

  • Modify the instance's kernel to be a pv-grub kernel

    Once the instance is moved to "stopped" state, you can modify its kernel to be a pv-grub kernel. The kernel you select depends on the arch and region. See the table below for selecting which you should use:
    regionarchaki id
    ap-southeast-1x86_64aki-11d5aa43
    ap-southeast-1i386aki-13d5aa41
    eu-west-1x86_64aki-4feec43b
    eu-west-1i386aki-4deec439
    us-east-1x86_64aki-427d952b
    us-east-1i386aki-407d9529
    us-west-1x86_64aki-9ba0f1de
    us-west-1i386aki-99a0f1dc

    Then, assuming you have $AKI represents the appropriate aki above, and $IID represents your instance id, and $REGION represents your region, you can update the instance and then start it with:

    $ ec2-modify-instance-attribute --region ${REGION} --kernel ${AKI} ${IID}
    $ ec2-start-instances --region ${REGION} ${IID}


Your instance will start with a new hostname/IP address, so get that out of describe-instances and ssh to your instance. You can check that it has worked by looking at /proc/cmdline. Your kernel command line should look something like this:


$ cat /proc/cmdline
root=UUID=7233f657-c156-48fe-8d60-31ae6400d0cf ro console=hvc0


In the future, your instance will now behave much more like a "normal server". If you apply software updates (apt-get dist-upgrade) and reboot, you'll boot into a fresh new kernel.

Getting ephemeral devices on EBS images

When you launch an instance in EC2, be it EBS root or instance-store, you are "entitled" to some amount of ephemeral storage. Ephemeral storage is basically just extra local disk that lives in the host machine where your instance will run. How much instance-store disk you are entitled to is described with the instance size descriptions.

The described amount of ephemeral storage is allocated to all instance-store instances. However, at register time of EBS-root instances the default "block-device-mapping" is set. Because an EBS-root AMI may be run with different instance-types, it is impossible to set the correct mapping for all scenarios. For the Ubuntu images, the default mapping only includes ephemeral0, which is present in all but t1.micro sizes.

So, if you want to "get all that you're paying for", you'll have to launch instances with different '--block-device-mapping' arguments to ec2-run-instances (or euca-run-instances). I've written a little utility named bdmapping to make that easier, and hold the knowledge.

You can use it to show the mappings for a given instance type:


$ ./bdmapping
--block-device-mapping=sdb=ephemeral0 --block-device-mapping=sdc=ephemeral1


Or, use it with command substitution when you launch an instance:

$ ec2-run-instances $(bdmapping -c m1.xlarge) --key mykey ami-c692ec94
#
# the above is the same as running:
# ec2-run-instances --block-device-mapping=sdb=ephemeral0 \
# --block-device-mapping=sdc=ephemeral1 --instance-type=m1.large \
# --key mykey ami-c692ec94


You can view or download 'bdmapping' at https://gist.github.com/809587


New Ubuntu 10.04 LTS images with pv-grub support

Yesterday I announced the availability of updated Ubuntu images on EC2 for 10.04 LTS (Lucid Lynx).

This post is mainly to mirror that one to a possibly wider audience, but also to go into more detail on the key change in these images:

You can now upgrade the kernel inside your 10.04 based EC2 image by running 'apt-get update && apt-get dist-upgrade && reboot!

Now, for a little more detail on that.

Around the time that 10.04 released, or shortly there after, Amazon released a new feature of EC2 as "Use Your Own Kernel with Amazon EC2". What this really meant was that there were now "kernels" on EC2 that were really versions of grub. By selecting this kernel, and providing a /boot/grub/menu.lst inside your image, the instance would be in control of the kernel booted. Previously, the loading of a kernel was done by the xen hypervisor, and could not be changed at all for a instance-store image, and only non-trivially for an EBS-root instance

Yes, you read that right, midway through the year of 2010 you were able to change the kernel that you were running in an EC2 instance with a software update and a reboot.

We took advantage of this support in our 10.10 images, but for many people, only LTS (Long Term Support) releases are interesting. So, to satisfy those people, we've brought the function into our 10.04 images.

If you're using our images, or have rebundled an image starting from them, I *strongly* suggest updating to the 20110201.1 images or anything later. You'll want to do that because

  • You can receive kernel upgrades as normal software upgrades to your running instance.
  • This is by far the most supportable route for upgrade from 10.04 to our next LTS (12.04). If your instance is not using the pv-grub kernel, and you want to upgrade to a newer release, you will have to upgrade, shutdown the instance, modify the associated kernel, and start up the instance. That is both more painful, and will result in larger downtime.

So, in short, grab our new images!

Tuesday, January 11, 2011

Failsafe and manual management of kernels on EC2

On 10.10 and later, Ubuntu images use the Amazon provided pv-grub to load kernels that live inside the image. The selected kernel is controlled by /boot/grub/menu.lst. This makes it possible to install a new kernel via 'dpkg -i' or 'apt-get dist-upgrade' and then reboot into the new kernel.

The file /boot/grub/menu.lst is managed by grub-legacy-ec2 package. The program 'update-grub-legacy-ec2' is called on installation of Ubuntu kernels through files that are installed in /etc/kernel/postinst.d and /etc/kernel/postrm.d.

By default, as with other Ubuntu systems, the kernel with the highest revision will be the behavior will be automatically selected as the default, and selected on the next boot. Because EC2 images is read-only, you may want to manually manage your selected kernel. This can be done by modifying /boot/grub/menu.lst to use the grub "fallback" code.

I'll launch an instance of the current released maverick (ami-ccf405a5 in us-east-1 ubuntu-maverick-10.10-i386-server-20101225). Then, on the instance, create hard links to the default kernel and ramdisk so even on apt removal, they'll stick around, and then change /boot/grub/menu.lst to use those kernels.


sudo ln /boot/vmlinuz-$(uname -r) /boot/vmlinuz-failsafe
sudo ln /boot/initrd.img-$(uname -r) /boot/initrd.img-failsafe


Then, copy the existing entry in /boot/grub/menu.lst to a new entry above the automatic section. I've changed/added:


# You can specify 'saved' instead of a number. In this case, the default entry
# is the entry saved with the command 'savedefault'.
# WARNING: If you are using dmraid do not use 'savedefault' or your
# array will desync and will not let you boot your system.
default saved

...<snip>...

# Put static boot stanzas before and/or after AUTOMAGIC KERNEL LIST

# this is the failsafe kernel, it will be '0' as it is the first
# entry in this file
title Failsafe kernel
root (hd0)
kernel /boot/vmlinuz-failsafe root=LABEL=uec-rootfs ro console=hvc0 FAILSAFE
initrd /boot/initrd.img-failsafe
savedefault

title Ubuntu 10.10, kernel 2.6.35-24-virtual
root (hd0)
kernel /boot/vmlinuz-2.6.35-24-virtual root=LABEL=uec-rootfs ro console=hvc0 TEST-KERNEL
initrd /boot/initrd.img-2.6.35-24-virtual
savedefault 0


And then update grub to store that the first kernel is the 'saved', which for grub 1 (or 0.97) modifies /boot/grub/default.


sudo grub-set-default 0
sudo reboot


Now, a reboot will boot into the failsafe kernel (which we can verify by checking /proc/cmdline) and see 'FAILSAFE'. Then, to test our "TEST-KERNEL", run:


sudo grub-set-default 1
sudo reboot


After this reboot, the system come up into "TEST-KERNEL" (per /proc/cmdline) but /boot/grub/default will contain '0', indicating that on subsequent boot, the FAILSAFE will run. In this way, if your kernel failed to boot all the way up, you can then just issue:


euca-reboot-instances i-15b77779


And you'll boot back into the FAILSAFE kernel.

The above basically allows you to manually manage your kernels while letting grub-legacy-ec2 still write entries to /boot/grub/menu.lst.

I chose to use hardlinks for the 'failsafe' kernels, so that even on dpkg removal, the files would still exist. Because the 10.10 Ubuntu kernels have the EC2 network and disk drivers built in, you'll still be able to boot even after a dpkg removal of the failsafe kernel or an errant 'rm -Rf /lib/modules/2*'

Friday, January 7, 2011

Using euca2ools rather than ec2-api-tools with EC2

The Ubuntu UEC Images that Ubuntu produces on EC2 are in every way fully supported, "Official Ubuntu". As with other official releases, access to source code for security and maintenance reasons affects our decisions on what is included.

In the UEC images, the most notable packages left out are 'ec2-api-tools' and 'ec2-ami-tools'. I personally use the ec2-api-tools and ec2-ami-tools quite frequently and Amazon has done a great job with them. However, the license and lack of source code prevents them from being in Ubuntu 'main'.

Fortunately
a.) There are packages made available in the Ubuntu 'multiverse' component.
b.) The euca2ools package is installed by default and provides an almost drop in replacement for the ec2-api-tools and ec2-ami-tools.

I think that many users of EC2 aren't aware of the euca2ools, so I'd like to give some information on how to use them here.

The ec2-api-tools use the SOAP interface and thus use the "EC2_CERT" and "EC2_PRIVATE_KEY". The euca2ools sit on top of the excellent boto project. Boto uses the AWS REST api, which means authentication is done with your "Access Key" and "Secret Key". As a result, configuration is a little different. (Note, bundling images, you still need the EC2_CERT and EC2_PRIVATE_KEY for encryption/signing).

Configuration for euca2ools can be done via environment variables (EC2_URL, EC2_ACCESS_KEY, EC2_SECRET_KEY, EC2_CERT, EC2_PRIVATE_KEY, S3_URL, EUCALYPTUS_CERT) or via config file. I personally prefer the configuration file approach.

Here is my ~/.eucarc that is configured to operate with the EC2 us-east-1 region.

CRED_D=${HOME}/creds/aws-smoser
EC2_REGION="${EC2_REGION:-us-east-1}"
EC2_CERT=${CRED_D}/cert.pem
EC2_PRIVATE_KEY=${CRED_D}/pk.pem
EC2_ACCESS_KEY=ABCDEFGHIJKLMNOPQRST
EC2_SECRET_KEY=UVWXYZ0123456789abcdefghijklmnopqrstuvwx
EC2_USER_ID=950047163771
EUCALYPTUS_CERT=/etc/ec2/amitools/cert-ec2.pem
EC2_URL=https://ec2.${EC2_REGION}.amazonaws.com
S3_URL=https://s3.amazonaws.com:443


Things to note above:
  • euca2ools sources the ~/.eucarc file with bash, and then reads out the values of EC2_REGION, EC2_CERT, EC2_PRIVATE_KEY, EC2_ACCESS_KEY, EC2_USER_ID, EC2_URL, S3_URL. This means that you use other bash functionality in the config file as I've done above with 'EC2_REGION'. This allows me to do something like:

    EC2_REGION=us-west-1 euca-describe-images

  • If there is no configuration file specified with '--config', then those values will be read from environment variables

  • Amazon's public certificate from the ami tools is included with euca2ools in ubuntu, and located in /etc/ec2/amitools/cert-ec2.pem

  • Many of the euca2ools commands will run significantly faster than the ec2-api-tools. The reason for slowness of the ec2-api-tools is their man java dependencies (please correct me if I'm wrong).
  • Your ~/.eucarc file contains credentials and therefore it should be protected with filesystem permissions (ie 'chmod go-r ~/.eucarc').
Hopefully this will make it easier for you to use euca2ools with EC2 on Ubuntu.

Tuesday, December 14, 2010

Ubuntu Natty Narwhal Cluster Compute Instances

Some time ago, Amazon announced two new instance types aimed at high performance computing. The new types differ from Amazon's previous offerings in that

  • They use Xen "hvm" (Hardware Virtualization Mode) rather than 'paravirtualization'
  • Only priviledged accounts can create images of the 'hvm' virtualization-type.

The result is that there are very few public images for cluster compute nodes, and up until today, there were no Ubuntu images.

I'm happy to announce that you can now run Official Ubuntu images on cluster compute instance types. From today forward we will be publishing daily builds of Natty Narwhal builds.

These images are identical to the other Ubuntu images. For AMI ids, you can browse the list at http://uec-images.ubuntu.com/server/natty/current/, or use more machine friendly data at http://uec-images.ubuntu.com/query.

There is one known bug (bug 690286) that prevents you from using ephemeral storage on the CC nodes.

If you've got a couple dollars burning a whole in your pocket, you can try one out with:

qurl="http://uec-images.ubuntu.com/query/"
ami_id=$(curl --silent "${qurl}/natty/server/daily.current.txt" |
    awk '-F\t' '$11 == "hvm" && $7 == "us-east-1" { print $8 }')
ec2-run-instances --key mykey --instance-type cc1.4xlarge "${ami_id}"

Tuesday, December 7, 2010

lvm resizing is easy

I have a local mirror of the ubuntu archive, using some scripts based on the Ubuntu Wiki. When I set up "/archive" on my local mirror, I used lvm. The reason for that was primarily so that I could use sbuild with lvm.

Since then, 2 things have happened:
  • sbuild has gained the ability to use aufs rather than LVM snapshots. The solution is much lighter weight, and doesn't require some lvm space sitting around waiting to be used.
  • The ubuntu archive has grown in size from ~ 250G to ~ 400G right now.

So, it was time for me to resize the filesystem that had my archive up to accommodate. For my own record, and possibly others, I thought I'd share what I did.


$ sudo pvscan
PV /dev/sdb VG smoser-vol1 lvm2 [931.51 GiB / 315.57 GiB free]
PV /dev/sda1 VG nelson lvm2 [148.77 GiB / 44.00 MiB free]
$ sudo lvscan
ACTIVE '/dev/smoser-vol1/smlv0' [585.94 GiB] inherit
ACTIVE '/dev/smoser-vol1/hardy_chroot-i386' [5.00 GiB] inherit
ACTIVE '/dev/smoser-vol1/lucid_chroot-i386' [5.00 GiB] inherit
ACTIVE '/dev/smoser-vol1/karmic_chroot-i386' [5.00 GiB] inherit
ACTIVE '/dev/smoser-vol1/karmic_chroot-amd64' [5.00 GiB] inherit
ACTIVE '/dev/smoser-vol1/lucid_chroot-amd64' [5.00 GiB] inherit
ACTIVE '/dev/smoser-vol1/hardy_chroot-amd64' [5.00 GiB] inherit
ACTIVE '/dev/nelson/root' [142.65 GiB] inherit
ACTIVE '/dev/nelson/swap_1' [6.07 GiB] inherit


I had 2 physical volumes, sdb and sda1. 'sdb' had my old sbuild snapshots on it, and also some free space. So, I deleted sbuild snapshots with:


$ sudo lvremove /dev/smoser-vol1/hardy_chroot-i386 \
/dev/smoser-vol1/lucid_chroot-i386 /dev/smoser-vol1/karmic_chroot-i386 \
/dev/smoser-vol1/karmic_chroot-amd64 /dev/smoser-vol1/lucid_chroot-amd64 \
/dev/smoser-vol1/hardy_chroot-amd64


Then, resized the 'smlv0' volume that had my '/archive' on it up to the largest that I could on that physical volume:


$ sudo vgdisplay smoser-vol1
VG Name smoser-vol1
System ID
Format lvm2
<snip>
VG Size 931.51 GiB
...
$ sudo lvresize /dev/smoser-vol1/smlv0 --size 931.51G
Rounding up size to full physical extent 931.51 GiB
Extending logical volume smlv0 to 931.51 GiB
Logical volume smlv0 successfully resized


Then, just resize the ext4 filesystem on that volume:

$ grep archive /proc/mounts
/dev/mapper/smoser--vol1-smlv0 /archive ext4 rw,relatime,barrier=1,data=ordered 0 0
$ sudo resize2fs /dev/mapper/smoser--vol1-smlv0
resize2fs 1.41.11 (14-Mar-2010)
Filesystem at /dev/mapper/smoser--vol1-smlv0 is mounted on /archive; on-line resizing required
old desc_blocks = 37, new_desc_blocks = 59
Performing an on-line resize of /dev/mapper/smoser--vol1-smlv0 to 244190208 (4k) blocks.

The filesystem on /dev/mapper/smoser--vol1-smlv0 is now 244190208 blocks long.


That last operation did take probably 30 minutes, but in the end, I now have:


$ df -h /archive/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/smoser--vol1-smlv0
917G 544G 327G 63% /archive