UDS-N Call for participation

The Ubuntu Developer Summit (UDS) is the event in which the Ubuntu community discusses and plans the upcoming Ubuntu release. UDS Natty begins Monday, October 25th (this monday) outside of Orlando, FL, USA. If you're in the Orlando area, this event is free, and open to anyone.

If you've not yet made plans to attend physically, then its unlikely that you'll be present in the rooms. However, the Canonical IS does an outstanding job of making remote participation possible. For more information, on how you can participate remotely, read the Remote Participation document. In short, you join an IRC channel, listen to a live high quality audio stream from the room, and can see edits to a live gobby document.

The comprehensive list of all sessions is available through the summit schedule, or a filtered list of only the 'Cloud track'.

Some of the sessions that I personally am interested (ok, interested *and* leading) in are:

• cloud-server-n-cloud-images: Here we'll discuss how we can make the Ubuntu Images on EC2 or UEC better. If you've used them, then we're interested in your feedback.
• cloud-server-n-image-rebundle: Here we'll discuss possible improvements in the rebundling process. Ie, how can you take one of the Ubuntu images, and customize it and turn it into your own AMI. This is a common operation, and unfortunately, one that has some sticking points. If you have other ideas for "cloud utilities" this is the place to bring them up.
• cloud-server-n-desktop-images: Ubuntu has made desktop images available on ec2 in a "tech preview" like manner for 2 releases. We've not fully supported these images, but a version of the cloud images with a remote-desktop interface is a common request. If you want to see what it might look like, try out the free Edubuntu demo using NX
• cloud-server-n-cloud-init: Here we'll discuss improvements to cloud-init or cloud-config. If you've customized images via user-data, then you've used cloud-init. How can we make it better ?
• cloud-server-n-ubuntu-trial: We threw together awstrial fairly quickly, and with the 10.10 release, we allowed anyone interested to Try Ubuntu Free. We got a great response and we see lots of ways this can be used. Come and let us know what you think

There are loads of other interesting sessions. If 'Server' or 'Cloud' isn't your thing, there are other tracks that might peek your interest.

How to rebundle Ubuntu 10.10 (Maverick Meerkat) EBS root

There was a post on the ec2ubuntu list regarding a problem the poster had with re-bundling a Ubuntu 10.10 instance. The poster followed another blog entry. I responded to the post, but figured that a blog entry might reach a larger audience.

The simple summary of my long winded post is this:

To rebundle an EBS root image, use ec2-create-image.

If you've launched one of the Official Ubuntu 10.10 Images and modified it a bit, the best way to create a new AMI with the modifications in it is to:

1.) stop the instance (do not terminate it)
2.) "create-image"
3.) wait for your new AMI to become available and run your new instance

Note, that all of the above steps can be done from the EC2 console as well as by using the command line tools. I don't think I previously realised how nice this api call is.

The reason that the title of this post includes 'Ubuntu 10.10' rather than just "Ubuntu", is that releases of 10.04 and earlier do not utilize pv-grub. If you're re-bundling one of those images, you have to take additional steps to get a new kernel upgrade. You really should take those steps, as kernel upgrades include important security fixes, and newly created AMIs should always be created with the most recent kernel available.

Try out Ubuntu Server 10.10 on EC2 for FREE!

I'm mostly disappointed that it wasn't my idea. Dustin deserves all the credit. The implementation (awstrial) was a fairly straightforward programming exercise. I don't mean at all to discount the work of the others who contributed to the awstrial project, and I had a blast in my first django experience, but the idea was the brilliance.

What idea ? The idea to allow anyone to try out Ubuntu 10.10 on EC2 for free for 55 minutes. No hardware is needed. While you're waiting for your desktop or server ISO to download, you can give a test drive to the server. You'll have 170Gb disk, 2G memory, and a very high speed internet connection with local LAN access to Ubuntu mirrors.

You do not need an AWS account or a credit card.

Here is how it works:

• Sometime on Sunday, October 10, 2010 Ubuntu 10.10 (Maverick Meerkat) will be released.
• At that point, and a limited time following, you'll be able to go to 10.cloud.ubuntu.com and launch an instance.
• wait 3 minutes or less
• ssh to the instance
• do something
• 55 minutes after launch, your instance will be terminated

In order to take part in this, you'll need to have a launchpad.net or Ubuntu Single Singon account. If you don't have one, then create one. If you're reading this, you probably have ssh keys in a file called ~/.ssh/id_rsa.pub, so go to https://launchpad.net/~YOUR_LOGIN/+editsshkeys and paste those keys in.

Now, back to the 'do something' item above. What should you do?

Heres some suggestions, but I'm certain that you're more creative than I am.

• Check if a bug you opened (or were annoyed by) in 10.04 LTS is still present. If it was fixed, then make sure the bug in launchpad is marked 'Fix Released'.
• Take Postgres 9.0for a spin, thanks to Martin Pitt's PPA builds
• run 'rm -Rf /' just to see what happens.
• Find something that is broken. Open a bug, using 'ubuntu-bug'.
• Hack on awstrial

Whatever you do, I hope you enjoy taking the "Official Ubuntu Image" for a spin. Do something cool, blog about it, tell people how easy it was.

Using Policies in AWS Identity and Access Management

For a project I'm working on here at work, I wanted to create a AWS user that could only launch instances, but could not write to S3, query SDB

{
 "Statement":[ {
    "Effect":"Allow",
    "Action":["ec2:RunInstances","ec2:RebootInstances",
              "ec2:GetConsoleOutput", "ec2:DescribeInstances" ],
    "Resource":"*"
  }, {
    "Effect":"Allow",
    "Action":["ec2:StopInstances","ec2:StartInstances",
              "ec2:TerminateInstances"],
    "Resource":"*"
  }, {
    "Effect":"Allow",
    "Action":["ec2:DescribeImages"],
    "Resource":"*"
  }, {
    "Effect":"Deny",
    "NotAction":["ec2:*"],
    "Resource":"*"
  } ]
}

I had hoped that I could allow this user to create his own security groups and keypairs (for launching instances with 'ec2-run-instances --key', and that I could also allow him to modify or delete those items as well. Unfortunately, I was not able to figure out how to do this. What I had hoped I could do was something like was something like:

{
  "Effect":"Allow",
  "Action":["ec2:*SecurityGroup*"],
  "Condition" :  {
     "StringLike": {
        "ec2:groupName":"foouser*"
     }
  },
  "Resource":"*"
},
{
  "Effect":"Allow",
  "Action":["ec2:*KeyPair*"],
  "Condition" :  {
     "StringLike": {
        "ec2:keyName":"foouser*"
     }
  },
  "Resource":"*"
}

The 'keyName' is an atribute of the [Add,Delete,Describe]KeyPair API calls, and 'groupName' is an attribute of the [Add,Delete]SecurityGroups and AuthorizeSecurityGroupIngress API calls as described in the EC2 API

My goal was to limit the user ('foouser') to manipulating SecurityGroups or KeyPairs that begain with 'foouser'. This would be a clear indication to other users of the account when they came across them.

However, the 'Condition' syntax isn't as "open" as that (couldn't think of a better term for than 'open'). I can think of reasons as to why it would be difficult or undesireable to make Conditions function like I wanted, but it would have been nice.

The IAM EC2 documentation indicates that EC2 only supports the following Condition types: aws:CurrentTime aws:EpochTime aws:SecureTransport aws:SourceIp and aws:UserAgent.

It seems to me that SecurityGroups and keypairs are an essential piece of Using EC2, but it seems like these are stuck at the account level, with no ability to limit them at the user level.

Another thing that I would like to do is give the user the ability to launch / stop / start / terminate her own instances, but not other users of the account. If I truly try to use IAM to split up my account usage, say with 'Development' and 'Production' users or groups, this is essential. When I use the 'Development' user I want to be protected from an accidental reboot or terminate of a 'Production' instance.

For example, I test our Official Ubuntu images. The testing scripts launch several instances. While they're running, I'll often be doing development, and also have an instance running. I would like my 'development' work to not accidentally terminate (or otherwise affect) my test runs. As it is right now, a 'euca-describe-instances' will show me all instances in either account, just waiting for me to copy and paste wrong and terminate one.

It is quite possible that I've missed something, if so, please let me know.

Playing with AWS Access Identity Management

Today I finally found some time to play with AWS Identity and Access Management. If you hadn't seen the announcement, or aren't familiar, the IAM tools basically allow you to create, manage, and limit multiple AWS accounts under a single account.

There are 2 reasons that immediately spring to mind for when you should use this:

• If you're sharing an single AWS account between multiple people, then using this is almost required.
• you want to use some AWS facility from inside an EC2 instance. Here, it just seems scary to put the entire keys to your account onto a remote machine.

To get started, I walked through the Getting Started guide. I downloaded the IAM Tools, and set them up as described. On Ubuntu, that consisted of:


$ wget http://awsiammedia.s3.amazonaws.com/public/tools/cli/latest/IAMCli.zip
$ unzip IAMCli.zip
$ vi my-account-creds.txt
$ cat my-account-creds.txt
AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST
AWSSecretKey=zyxwvutsrqponmlkjihgfedcbazyxwvutsrqponm
$ export AWS_CREDENTIAL_FILE=my-account-creds.txt
$ export AWS_IAM_HOME=$PWD/IAMCli
$ export PATH=$AWS_IAM_HOME:$PATH JAVA_HOME=/usr


Then, I created a user and admin group as described in the guide:

$ iam-groupcreate -g admins
$ cat AdminGroupPolicy.txt

{
   "Statement":[{
      "Effect":"Allow",
      "Action":"*",
      "Resource":"*"
      }
   ]
}


$ iam-groupuploadpolicy -g admins -p AdminGroupPolicy -f AdminGroupPolicy.txt
$ iam-usercreate -u smoser -g Admins -k -v
TSRQPONMLKJIHGFEDCBA
mnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz
arn:aws:iam::950047163771:user/smoser
AHDAIABBGZ3Q31XMUE4AN


The first line of the iam-createuser output is the AWSAccessKeyId, and the second is the AWSSecretKey. I quickly added those to a file my-user-creds.txt as shown above, and set AWS_CREDENTIAL_FILE=my-user-creds.txt .

That's all it took. Now I have a set of credentials that I can use, and if they're lost or stolen, I can revoke them with the (now safely locked up) account credentials.

At this point, I could use the euca2ools with a config file like:

$ cat my-user-eucarc
AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST
AWSSecretKey=mnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz
EC2_SECRET_KEY=${AWSSecretKey}
EC2_ACCESS_KEY=${AWSAccessKeyId}
EC2_USER_ID=950047163771
EC2_URL=https://ec2.amazonaws.com
S3_URL=https://s3.amazonaws.com:443
EC2_CERT=/etc/ec2/cert-ec2.pem
$ euca-describe-instances --config my-user-eucarc


Additionally, the above will also suffice as an AWS_CREDENTIAL_FILE for the iam-tools.

Thats great, but for one reason or another, I end up using the ec2-api-tools for a large amount of my work. Those tools require a private key and certificate. So, I had to go about creating one for my new user. Thanks to Nate@AWS in an EC2 Forum Post, that was easy also.


$ openssl version
OpenSSL 0.9.8o 01 Jun 2010
$ openssl genrsa 1024 > pk.pem
$ openssl req -new -x509 -nodes -sha1 -days 730 -key pk.pem -out cert.pem
# follow prompts here
$ iam-useraddcert -u smoser -f cert.pem
$ export EC2_PRIVATE_KEY=$PWD/pk.pem EC2_CERT=$PWD/cert.pem
$ ec2-describe-instances ...


Now my ec2-api-tools are functional. I have to admit to not completely understanding the implications of self signing a certificate here and uploading it. However, as I was authenticated to do the upload (via https and the given credentials) and only my user will use that signing key, I don't know what harm there could be.

Now I have the following TODOs:

• Post about creating an IAM Policy
  
• package the IAM tools for Ubuntu multiverse
  

Updates:

• 2010-10-12: update case in AWS_IAM_HOME string ('IamCli' -> 'IAMCli')
  

running Ubuntu on an Amazon "micro" instance

Amazon announced today a new instance type. The "micro" instance type (t1.micro) has 613 MB of memory and can be used with AMIs of either x86 or x86_64. The cost for either is only $0.02 per hour.

That means that you can try out Ubuntu on EC2 for 2 measly cents. The official images of Ubuntu 10.04 LTS (Lucid Lynx) are perfect for this. If you're more adventurous and want to try out 10.10, please try out a daily build.

I think this smaller instance size really makes EC2 available to a lot more people. The minimum price for running an instance for a month goes from something like $70 (31 * 24 * 0.095) to less than $15. If you add to that the fact that you can shut down the instance and not be charged for unused CPU time, then start it back up when you need it, it can be really cheap. If you happen to need more CPU power, you can even shut it down, and start it back up with more resources!

So what are you waiting for!

$ MY_LAUNCHPAD_ID="smoser"
$ printf "#cloud-config\n%s\n" "ssh_import_id: [${MY_LAUNCHPAD_ID}]" > user-data.txt
$ ec2-run-instances --region us-east-1 --user-data-file=user-data.txt ami-1234de7b


A few things to note:

• Bug 634102 has to be worked around or you will not be able to reboot your instance. There are cloud-init debs available in my personal PPA with fixes. 10.10 won't have the issue, and I'll work on getting the fix backported to 10.04.  You can easily fix that by running:
  
arch=$(uname -m)
[ "$arch" = "x86_64" ] && ephd=/dev/sdb || ephd=/dev/sda2
sudo sed -i.dist "\,${ephd},s,^,#," /etc/fstab

  
• One interesting feature of this instance type is that it will run images that are x86 or x86_64 arch. Previously each instance type only ran a single arch.
• Above, I launched the instance with cloud-config syntax that will pull in my keys from launchpad rather than using public ssh keys stored in EC2. Thats why I didn't need to pass '--key <mykey>'

edits

• I fixed the ami id above, I had listed a instance-store image.  Also, a thing to note is that with t1.micro instances you *have* to use EBS images.
• I fixed the 'sed' command above, and fixed the price for m1.small (0.095, not 0.95)

Verify SSH Keys on EC2 Instances

Like every server, every EC2 instance should have a unique ssh fingerprint. On "real servers" this fingerprint is generated at first installation of the openssh-server package. On EC2, instead, it is done on first boot of an instance. This is because each instance is a byte for byte copy of a registered image.

What this means to you, is that when you launch an instance and then connect with ssh, you'll see something like:


$ ssh -F /tmp/smoser/foo ec2-67-202-47-56.compute-1.amazonaws.com
The authenticity of host 'ec2-67-202-47-56.compute-1.amazonaws.com (67.202.47.56)' can't be established.
RSA key fingerprint is f1:40:a7:4e:0f:28:8d:12:21:59:f1:ff:03:5f:63:54.
Are you sure you want to continue connecting (yes/no)?


The ssh client is informing you that you are connecting to a host that you do not have ssh keys stored for. In short, it cannot confirm the identity of 'ec2-67-202-47-56'. There could be a "Man in the Middle" who is attempting to trick you. Just as with "real servers", you should identify that remote system via an out of band method. To do this outside of EC2, you might call a hosting provider up and ask them to verify the fingerprint that you see. On EC2, the only out of band transport is the ec2 console.

In order to provide you with the fingerprint that you need, the ssh fingerprint is written to the console when it is booted. You can see this with ec2-get-console-output.

As seen with the results of Eric's poll on alestic.com, this is a very little known or used piece of information. Over 50% of alestic.com voters have "never verified the fingerprint".


$ euca-get-console-output i-72bf1518 | grep ^ec2:
ec2:
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: 2048 f1:40:a7:4e:0f:28:8d:12:21:59:f1:ff:03:5f:63:54 /etc/ssh/ssh_host_rsa_key.pub (RSA)
ec2: 1024 28:f3:ef:a6:86:05:50:33:76:16:24:32:56:14:06:13 /etc/ssh/ssh_host_dsa_key.pub (DSA)
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################


Note that the ssh fingerprint reported on the console matches the one that ssh client asked me to confirm above. So, I now know that the host I've connected to is the one that I just started.

Putting this all together, lets say you have booted a new EC2 instance, with instance-id i-72bf1518 and hostname ec2-67-202-47-56.compute-1.amazonaws.com.

First we will use ssh-keyscan to get the fingerprint that is being reported by the remote host, and store that in a shell variable 'fp'


$ iid=i-72bf1518
$ ihost=ec2-67-202-47-56.compute-1.amazonaws.com

$ ssh-keyscan ${ihost} 2>/dev/null > ${iid}.keys
$ ssh-keygen -lf ${iid}.keys > ${iid}.fprint
$ read length fp hostname id < ${iid}.fprint
$ echo $fp
f1:40:a7:4e:0f:28:8d:12:21:59:f1:ff:03:5f:63:54



This fingerprint should also appear on the console output of the instance. If it doesn't, then something is wrong. So, we'll get the console output, and grep through it looking for the fingerprint:


$ euca-get-console-output ${iid} > ${iid}.console
$ grep "ec2: ${length} ${fp}" ${iid}.console
ec2: 2048 f1:40:a7:4e:0f:28:8d:12:21:59:f1:ff:03:5f:63:54 /etc/ssh/ssh_host_rsa_key.pub (RSA)


We've now verified that the host we're connecting to is the host we just launched, so we can connect safely. Now you can clean out any old occurences of that host in known_hosts and tell the ssh client that this is a "known_host"


# remove existing entries in ~/.ssh/known_hosts for this host
$ ssh-keygen -R "${ihost}"

# hash the output of the known hosts file. This prevents someone
# from reading known_hosts as simple list of remote hosts you have
# access to in the event that one of your keys was compromised.

$ ssh-keygen -H -f ${iid}.keys

# Add the key to your known_hosts
$ cat ${iid}.keys >> ~/.ssh/known_hosts

# remove the temporary files we created
$ shred -u "${iid}."*


There, we've now verified that the remote host is the instance we started and told the ssh client about it.

Unfortunately, console output on ec2 is only updated approximately every 4 minutes. So, you can't run through this process until you have console output to check.

Updates

• [2010-09-22]: fix mismatched use of 'iid' and 'ihost'