Amazon's Elastic Compute Cloud (EC2)

Introduction

• A full listing of instance types: Instance Types
• A full description of purchasing options: Purchasing Options
• A full listing of public Amazon Machine Images: AMIs

Many of the basic features of EC2 can be used for no charge when signing up under the Free Tier. The Free Tier includes 750 hours of micro instance usage every month as well as a limited free usage of many accompanying servicess. See the Free Tier page for further details. The following section will walk through setting up an example instance for free.

Micro Instance Quick Start

Create an account and access the management console

• To create an Amazon Web Services account, go to aws.amazon.com and select "sign up"
• You will need to create an Amazon account or sign in with an existing one
• Following this, you will need to supply some personal information and payment information, verify yourself by telephone, and select a support plan

EC2 signup: account creation

EC2 signup: credit card info

EC2 signup: telephone verification

EC2 signup: support plan

EC2 signup: finalization

• After signing up, you can access the management console to begin using EC2
• The management console can be accessed by a menu in the upper right hand corner
• The management console provides access to all of the Amazon Web Services, but for now we only need EC2

Accessing the management console

The management console

Create the micro instance

• In the management console, select "EC2"
• From the menu on the left-hand side, select "Instances"
• As no instances are currently running, this screen is blank. Select "Launch Instance" to create one
• First, you must choose an AMI to run on your instance. For this example, choose the 64 bit Amazon Linux AMI
• Next, you choose how many and of what type of instance to launch. Enter 1 T1 micro instance. Use the default subnet setting
• At the advanced options, all settings can be left at their defaults
• At the storage device configuration, all settings can be left at their default
• Tags can be added to instances to make them more recognizable. Name this instance "MicroDemo"
• A key pair must be created to connect to your instance via SSH. Create a key pair named "ec2DemoKey" and download this
• A firewall must be configured around your instance. There should be an existing Security Group called "quick-start-1". This sets up a firewall that blocks all network traffic except SSH (port 22)
• Finally, review your settings and click "Launch"
• Your instance should show up in the instance list

Launch Instance: AMI selection

Launch Instance: type selection

Launch Instance: advanced options

Launch Instance: storage configuration

Launch Instance: tags

Launch Instance: key pair

Launch Instance: firewall

Launch Instance: review

Connect to the micro instance

• To connect to our micro instance we will need to SSH to it and provide our key that was selected for the instance
• First, move your key to a suitable location. For example: "mv Downloads/ec2DemoKey.pem .ssh/"
• Next, ensure your key is publicly viewable. For example: "chmod 400 .ssh/ec2DemoKey.pem"
• Obtain the instance DNS address by clicking the instance and finding the "Public DNS" field in the lower window
• The username for instances running the Amazon Linux AMI is "ec2-user", though this can vary for other AMIs
• Finally, SSH into the instance. For example: "ssh -i .ssh/ec2DemoKey.pem ec2-user@ec2-54-213-135-73.us-west-2.compute.amazonaws.com"
• Alternatively, right click on the instance and click "connect". If Java is enabled in your browser, you can connect using the browser SSH client

Connecting to an instance using SSH

Set up and run the configuration/validation pipeline

• Now that we have connected to our instance, we can go about running something on it
• We will run the configuration/validation pipeline which uses SMAC and ParamILS to configure Spear and Autoweka. We will run this in fast mode to see results quickly.
• In order to do this, we will get OS updates, an updated version of ruby, download and decompress the package, and run it
• Follow these steps:
  • sudo yum -y update
  • curl -L https://get.rvm.io | bash -s stable --ruby
  • source /home/ec2-user/.rvm/scripts/rvm
  • wget www.cs.ubc.ca/~hutter/AC_on_the_Cloud.tar.gz
  • tar xzvf AC_on_the_Cloud.tar.gz
  • cd AC_on_the_Cloud
  • ruby scripts/blackbox_evaluator.rb -f
• It should then run and finish in a few minutes. You will notice the Autoweka runs terminated improperly. This is because micro instances have too little memory to run Autoweka

CLI tools

Setting up the CLI tools

• Download and unzip the tools:
  • wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
  • unzip ec2-api-tools.zip
• Next, you need to generate keys which will allow you to access your EC2 instances without being logged in through a browser
  • Go to the management console and select "Security Credentials" from the menu in the upper right-hand corner
  • On the Security Credentials page, go to "Access Keys" and select "Create New Root Key"
  • This will download a csv file containing both your "AWSAccessKeyId" and "AWSSecretKey"
• Now we will need to set some environment variables

export EC2_HOME=/opt/ec2-api-tools-1.6.8.1	(location of the unzipped ec2 tools)
export PATH=$PATH:$EC2_HOME/bin	(add the directory to your path)
export AWS_ACCESS_KEY=ABCDEFGHIJKLMNOPQRST	(value of "AWSAccessKeyId" downloaded earlier)
export AWS_SECRET_KEY=abc123DEFghiJKL456789+mnoPqRsTUV0wxyZ123	(value of "AWSSecretKey" downloaded earlier)
export EC2_URL=https://ec2.us-west-2.amazonaws.com	(The URL of your default EC2 region. Can be found here)

• Everything should now be set up to use

Using the CLI tools

• Now that the CLI tools have been set up, we will use them to launch an instance
• If your PATH has been set properly, the commands can be used from anywhere. All commands begin with "ec2"
• To launch an instance, we will use the command "ec2-run-instances" or the short form "ec2run"
• The full command will resemble this: "ec2run ami-0358ce33 -t t1.micro -n 1 -k ec2DemoKey -g sg-565ab739"
  • "ami-0358ce33" specifies the AMI to use. This is the ID for the Amazon Linux AMI
  • "-t t1.micro" specifies what type of instance we wish to launch
  • "-n 1" indicates that we want to launch a single instance
  • "-k ec2DemoKey" specifies the key that this instance requires to connect
  • "-g sg-565ab739" specifies the security group the instance uses. This is the ID for the "quick-start-1" group
• After executing this command, output will be generated, giving all the generated instance's properties. Check the management console to verify the instance is launching

• A full listing and description of the commands can be found here: Commands (CLI Tools)
• A quick reference card for common commands can be found here: Reference Card
• All commands provide usage information if "-h" is used on the command line

Working from an Instance

User Data Scripts

• When launching an instance, the "User Data" field may be utilized to pass any information to an instance
• This data can then be read from within the instance (next section) or executed as a script on startup
• Anytime the user data begins with the characters "#!", it will be run at startup under the root user
• From command line, user data can be provided inline with "-d" or as a file with "-f"
• The following script provided as user data will execute the quick start instructions from above automatically:

#!/bin/bash
sudo yum -y update
curl -L https://get.rvm.io | bash -s stable --ruby
source /home/ec2-user/.rvm/scripts/rvm
wget www.cs.ubc.ca/~hutter/AC_on_the_Cloud.tar.gz
tar xzvf AC_on_the_Cloud.tar.gz
cd AC_on_the_Cloud
ruby scripts/blackbox_evaluator.rb -f

Accessing Instance Metadata

• Information about an instance such as AMI, instance ID, and user data can be accessed from within the instance itself
• The following examples show various pieces of information that can be accessed:

GET http://169.254.169.254/latest/user-data
GET http://169.254.169.254/latest/meta-data/ami-id
GET http://169.254.169.254/latest/meta-data/instance-type
GET http://169.254.169.254/latest/meta-data/instance-id

• Using just "http://169.254.169.254/latest/meta-data" will list names of the pieces of meta-data that can be accessed

CLI Tools on instances

• If your instance is running the Amazon Linux AMI, it will have the CLI Tools already installed
• The keys and region will not be set
• You can either set the environment variables for "AWS_ACCESS_KEY", "AWS_SECRET_KEY", and "EC2_URL" or
• You can provide these as command line options with each command: "-O, --aws-access-key", "-W, --aws-secret-key", and "-U, --url"