Reading S3 via s3cmd from Ubuntu 12.04 EC2 instance

Getting EC2 to read S3

I have a simple need to be able to read/write data at my S3 from an EC2 instance.   You would think there would be a nice simple way of doing that, like allowing EC2 instances with a certain (ec2) security group have permissions to a bucket or a folder in a bucket of S3.  But there is not, at least not from what I can see.   The good news is there is a way to do this, but its more complex and it gives you a lot more tools than you could ask for.

So with that let’s start diving into it all…

S3

First I am going to log into S3 via the web console and make a new bucket and place a few folders and files in it.

Go to http://aws.amazon.com/

Click on My Account/Console à AWS management Console

Sign in with your amazon account email and password

Click on S3

Click “Create Bucket”

I will call this bucket “pats-test-bucket”    click create

Select the pats-test-bucket then click on “Create Folder”

Call it test_folder

Open up the test_folder and click on Upload.  Upload a few files.

 I uploaded two files Test.xls and test_file.txt

Get your keys

Get your main keys for your account.

Go to http://aws.amazon.com/

Click on My Account/Console à Security Credentials

Go down to the Access Credentials and copy the Access Key ID,  mine of course is blanked out for security.

But let’s suppose it’s

Access Key ID = KAIAA3478XOQQHNYF54A

Now click on the show button under the Secret Access Key

Copy the Secret Access Key, again I blanked mine out for security reason.

Let’s assume its  xdg4x/26cfr9+XqVInnf438Hdd34PjiLzhAi43Dd

For example purposes we have

Access Key ID       = KAIAA3478XOQQHNYF54A

Secret Access Key = xdg4x/26cfr9+XqVInnf438Hdd34PjiLzhAi43Dd

If you do not have an Ubuntu 12.04 ec2 running, here is the command line to get one created.  This assumes you have AWS command line tools set up on your system and have a keypair created.  If not you can use the AWS web console.

> ec2-run-instances ami-9c78c0f5 -b /dev/sda1=:8:true -k my-keypair -t t1.micro -g default  --availability-zone us-east-1a

(use your own keypair)

Now log into your Ubuntu 12.04 EC2 instance

In my case it’s at ec2-184-72-175-14.compute-1.amazonaws.com

>  ssh -i .ec2/my-keypair.pem ubuntu@ec2-184-72-175-14.compute-1.amazonaws.com

(again this assumes you have your keypair in the given location)

s3cmd

Use apt-get to install s3cmd which is made by s3tools http://s3tools.org/s3cmd [1]

>  sudo apt-get install s3cmd

After it installs configure it

>  s3cmd --configure

It will ask for you Access Key and then your Secret Key Enter them both

It will then ask for encryption password/GPG/HTTPS, etc just choose the defaults.

Click Y to test it

Success!

Run the following command to download files from the S3 bucket

>  s3cmd get --recursive s3://pats-test-bucket/ .

This will download all the contents within the bucket pats-test-bucket

Here you can see the files have been downloaded.

The problem

The big problem with this set up is the keys.  These keys, that were obtained on the Security Credentials page, have access to everything on AWS for your account.    In the wrong hands these keys can wreak havoc.   So I for one do not want them on my ec2 running instance.

The good news is that AWS provides a tool for still using keys but giving them limited permissions.   The tool is called Identity and Access Management (IAM) http://aws.amazon.com/iam/ [2]

IAM

This is my first journey into using IAM, so bear with me if I make a few mistakes or can’t fully explain what I am doing. J

In this example I am going to create a user called test_bob and give that user special permissions that only allow him to read from a specific S3 bucket.

Log into the web console and click on IAM

Click on Users.

Click on Create New Users

Enter the name test_bob and click create.

Click on Download Credentials

This file will contain something like this. 

"User Name","Access Key Id","Secret Access Key"

"test_bob","AKIAJBFSHWME4UTQDXHQ","knR6B8Slm8sHFZ6URhZtgvwlfzWoVOPRlV6jjON9"

Access Key Id     = AKIAJBFSHWME4UTQDXHQ

Secret Access Key = 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

Click on Close Window

Select the user then click on Permissons à Attach User Policy

Scroll down and select “Amazon S3 Read Only Access” and click on Select.

Review then click on Apply Policy (You can always change it later)

Here is the full policy

{   "Statement": [     {       "Effect": "Allow",       "Action": [         "s3:Get*",         "s3:List*"       ],       "Resource": "*"     }   ] }

This policy will allow this user to get and list all document in all of your S3 buckets.  Lets keep it that way for now so we can confirm it when we limit it later.

Go back to the ec2 instance and reconfigure s3cmd and use the keys for the test_bob user.

>  s3cmd --configure

Enter test_bob’s keys

Access Key Id     = AKIAJBFSHWME4UTQDXHQ

Secret Access Key = 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

After it passes its test run the following command

>  s3cmd ls

>  s3cmd ls S3://pats-test-bucket/test_folder/

>  s3cmd get S3://pats-test-bucket/test_folder/Test.xlsx

The following command is denied since this user does not have write permission.

>  s3cmd put S3://pats-test-bucket/

Create a new bucket

Now open up S3 in the web console and add another bucket called pat2-test-bucket

Put a few files in the new bucket

Run the following command

>  s3cmd ls

You see both buckets

Now limit policy of this user so that only the first bucket can be seen.  You can find more about policies at http://docs.amazonwebservices.com/IAM/latest/UserGuide/ExampleIAMPolicies.html [3]

From the IAM web console click on the useràPermissions then finally “Manage Policy”

Update the policy to

{   "Statement": [     {       "Effect": "Allow",       "Action": [         "s3:Get*",         "s3:List*"       ],       "Resource": [           "arn:aws:s3:::pat-test-bucket",           "arn:aws:s3:::pat-test-bucket/*"       ]     }   ] }

Apply the policy

Run the following command

>  s3cmd ls

But we get an error

I think this policy is correct but it will not work with the s3cmd.  I found this site http://blog.tcs.de/s3cmd-set-iam-permissions/ [4]

Change the policy to

{   "Statement": [     {       "Effect": "Allow",       "Action": [         "s3:ListAllMyBuckets"       ],       "Resource": "arn:aws:s3:::*"     },     {       "Effect": "Allow",       "Action": [           "s3:Get*",           "s3:List*"       ],       "Resource": [           "arn:aws:s3:::pats-test-bucket",           "arn:aws:s3:::pats-test-bucket/*"       ]     }   ] }

According to the website S3cmd needs s3:ListAllMyBuckets.  Which though not ideal we have to do.

Run the following command

>  s3cmd ls

You can see both buckets but if you run

>  s3cmd ls S3://pat2-test-bucket/

Perfect we cannot read anything in that bucket

Run

>  s3cmd ls S3://pats-test-bucket/test_folder/

And we can see the other bucket’s contents.

References

[1]  S3 Tools

       http://s3tools.org/s3cmd

       Visited 1/2012

[2]  AWS IAM

       http://aws.amazon.com/iam/

       Visited 11/2012

[3]  IAM Policies

       http://docs.amazonwebservices.com/IAM/latest/UserGuide/ExampleIAMPolicies.html

       Visited 11/2012

[4]  Set AWS IAM Permission for s3cmd backup to single bucket

       http://blog.tcs.de/s3cmd-set-iam-permissions/

       Visited 11/2012