Skip to content

Latest commit

 

History

History
142 lines (92 loc) · 6.28 KB

amazon-s3.md

File metadata and controls

142 lines (92 loc) · 6.28 KB
Raw

Amazon S3

What is S3 ? Basic concepts

S3 is Amazon's platform for data storage. The main idea of S3 is based on buckets and keys. Buckets are containers which store data. You create one bucket for one project. Keys are the unique ID for an object inside the bucket (container).

If the bucket's name is 'abc' and key is 'def', then the full url for the object will be http://abc.s3.amazonaws.com/def . It is interesting to note that key can include slashes. So if a key is 'image/2/full.png', then the full url will be http://abc.s3.amazonaws.com/image/2/full.png This makes S3 easy and flexible for data storage.

Setting up S3 for orga-server

1 - Create account on AWS (https://aws.amazon.com/).

2 - Once account is created, # to AWS Management Console

3 - Open S3 from the massive list of web services Amazon provides.

4 - Click on the create bucket button to create a bucket. When the dialog opens, just enter a unique name and click "Create".

create_bucket

5 - Bucket will be created and shown in the all bucket list.

6 - Now it's time to create access tokens for your bucket. Choose "Security Credentials" option from the dropdown menu with your full name as label.

sec_cred

7 - Choose IAM when the following dialog appears.

iam_box

8 - You will be presented with the IAM Management Console. Click on "Create New Users".

9 - Enter a username (example opev_user) and click on "Create".

create_user

10 - You will be shown the user credentials. Don't download them. Copy them and keep them safe. Then click on "Close".

user_creds

11 - You will be at IAM Users list now.

users_list

12 - Now that the user has been created, it's time to give it bucket permissions. Click on Policies in the sidebar.

13 - Click on "Create Policy" button and choose "Create your own Policy".

create_own_policy

14 - Enter a policy name, a description and the following in the policy document. (Be sure to replace 'opevbucketname' with your bucket name.)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::opevbucketname"
        },
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "arn:aws:s3:::opevbucketname/*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
        }
    ]
}

policy_docu

15 - Once policy is created, go back to Users tab from the sidebar.

16 - Click on opev_user (the user you just created) and when the user loads, open "Permissions" tab.

user_policy

17 - Click on "Attach policy" button, search for the policy you just created (here opev_bucket_access) and attach it.

18 - You will be redirected back to user permissions tab but now it will have that policy.

policy_done

19 - That's it. Set the environment variables BUCKET_NAME, AWS_KEY and AWS_SECRET to their respective values and the server will use S3 for storage.

20 - Here BUCKET_NAME is 'opevbucketname' and AWS_KEY and AWS_SECRET are what you got from step 10.

Using the (S3) storage module in code

S3 works on the concept of keys. Using the same idea, app.helpers.storage module has been created to allow uploading files to both S3 and local server. Here is a basic example.

from flask import request, current_app as app
from app.helpers.storage import upload
import flask_login as login

@app.route('/users/upload/', methods=('POST'))
def view():
    profile_pic = request.files['profile_pic']
    url = upload(profile_pic, 'users/%d/profile_pic' % login.current_user.id)
    print url  # full URL to the uploaded resource, either on local server or S3

upload takes 2 parameters; the file object and the key. The key should be chosen wisely according to the purpose. For example,

  • When uploading user avatar, key should be 'users/{userId}/avatar'
  • When uploading event logo, key should be 'events/{eventId}/logo'
  • When uploading audio of session, key should be 'events/{eventId}/sessions/{sessionId}/audio'

This helps to avoid conflicts on the server and keep data distinct.

Another important feature of upload is that it automatically switches to uploading on local server if AWS env vars are not set. So it doesn't affect development workflow. upload can be used in a uniform way without worrying where the data will be stored.

Also note that upload always returns the absolute link of the file that is uploaded. So you can use the returned url directly in templates i.e. no need to use ``{{ url_for('static', uploadedUrl) }}, just use {{uploadedUrl}}`.

References