Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Support path-style access URLs #66

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

elliot42
Copy link
Contributor

There are two ways to generate S3 object URLs, virtual-hosted style
URLs vs. path-style URLs 1:

  • Virtual-hosted-style: http://bucket.s3.amazonaws.com
  • Path-style: http://s3.amazonaws.com/bucket

Virtual-hosted-style is Amazon's preferred default, however HTTPS is
not compatible with all virtual-hosted-style URLs. Specifically,
virtual-hosted-style buckets with dots in their names always cause
HTTPS cert validation errors, as per RFC 2818 2:

https://foo.bar.s3.amazonaws.com/key

Path-style access works fine with HTTPS, without forcing a bucket rename:

https://s3.amazonaws.com/foo.bar/key

This commit allows configuring the client for path-style access via
the cred map:

(let [cred {:access-key ...
            :secret-key ...
            :path-style-access? true}]
  (generate-presigned-url cred bucket key))

Note that when using path style access you may need to manually specify your
region-specific S3 endpoint 1:

(let [cred {...
            :path-style-access? true
            :endpoint "s3-us-west-1.amazonaws.com"}]
  ...)

@weavejester
Copy link
Owner

As :path-style-access is a boolean, can you add a ? onto the end?

@elliot42 elliot42 force-pushed the path-style-access branch 2 times, most recently from f134bc8 to 66b959f Compare September 26, 2014 12:53
There are two ways to generate S3 object URLs, virtual-hosted style
URLs vs. path-style URLs [1]:

- Virtual-hosted-style: `http://bucket.s3.amazonaws.com`
- Path-style: `http://s3.amazonaws.com/bucket`

Virtual-hosted-style is Amazon's preferred default, however HTTPS is
not compatible with all virtual-hosted-style URLs.  Specifically,
virtual-hosted-style buckets with dots in their names always cause
HTTPS cert validation errors, as per RFC 2818 [2][3]:

    https://foo.bar.s3.amazonaws.com/key

Path-style access works fine with HTTPS, without forcing a bucket rename:

    https://s3.amazonaws.com/foo.bar/key

This commit allows configuring the client for path-style access via
the cred map:

    (let [cred {:access-key ...
                :secret-key ...
                :path-style-access? true}]
      (generate-presigned-url cred bucket key))

Note that when using path style access you may need to manually specify your
region-specific S3 endpoint [1]:

    (let [cred {...
                :path-style-access? true
                :endpoint "s3-us-west-1.amazonaws.com"}]
      ...)

[1]: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html
[2]: http://www.ietf.org/rfc/rfc2818.txt
[3]: http://shlomoswidler.com/2009/08/amazon-s3-gotcha-using-virtual-host.html
@elliot42
Copy link
Contributor Author

Updated to include ? at ends of keyword keys/symbols.

@weavejester
Copy link
Owner

Thanks. Could you also slim down the commit message? There's no need to explain everything about the change. You can just provide a small explanation and a link to more details.

Add support for path-style access

This is necessary to support HTTPS URLs to buckets with periods in
their name.
See: http://shlomoswidler.com/2009/08/amazon-s3-gotcha-using-virtual-host.html

Conversely, it would be useful to add more information to the namespace docstring, detailing the option you've added. The docstring is currently a little cluttered, so you may want to reformat it like so:

  "Functions to access the Amazon S3 storage service.

  Each function takes a map of credentials as its first argument, which must
  contain the following keys:

    :access-key - the AWS access key ID
    :secret-key - the AWS secret access key

  The credentials may optionally have keys:

    :endpoint           - a URL or keyword refering to the AWS endpoint
    :proxy              - a map locating a HTTP proxy
    :path-style-access? - true if path-style access should be used

  The :proxy map must contain the following keys:

    :host - the hostname of the proxy
    :port - the port of the proxy

  And may contain:

    :user        - the user to use for the proxy
    :password    - the password of the user
    :domain      - the domain of the user, if applicable
    :workstation - the workstation of the user, if applicable"

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants