s3 head object, erroneously modified object name

[luoyanpeng-bytedancer]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

s3 head object：If the object name starts with"/+ bucket name", the SDK will remove the bucket name from the object name in the path of the URL, causing the request to return 404.

code :
s3@v1.73.2 internal/customizations/update_endpoint.go
func moveBucketNameToHost(u *url.URL, bucket string) { u.Host = bucket + "." + u.Host removeBucketFromPath(u, bucket) }

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Object names should not be modified

Current Behavior

If the object name starts with"/+ bucket name", the SDK will remove the bucket name from the object name in the path of the URL

Reproduction Steps

Head An object prefixed with "/+ bucket name"

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2 v1.33.0
github.com/aws/aws-sdk-go-v2/config v1.5.0
github.com/aws/aws-sdk-go-v2/service/s3 v1.73.2
github.com/aws/smithy-go v1.22.1

Compiler and Version used

go version go1.21.7 darwin/amd64

Operating System and version

macOS 14.0 (23A344)

[adev-code]

Hello @luoyanpeng-bytedancer, thanks for reaching out. For doing head object operations on S3 key names that contains characters such as "/" and "+", you would need to encode those characters as they are identified as special characters for S3 (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html). So for the request, it would need to have the encoded version of the characters. If you have any questions, please do let me know. Thanks.

[luoyanpeng-bytedancer]

Hello @luoyanpeng-bytedancer, thanks for reaching out. For doing head object operations on S3 key names that contains characters such as "/" and "+", you would need to encode those characters as they are identified as special characters for S3 (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html). So for the request, it would need to have the encoded version of the characters. If you have any questions, please do let me know. Thanks.

Why boto3 can not encode "/", and why only "/+ bucket name" has this problem

[adev-code]

The way I have replicated on my side is that:

1. Uploaded a sample object inside a prefix. From local machine, it looked like this: "/ +bucket-name/sample.png". So, the name of the folder actually has "/" then followed by a space then a "+".
2. Then uploaded it using the AWS Go SDK to S3. The S3 URI appeared as follows: s3://bucket-name// +bucket-name/sample.png . As shown, the is a double slash added.
3. Then I ran the Head Object operation and got 200OK. I did not get the 404. Not that I have a / and a space and a + on the name of the object key and it worked fine.

My snippet of code for Head Object:

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

const (
    bucketName = "bucket-name"               
    objectKey  = "/ +bucket-name/sample.png" 
    region     = "us-east-1"                                     
)

func main() {
    ctx := context.Background()

    logFile, err := os.OpenFile("s3_head_object_go.log", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
    if err != nil {
        log.Fatalf("Failed to open log file: %v", err)
    }
    log.SetOutput(logFile) 

    cfg, err := config.LoadDefaultConfig(ctx,
        config.WithRegion(region),
        config.WithClientLogMode(aws.LogRequest|aws.LogResponse),
    )
    if err != nil {
        log.Fatalf("failed to load AWS config: %v", err)
    }

    client := s3.NewFromConfig(cfg)

    resp, err := client.HeadObject(ctx, &s3.HeadObjectInput{
        Bucket: aws.String(bucketName),
        Key:    aws.String(objectKey),
    })
    if err != nil {
        log.Printf("Error occurred: %v\n", err)
        return
    }

    fmt.Println("Metadata:", resp.Metadata)
    fmt.Println("Content Length:", resp.ContentLength, "bytes")
    fmt.Println("Last Modified:", resp.LastModified)
    fmt.Println("ETag:", *resp.ETag)
    fmt.Println("Content Type:", *resp.ContentType)

    log.Printf("HeadObject Response: %+v\n", resp)
}

From the above,
A) How are you uploading the object with the prefix ? Is it straight to the console via drag and drop or via the SDK?
B) When you upload the object on S3, wanted to clarify if the prefix actually have the forward slash in its name?
C) On your code, does the object key match to the S3 URI that is shown on your console?

If you have any other questions, please let me know. Thanks.

[luoyanpeng-bytedancer]

objectKey = "/ +bucket-name/sample.png"

There are some misunderstandings here, try objectKey = "/bucket-name/sample.png" and set the endpoint

[adev-code]

I tried with "/bucket-name/sample.png" and I did not get a 404 error. Can you provide (and code) :
1. How you are uploading the object name to S3, is it via SDK, CLI or drag and drop to S3?
2. After uploading, please provide the format of S3 URI shown on S3 Console?
3. The Region of the bucket you are using?
4. When doing the HEAD operation, can you have the logging enabled (https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-logging.html) : cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithClientLogMode(aws.LogRequest | aws.LogResponse))

[github-actions]

This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

[luoyanpeng-bytedancer]

Sorry, an error message was given, the object name should be "bucket-name/sample.png".
Here is my test code

`region := "ap-southeast-1"
ctx := context.Background()

customResolver := aws.EndpointResolverFunc(func(service, region string) (aws.Endpoint, error) {
	return aws.Endpoint{
		URL:           "http://s3.ap-southeast-1.amazonaws.com",
		SigningRegion: region,
		Source:        aws.EndpointSourceCustom,
	}, nil
})

cfg, err := config.LoadDefaultConfig(ctx,
	config.WithRegion(region),
	config.WithEndpointResolver(customResolver),
	config.WithCredentialsProvider(credentials.StaticCredentialsProvider{
		Value: aws.Credentials{AccessKeyID: "", SecretAccessKey: ""}}),
)
if err != nil {
	return
}

client := s3.NewFromConfig(cfg)

_, err = client.HeadObject(ctx, &s3.HeadObjectInput{
	Bucket: aws.String("bzy-mv"),
	Key:    aws.String("bzy-mv/object"),
})
if err != nil {
	fmt.Printf("head failed, err = %s", err.Error())
}`

I tried it, if I don't use "AWS. EndpointResolverFunc" is fine, but since I will use storage services from other vendors, I need to use this

[lucix-aws]

aws.EndpointResolver, aws.EndpointResolverFunc, etc. are all deprecated. You will likely experience buggy behavior with these as the S3 endpoint resolution model has evolved.

Please read through https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-endpoints.html.

since I will use storage services from other vendors

Aside, this SDK is designed to work with official AWS services, I can't guarantee you it will function against anything else.

Since @adev-code has verified that the behavior you're seeing isn't occuring with the (default) v2 endpoint resolution, I'm going to close this.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.