Skip to content

Creating a session to do a S3 directory (S3Express) list_objects_v2 call fails. #991

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Closed
aprxi opened this issue Dec 6, 2023 · 3 comments
Closed

Creating a session to do a S3 directory (S3Express) list_objects_v2 call fails. #991

aprxi opened this issue Dec 6, 2023 · 3 comments
Labels
bug This issue is a bug.

Comments

@aprxi
Copy link

aprxi commented Dec 6, 2023 •
edited
Loading

Describe the bug

Creating a session to do a S3 directory (S3Express) list_objects_v2 call fails.

Could not find any exact documentation or examples. It is my understanding that to get the benefit of low latency retrievals, I need to comment out this line and setup a session on the Client. At least that is how it works in the Python SDK. When I use a similar implementation on Boto3 Python -- using same bucket-configuration and credentials -- it works correct.

It also works correct when I set " .disable_s3_express_session_auth(true)" on the config Builder.

Expected Behavior

No error when doing a client operation using create_session. Using list_objects_v2 as example, but the issue also happens on other bucket operations (e.g. get_object).

Current Behavior

Tracing shows that the CreateSession call itself is succesful:

TRACE invoke{service=s3 operation=CreateSession}:try_op:try_attempt:deserialization:deserialize_nonstreaming: aws_smithy_runtime::client::orchestrator::http: read HTTP response body response=Response { status: StatusCode(200), headers: Headers { headers: {"server": HeaderValue { _private: "AmazonS3" }, "x-amz-request-id": HeaderValue { _private: REDACTED }, "x-amz-id-2": HeaderValue { _private: REDACTED }, "content-type": HeaderValue { _private: "application/xml" }, "content-length": HeaderValue { _private: "1007" }, "date": HeaderValue { _private: "Wed, 06 Dec 2023 12:11:04 GMT" }} }, body: SdkBody { inner: Once(Some(b"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<CreateSessionResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Credentials><SessionToken> REDACTED </SessionToken><SecretAccessKey> REDACTED </SecretAccessKey><AccessKeyId> REDACTED </AccessKeyId><Expiration>2023-12-06T12:16:04Z</Expiration></Credentials></CreateSessionResult>")), retryable: true }, extensions: Extensions }

Parsing however seems to fail:

TRACE hyper::proto::h1::conn: shut down IO complete
Error: Unhandled(Unhandled { source: XmlDecodeError { kind: Custom("encountered invalid XML root: expected CreateSessionOutput but got StartEl { name: Name { prefix: \"\", local: \"CreateSessionResult\" }, attributes: [Attr { name: Name { prefix: \"\", local: \"xmlns\" }, value: \"http://s3.amazonaws.com/doc/2006-03-01/\" }], closed: false, depth: 0 }. This is likely a bug in the SDK.") }, meta: ErrorMetadata { code: None, message: None, extras: None } })

Reproduction Steps

async fn handler() -> Result<(), Error> {
    let bucket = "REDACTED--use1-az4--x-s3";

    let zone_id = "use1-az4";
    let region = Region::new("us-east-1");

    let zonal_endpoint = format!("s3express-{}-{}.{}.amazonaws.com", region, zone_id, region);

    let access_key_id = std::env::var("AWS_ACCESS_KEY_ID").unwrap();
    let secret_access_key = std::env::var("AWS_SECRET_ACCESS_KEY").unwrap();
    let session_token = std::env::var("AWS_SESSION_TOKEN").unwrap();

    let config = aws_sdk_s3::config::Builder::from(
            &aws_config::ConfigLoader::default()
                .credentials_provider(Credentials::from_keys(
                    access_key_id,
                    secret_access_key,
                    Some(session_token),
                ))
                .load()
                .await,
        )
        .region(region)
        //.disable_s3_express_session_auth(true)
        .endpoint_url(zonal_endpoint)
        .behavior_version(BehaviorVersion::latest())
        .build();

    let session_mode = aws_sdk_s3::types::SessionMode::ReadOnly;
    let client = Client::from_conf(config);

    // enable session auth
    let _ = client
        .create_session()
        .bucket(bucket)
        .session_mode(session_mode)
        .send()
        .await?;

    list_bucket(&client, bucket).await?;
    //get_bucket_object(&client, bucket, key).await?;
    Ok(())
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    tracing_subscriber::fmt()
        .with_max_level(Level::TRACE)
        .init();
    handler().await?;
    Ok(())
}

Possible Solution

It seems AWS API gives back CreateSessionResult, while parsing expects CreateSessionOutput ?

Additional Information/Context

No response

Version

├── aws-config v1.0.3
│   ├── aws-credential-types v1.0.3
│   │   ├── aws-smithy-async v1.0.3
│   │   ├── aws-smithy-runtime-api v1.0.3
│   │   │   ├── aws-smithy-async v1.0.3 (*)
│   │   │   ├── aws-smithy-types v1.0.3
│   │   ├── aws-smithy-types v1.0.3 (*)
│   ├── aws-http v0.60.0
│   │   ├── aws-smithy-runtime-api v1.0.3 (*)
│   │   ├── aws-smithy-types v1.0.3 (*)
│   │   ├── aws-types v1.0.3
│   │   │   ├── aws-credential-types v1.0.3 (*)
│   │   │   ├── aws-smithy-async v1.0.3 (*)
│   │   │   ├── aws-smithy-runtime-api v1.0.3 (*)
│   │   │   ├── aws-smithy-types v1.0.3 (*)
│   ├── aws-runtime v1.0.3
│   │   ├── aws-credential-types v1.0.3 (*)
│   │   ├── aws-http v0.60.0 (*)
│   │   ├── aws-sigv4 v1.0.3
│   │   │   ├── aws-credential-types v1.0.3 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.0
│   │   │   │   ├── aws-smithy-types v1.0.3 (*)
│   │   │   ├── aws-smithy-http v0.60.0
│   │   │   │   ├── aws-smithy-eventstream v0.60.0 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.0.3 (*)
│   │   │   │   ├── aws-smithy-types v1.0.3 (*)
│   │   │   ├── aws-smithy-runtime-api v1.0.3 (*)
│   │   │   ├── aws-smithy-types v1.0.3 (*)
│   │   ├── aws-smithy-async v1.0.3 (*)
│   │   ├── aws-smithy-eventstream v0.60.0 (*)
│   │   ├── aws-smithy-http v0.60.0 (*)
│   │   ├── aws-smithy-runtime-api v1.0.3 (*)
│   │   ├── aws-smithy-types v1.0.3 (*)
│   │   ├── aws-types v1.0.3 (*)
│   ├── aws-sdk-sso v1.4.0
│   │   ├── aws-credential-types v1.0.3 (*)
│   │   ├── aws-http v0.60.0 (*)
│   │   ├── aws-runtime v1.0.3 (*)
│   │   ├── aws-smithy-async v1.0.3 (*)
│   │   ├── aws-smithy-http v0.60.0 (*)
│   │   ├── aws-smithy-json v0.60.0
│   │   │   └── aws-smithy-types v1.0.3 (*)
│   │   ├── aws-smithy-runtime v1.0.3
│   │   │   ├── aws-smithy-async v1.0.3 (*)
│   │   │   ├── aws-smithy-http v0.60.0 (*)
│   │   │   ├── aws-smithy-runtime-api v1.0.3 (*)
│   │   │   ├── aws-smithy-types v1.0.3 (*)
│   │   ├── aws-smithy-runtime-api v1.0.3 (*)
│   │   ├── aws-smithy-types v1.0.3 (*)
│   │   ├── aws-types v1.0.3 (*)
│   ├── aws-sdk-ssooidc v1.4.0
│   │   ├── aws-credential-types v1.0.3 (*)
│   │   ├── aws-http v0.60.0 (*)
│   │   ├── aws-runtime v1.0.3 (*)
│   │   ├── aws-smithy-async v1.0.3 (*)
│   │   ├── aws-smithy-http v0.60.0 (*)
│   │   ├── aws-smithy-json v0.60.0 (*)
│   │   ├── aws-smithy-runtime v1.0.3 (*)
│   │   ├── aws-smithy-runtime-api v1.0.3 (*)
│   │   ├── aws-smithy-types v1.0.3 (*)
│   │   ├── aws-types v1.0.3 (*)
│   ├── aws-sdk-sts v1.4.0
│   │   ├── aws-credential-types v1.0.3 (*)
│   │   ├── aws-http v0.60.0 (*)
│   │   ├── aws-runtime v1.0.3 (*)
│   │   ├── aws-smithy-async v1.0.3 (*)
│   │   ├── aws-smithy-http v0.60.0 (*)
│   │   ├── aws-smithy-json v0.60.0 (*)
│   │   ├── aws-smithy-query v0.60.0
│   │   │   ├── aws-smithy-types v1.0.3 (*)
│   │   ├── aws-smithy-runtime v1.0.3 (*)
│   │   ├── aws-smithy-runtime-api v1.0.3 (*)
│   │   ├── aws-smithy-types v1.0.3 (*)
│   │   ├── aws-smithy-xml v0.60.0
│   │   ├── aws-types v1.0.3 (*)
│   ├── aws-smithy-async v1.0.3 (*)
│   ├── aws-smithy-http v0.60.0 (*)
│   ├── aws-smithy-json v0.60.0 (*)
│   ├── aws-smithy-runtime v1.0.3 (*)
│   ├── aws-smithy-runtime-api v1.0.3 (*)
│   ├── aws-smithy-types v1.0.3 (*)
│   ├── aws-types v1.0.3 (*)
├── aws-credential-types v1.0.3 (*)
├── aws-sdk-s3 v1.5.0
│   ├── aws-credential-types v1.0.3 (*)
│   ├── aws-http v0.60.0 (*)
│   ├── aws-runtime v1.0.3 (*)
│   ├── aws-sigv4 v1.0.3 (*)
│   ├── aws-smithy-async v1.0.3 (*)
│   ├── aws-smithy-checksums v0.60.0
│   │   ├── aws-smithy-http v0.60.0 (*)
│   │   ├── aws-smithy-types v1.0.3 (*)
│   ├── aws-smithy-eventstream v0.60.0 (*)
│   ├── aws-smithy-http v0.60.0 (*)
│   ├── aws-smithy-json v0.60.0 (*)
│   ├── aws-smithy-runtime v1.0.3 (*)
│   ├── aws-smithy-runtime-api v1.0.3 (*)
│   ├── aws-smithy-types v1.0.3 (*)
│   ├── aws-smithy-xml v0.60.0 (*)
│   ├── aws-types v1.0.3 (*)

Environment details (OS name and version, etc.)

Linux - 6.1.0-10-amd64 ( Debian )

Logs

No response
@aprxi aprxi added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Dec 6, 2023
@aprxi aprxi changed the title (short issue description) Creating a session to do a S3 directory (S3Express) list_objects_v2 call fails. Dec 6, 2023
@rcoh
Copy link
Contributor

rcoh commented Dec 6, 2023

The Rust SDK does not currently support S3 Express One Zone. Please upvote #992 to guide prioritization

@rcoh rcoh closed this as completed Dec 6, 2023
@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 6, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@Velfi Velfi removed the needs-triage This issue or PR still needs to be triaged. label Mar 20, 2024
@ysaito1001
Copy link
Collaborator

This should be fully working now, with S3 Express One Zone released as of release-2024-03-13.

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
bug This issue is a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rcoh @Velfi @ysaito1001 @aprxi