Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

fix(services/s3): Ignore empty query to make it more compatible #2028

Merged
merged 1 commit into from
Apr 18, 2023

Conversation

Xuanwo
Copy link
Member

@Xuanwo Xuanwo commented Apr 18, 2023

Services like cos can't handle empty query correctly:

[2023-04-18T11:15:03Z ERROR opendal::services] service=s3 operation=scan path=x/ -> failed: Unexpected (permanent) at Pager::next => S3Error { code: "InvalidArgument", message: "Invalid Argument", resource: "/", request_id: "NjQzZTdiYjdfMTQ5MDBjMGJfMjZjMmRfMThhM2NkOQ==" }
    
    Context:
        response: Parts { status: 400, version: HTTP/1.1, headers: {"content-type": "application/xml", "content-length": "411", "connection": "keep-alive", "date": "Tue, 18 Apr 2023 11:15:03 GMT", "server": "tencent-cos", "x-amz-request-id": "NjQzZTdiYjdfMTQ5MDBjMGJfMjZjMmRfMThhM2NkOQ==", "x-amz-trace-id": "OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODczNTBmNjMwZmQ0MTZkMjg0NjlkNTYyNmY4ZTRkZTk0N2U1NmFlOGIyM2Y0NWQyNTA2YzEzYjVkZDExYzhhNmI2YzU0MTZkYTQxZDQwYWU2ODNmMDZlN2Q4ZjEzZTA0MTg="} }
        service: s3
        path: x/
    
[2023-04-18T11:15:03Z DEBUG opendal::services] service=s3 operation=scan path=x/ -> partial entries read finished
Error: Unexpected (permanent) at Pager::next, context: { response: Parts { status: 400, version: HTTP/1.1, headers: {"content-type": "application/xml", "content-length": "411", "connection": "keep-alive", "date": "Tue, 18 Apr 2023 11:15:03 GMT", "server": "tencent-cos", "x-amz-request-id": "NjQzZTdiYjdfMTQ5MDBjMGJfMjZjMmRfMThhM2NkOQ==", "x-amz-trace-id": "OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODczNTBmNjMwZmQ0MTZkMjg0NjlkNTYyNmY4ZTRkZTk0N2U1NmFlOGIyM2Y0NWQyNTA2YzEzYjVkZDExYzhhNmI2YzU0MTZkYTQxZDQwYWU2ODNmMDZlN2Q4ZjEzZTA0MTg="} }, service: s3, path: x/ } => S3Error { code: "InvalidArgument", message: "Invalid Argument", resource: "/", request_id: "NjQzZTdiYjdfMTQ5MDBjMGJfMjZjMmRfMThhM2NkOQ==" }


failures:
    services_s3_list::test_scan
    services_s3_list::test_scan_root

After this PR, scan related feature works. However, COS still can't handle special_chars:

---- services_s3_write::test_delete_with_special_chars stdout ----
[2023-04-18T11:20:45Z ERROR opendal::services] service=s3 operation=Writer::write path=d3e07840-8c82-4537-8079-5de3fbfe19ad !@#$%^&()_+-=;',.txt written=0 -> data write failed: PermissionDenied (permanent) at Writer::write => S3Error { code: "SignatureDoesNotMatch", message: "The Signature you specified is invalid.", resource: "/14173046-3958-431c-871a-b5cf419126ab/d3e07840-8c82-4537-8079-5de3fbfe19ad !@#$%^&()_+-=;',.txt", request_id: "NjQzZTdkMGNfNjkxMjI0MDlfYzU5MF80MjUwNmQ2" }
    
    Context:
        response: Parts { status: 403, version: HTTP/1.1, headers: {"content-type": "application/xml", "content-length": "1091", "connection": "keep-alive", "date": "Tue, 18 Apr 2023 11:20:44 GMT", "server": "tencent-cos", "x-amz-request-id": "NjQzZTdkMGNfNjkxMjI0MDlfYzU5MF80MjUwNmQ2", "x-amz-trace-id": "OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODc0OWRkZjk0ZDM1NmI1M2E2MTRlY2MzZDhmNmI5MWI1OTQyYWVlY2QwZTk2MDVmZDQ3MmI2Y2I4ZmI5ZmM4ODFjZTg1NTNjZWFlZTA5YzhjYTFlOWFlOTA4OTBiZTI3Yjk="} }
        service: s3
        path: d3e07840-8c82-4537-8079-5de3fbfe19ad !@#$%^&()_+-=;',.txt
    
thread 'services_s3_write::test_delete_with_special_chars' panicked at 'write must succeed: PermissionDenied (permanent) at Writer::write => S3Error { code: "SignatureDoesNotMatch", message: "The Signature you specified is invalid.", resource: "/14173046-3958-431c-871a-b5cf419126ab/d3e07840-8c82-4537-8079-5de3fbfe19ad !@#$%^&()_+-=;',.txt", request_id: "NjQzZTdkMGNfNjkxMjI0MDlfYzU5MF80MjUwNmQ2" }

Context:
    response: Parts { status: 403, version: HTTP/1.1, headers: {"content-type": "application/xml", "content-length": "1091", "connection": "keep-alive", "date": "Tue, 18 Apr 2023 11:20:44 GMT", "server": "tencent-cos", "x-amz-request-id": "NjQzZTdkMGNfNjkxMjI0MDlfYzU5MF80MjUwNmQ2", "x-amz-trace-id": "OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODc0OWRkZjk0ZDM1NmI1M2E2MTRlY2MzZDhmNmI5MWI1OTQyYWVlY2QwZTk2MDVmZDQ3MmI2Y2I4ZmI5ZmM4ODFjZTg1NTNjZWFlZTA5YzhjYTFlOWFlOTA4OTBiZTI3Yjk="} }
    service: s3
    path: d3e07840-8c82-4537-8079-5de3fbfe19ad !@#$%^&()_+-=;',.txt
', core/tests/behavior/write.rs:642:36
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    services_s3_write::test_delete_with_special_chars

We will address it in the future.

@Xuanwo Xuanwo merged commit 72942cb into main Apr 18, 2023
@Xuanwo Xuanwo deleted the fix-cos-support branch April 18, 2023 12:27
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants