Intermittent failure with object_store aws::tests::s3_test emulator

[alamb]

Describe the bug

aws::tests::s3_test emulator CI tests are failing on main

To Reproduce

https://github.com/apache/arrow-rs/actions/runs/12672198886/job/35315696126

---- aws::tests::s3_test stdout ----
thread 'aws::tests::s3_test' panicked at src/integration.rs:960:5:
assertion failed: content_list.is_empty()
stack backtrace:
   0: rust_begin_unwind
             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt
             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:74:14
   2: core::panicking::panic
             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:148:5
   3: object_store::integration::list_with_delimiter::{{closure}}
             at ./src/integration.rs:960:5
   4: object_store::aws::tests::s3_test::{{closure}}
             at ./src/aws/mod.rs:569:43
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/future/future.rs:123:9
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/future/future.rs:123:9
   7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:729:57
   8: tokio::runtime::coop::with_budget
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/coop.rs:107:5
   9: tokio::runtime::coop::budget
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/coop.rs:73:5
  10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:729:25
  11: tokio::runtime::scheduler::current_thread::Context::enter
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:428:19
  12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:728:36
  13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:807:68
  14: tokio::runtime::context::scoped::Scoped<T>::set
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/context/scoped.rs:40:9
  15: tokio::runtime::context::set_scheduler::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/context.rs:180:26
  16: std::thread::local::LocalKey<T>::try_with
             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/thread/local.rs:283:12
  17: std::thread::local::LocalKey<T>::with
             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/thread/local.rs:260:9
  18: tokio::runtime::context::set_scheduler
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/context.rs:180:9
  19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:807:27
  20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:716:19
  21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:196:28
  22: tokio::runtime::context::runtime::enter_runtime
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/context/runtime.rs:65:16
  23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:184:9
  24: tokio::runtime::runtime::Runtime::block_on_inner
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/runtime.rs:368:47
  25: tokio::runtime::runtime::Runtime::block_on
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/runtime.rs:342:13
  26: object_store::aws::tests::s3_test
             at ./src/aws/mod.rs:610:9
  27: object_store::aws::tests::s3_test::{{closure}}
             at ./src/aws/mod.rs:[557](https://github.com/apache/arrow-rs/actions/runs/12672198886/job/35315696126#step:8:558):23
  28: core::ops::function::FnOnce::call_once
             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/ops/function.rs:250:5
  29: core::ops::function::FnOnce::call_once
             at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.


failures:
    aws::tests::s3_test

test result: FAILED. 145 passed; 1 failed; 4 ignored; 0 measured; 0 filtered out; finished in 15.49s

Expected behavior
Test should pass

Additional context
It appears to have first failed with this PR:

• Return BoxStream with 'static lifetime from ObjectStore::list arrow-rs#6619

However, the tests passed when run on that PR ( https://github.com/apache/arrow-rs/actions/runs/11842800687/job/33002269831) which suggests to me this is related to some change in the environment / emulator rather than our code

The test passed a day before on main

• https://github.com/apache/arrow-rs/actions/runs/12640445514/job/35220876783

[alamb]

There is an example from this PR where where the emulator test passes on a rerun

• Use u64 range instead of usize, for better wasm32 support arrow-rs#6961
• Failure: https://github.com/apache/arrow-rs/actions/runs/12770295148/job/35594807418?pr=6961
• Success (same code): https://github.com/apache/arrow-rs/actions/runs/12770295148/job/35607985446?pr=6961

[alamb]

Migrating from arrow-rs issue #6981