indexDocument not handled when bucket contains > 1k objects

[chiefy]

Describe the bug
indexDocument doesn't work when folder contains large number of files before i

To Reproduce
Steps to reproduce the behavior:

1. create s3 bucket with hundreds of files - mostly numbered, but with one index.html
2. create target config to that bucket etc.
3. page will display index of bucket up to about page 3-4/5, but not the actual index.html

Expected behavior
Proxy serves index.html

Screenshots
If applicable, add screenshots to help explain your problem.

Version and platform (please complete the following information):

• Platform Linux
• Arch: x86_64
• Version latest docker

[chiefy]

Looking some more into this, our bucket has 1.4k objects, and it looks like by default the s3 client will only return MaxKeys of 1k by default

[oxyno-zeta]

Hello @chiefy !

Thanks for this issue and all informations you give me. You are right. I must admit that I didn't though that this project will be used for this kind of huge bucket 😄 .

I will work on a quick fix within next days (I have it in mind, need to be tested) and I probably found a solution to allow more results to be served by allowing people to configure the max results limit they want (to avoid looping over all elements and eating all memory !).

I will keep you posted.

Regards,

Oxyno-zeta

[chiefy]

@oxyno-zeta thanks. As it turns out our CI is publishing stuff to s3 without cleaning the bucket beforehand. So I'd say it's an edge-case.

[oxyno-zeta]

This should be fixed with the 3.1.1 release 🎉

[oxyno-zeta]

Hello, Can you confirm or not that the issue is fixed ? Just to know I need to think about another solution / another fix 😅 . Thanks !

[oxyno-zeta]

This should be ok and now you can manage the pagination if needed. I'm closing this :)