Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

feat: Manage S3 list pagination #125

Merged
merged 4 commits into from
Jan 14, 2021
Merged

feat: Manage S3 list pagination #125

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
766f56b
feat: Manage S3 list pagination
oxyno-zeta Jan 13, 2021
62d1610
chore: Add setup tracing-services for dev usages
oxyno-zeta Jan 13, 2021
534e3d8
docs: Upgrade documentation for s3 list max keys
oxyno-zeta Jan 13, 2021
454ba21
chore: Upgrade all dependencies
oxyno-zeta Jan 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,10 @@ down/services:
docker rm -f opa || true
docker rm -f keycloak || true

.PHONY: down/tracing-services
down/tracing-services:
docker rm -f jaeger || true

.PHONY: down/metrics-services
down/metrics-services:
docker rm -f prometheus || true
Expand All @@ -113,6 +117,11 @@ setup/metrics-services:
docker run --rm -d --name prometheus -v $(CURRENT_DIR)/local-resources/prometheus/prometheus.yml:/prometheus/prometheus.yml --network=host prom/prometheus:v2.18.0 --web.listen-address=:9191
docker run --rm -d --name grafana --network=host grafana/grafana:7.0.3

.PHONY: setup/tracing-services
setup/tracing-services: down/tracing-services
@echo "Setup tracing services"
docker run --name jaeger -d -p 6831:6831/udp -p 16686:16686 jaegertracing/all-in-one:latest

.PHONY: setup/services
setup/services: down/services
tar czvf local-resources/opa/bundle.tar.gz --directory=local-resources/opa/bundle example/
Expand Down
1 change: 1 addition & 0 deletions conf/config-example.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,7 @@ targets:
region: eu-west-1
s3Endpoint:
disableSSL: false
# s3ListMaxKeys: 1000
# credentials:
# accessKey:
# env: AWS_ACCESS_KEY_ID
Expand Down
18 changes: 10 additions & 8 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,14 +132,15 @@ This feature is powered by [go-chi/cors](https://github.com/go-chi/cors). You ca

## BucketConfiguration

| Key | Type | Required | Default | Description |
| ----------- | --------------------------------------------------------------- | -------- | ----------- | ---------------------------------------- |
| name | String | Yes | None | Bucket name in S3 provider |
| prefix | String | No | None | Bucket prefix |
| region | String | No | `us-east-1` | Bucket region |
| s3Endpoint | String | No | None | Custom S3 Endpoint for non AWS S3 bucket |
| credentials | [BucketCredentialConfiguration](#bucketcredentialconfiguration) | No | None | Credentials to access S3 bucket |
| disableSSL | Boolean | No | `false` | Disable SSL connection |
| Key | Type | Required | Default | Description |
| ------------- | --------------------------------------------------------------- | -------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| name | String | Yes | None | Bucket name in S3 provider |
| prefix | String | No | None | Bucket prefix |
| region | String | No | `us-east-1` | Bucket region |
| s3Endpoint | String | No | None | Custom S3 Endpoint for non AWS S3 bucket |
| credentials | [BucketCredentialConfiguration](#bucketcredentialconfiguration) | No | None | Credentials to access S3 bucket |
| disableSSL | Boolean | No | `false` | Disable SSL connection |
| s3ListMaxKeys | Integer | No | `1000` | This flag will be used for the max pagination list management of files and "folders" in S3. In S3 list requests, the limit is fixed to 1000 items maximum. S3-Proxy will allow to increase this by making multiple requests to S3. Warning: This will increase the memory and CPU usage. |

## BucketCredentialConfiguration

Expand Down Expand Up @@ -470,6 +471,7 @@ targets:
region: eu-west-1
s3Endpoint:
disableSSL: false
# s3ListMaxKeys: 1000
# credentials:
# accessKey:
# env: AWS_ACCESS_KEY_ID
Expand Down
24 changes: 12 additions & 12 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ module github.com/oxyno-zeta/s3-proxy
go 1.13

require (
github.com/HdrHistogram/hdrhistogram-go v0.9.0 // indirect
github.com/HdrHistogram/hdrhistogram-go v1.0.1 // indirect
github.com/Masterminds/goutils v1.1.0 // indirect
github.com/Masterminds/semver v1.5.0 // indirect
github.com/Masterminds/sprig v2.22.0+incompatible
github.com/aws/aws-sdk-go v1.32.6
github.com/aws/aws-sdk-go v1.36.26
github.com/coreos/go-oidc v2.2.1+incompatible
github.com/dimiro1/health v0.0.0-20191019130555-c5cbb4d46ffc
github.com/dustin/go-humanize v1.0.0
Expand All @@ -18,29 +18,29 @@ require (
github.com/go-playground/validator/v10 v10.4.1
github.com/gobwas/glob v0.2.3
github.com/golang/mock v1.4.4
github.com/google/uuid v1.1.1 // indirect
github.com/huandu/xstrings v1.3.0 // indirect
github.com/imdario/mergo v0.3.8 // indirect
github.com/google/uuid v1.1.4 // indirect
github.com/huandu/xstrings v1.3.2 // indirect
github.com/imdario/mergo v0.3.11 // indirect
github.com/johannesboyne/gofakes3 v0.0.0-20200716060623-6b2b4cb092cc
github.com/mitchellh/copystructure v1.0.0 // indirect
github.com/mitchellh/reflectwalk v1.0.1 // indirect
github.com/opentracing/opentracing-go v1.2.0
github.com/pelletier/go-toml v1.6.0 // indirect
github.com/pquerna/cachecontrol v0.0.0-20180517163645-1555304b9b35 // indirect
github.com/pelletier/go-toml v1.8.1 // indirect
github.com/pquerna/cachecontrol v0.0.0-20201205024021-ac21108117ac // indirect
github.com/prometheus/client_golang v1.9.0
github.com/sirupsen/logrus v1.7.0
github.com/spf13/afero v1.2.2 // indirect
github.com/spf13/afero v1.5.1 // indirect
github.com/spf13/cast v1.3.1 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/spf13/viper v1.7.1
github.com/stretchr/testify v1.6.1
github.com/stretchr/testify v1.7.0
github.com/thoas/go-funk v0.7.0
github.com/uber/jaeger-client-go v2.25.0+incompatible
github.com/uber/jaeger-lib v2.4.0+incompatible
golang.org/x/net v0.0.0-20201224014010-6772e930b67b
golang.org/x/oauth2 v0.0.0-20201208152858-08078c50e5b5
golang.org/x/oauth2 v0.0.0-20210113205817-d3ed898aa8a3
golang.org/x/sync v0.0.0-20201207232520-09787c993a3a
gopkg.in/ini.v1 v1.52.0 // indirect
gopkg.in/square/go-jose.v2 v2.4.1 // indirect
gopkg.in/ini.v1 v1.62.0 // indirect
gopkg.in/square/go-jose.v2 v2.5.1 // indirect
)
Loading