Skip to content

Commit

Permalink
feat(bucket): Add templating on PUT metadata and storage class
Browse files Browse the repository at this point in the history
  • Loading branch information
@oxyno-zeta
oxyno-zeta committed Jul 3, 2021
1 parent 178cb61 commit 7fe1f2c
Show file tree
Hide file tree
Showing 9 changed files with 389 additions and 15 deletions.
4 changes: 3 additions & 1 deletion conf/config-example.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -232,11 +232,13 @@ targets:
# enabled: true
# # Configuration for PUT requests
# config:
# # Metadata key/values that will be put on S3 objects
# # Metadata key/values that will be put on S3 objects.
# # Values can be templated. Empty values will be flushed.
# metadata:
# key: value
# # Storage class that will be used for uploaded objects
# # See storage class here: https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html
# # Values can be templated. Empty values will be flushed.
# storageClass: STANDARD # GLACIER, ...
# # Will allow override objects if enabled
# allowOverride: false
Expand Down
4 changes: 3 additions & 1 deletion docs/configuration/example.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -242,11 +242,13 @@ targets:
# enabled: true
# # Configuration for PUT requests
# config:
# # Metadata key/values that will be put on S3 objects
# # Metadata key/values that will be put on S3 objects.
# # Values can be templated. Empty values will be flushed.
# metadata:
# key: value
# # Storage class that will be used for uploaded objects
# # See storage class here: https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html
# # Values can be templated. Empty values will be flushed.
# storageClass: STANDARD # GLACIER, ...
# # Will allow override objects if enabled
# allowOverride: false
Expand Down
10 changes: 5 additions & 5 deletions docs/configuration/structure.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -178,11 +178,11 @@ See more information [here](../feature-guide/key-rewrite.md).

## PutActionConfigConfiguration

| Key | Type | Required | Default | Description |
| ------------- | ----------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| metadata | Map[String]String | No | None | Metadata key/values that will be put on S3 objects |
| storageClass | String | No | `""` | Storage class that will be used for uploaded objects. See storage class here: [https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html](https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html) |
| allowOverride | Boolean | No | `false` | Will allow override objects if enabled |
| Key | Type | Required | Default | Description |
| ------------- | ----------------- | -------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| metadata | Map[String]String | No | None | Metadata key/values that will be put on S3 objects. Map Values can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |
| storageClass | String | No | `""` | Storage class that will be used for uploaded objects. See storage class here: [https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html](https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html). Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-storage-class) |
| allowOverride | Boolean | No | `false` | Will allow override objects if enabled |

## DeleteActionConfiguration

Expand Down
37 changes: 36 additions & 1 deletion docs/feature-guide/templates.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -123,11 +123,37 @@ Available data:
| Request | [http.Request](https://golang.org/pkg/net/http/#Request) | HTTP Request object from golang |
| Error | Error | Error raised and caught |

## PUT Metadata and Storage class

### PUT Metadata

This case will be used for all PUT metadata templates.

Available data:

| Name | Type | Description |
| ----- | --------------------------- | ------------------------------------------------- |
| User | [GenericUser](#genericuser) | Authenticated user if present in incoming request |
| Input | [PutInput](#putinput) | PutInput structure data |
| Key | String | The final S3 key generated for upload request |

### PUT Storage class

This case will be used for all PUT storage class templates.

Available data:

| Name | Type | Description |
| ----- | --------------------------- | ------------------------------------------------- |
| User | [GenericUser](#genericuser) | Authenticated user if present in incoming request |
| Input | [PutInput](#putinput) | PutInput structure data |
| Key | String | The final S3 key generated for upload request |

## Headers templates and structures

### Generic case

This case is the main one and used for all templates rendered explained before.
This case is the main one and used for all header templates rendered explained before.

The following table will show the data structure available for the header template rendering:

Expand Down Expand Up @@ -195,3 +221,12 @@ These are the properties available:
| ETag | String | ETag value from S3 |
| LastModified | [Time](https://golang.org/pkg/time/#Time) | Last modified value from S3 |
| Metadata  | Map[String]String | Metadata value from S3 |

### PutInput

| Name | Type | Description |
| ----------- | ------- | ---------------------------- |
| RequestPath | String | Request path |
| Filename | String | Filename used for upload |
| ContentType | String | File content type for upload |
| ContentSize | Integer | File content size for upload |
8 changes: 8 additions & 0 deletions pkg/s3-proxy/bucket/client.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"io"
"time"

"github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/authx/models"
"github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/config"
"github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/s3client"
)
Expand Down Expand Up @@ -45,6 +46,13 @@ type PutInput struct {
ContentSize int64
}

// PutData Put Data represents a put data structure used in put templates rendering.
type PutData struct {
User models.GenericUser
Input *PutInput
Key string
}

// NewClient will generate a new client to do GET,PUT or DELETE actions.
func NewClient(
tgt *config.TargetConfig,
Expand Down
58 changes: 56 additions & 2 deletions pkg/s3-proxy/bucket/requestContext.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,11 @@ import (
"path"
"strings"

"github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/authx/models"
"github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/config"
responsehandler "github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/response-handler"
"github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/s3client"
"github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/utils"
)

// requestContext Bucket request context.
Expand Down Expand Up @@ -218,12 +220,64 @@ func (rctx *requestContext) Put(ctx context.Context, inp *PutInput) {
rctx.targetCfg.Actions.PUT.Config != nil {
// Check if metadata is configured in target configuration
if rctx.targetCfg.Actions.PUT.Config.Metadata != nil {
input.Metadata = rctx.targetCfg.Actions.PUT.Config.Metadata
// Store templated data
metadata := map[string]string{}

// Render templates
for k, v := range rctx.targetCfg.Actions.PUT.Config.Metadata {
// Execute template
buf, err := utils.ExecuteTemplate(v, &PutData{
User: models.GetAuthenticatedUserFromContext(ctx),
Input: inp,
Key: key,
})
// Check error
if err != nil {
resHan.InternalServerError(rctx.LoadFileContent, err)

return
}

// Store value
val := buf.String()
// Remove all new lines
val = utils.NewLineMatcherRegex.ReplaceAllString(val, "")
// Check if value is empty or not
if val != "" {
// Store
metadata[k] = val
}
}

// Store all metadata
input.Metadata = metadata
}

// Check if storage class is present in target configuration
if rctx.targetCfg.Actions.PUT.Config.StorageClass != "" {
input.StorageClass = rctx.targetCfg.Actions.PUT.Config.StorageClass
// Execute template
buf, err := utils.ExecuteTemplate(rctx.targetCfg.Actions.PUT.Config.StorageClass, &PutData{
User: models.GetAuthenticatedUserFromContext(ctx),
Input: inp,
Key: key,
})

// Check error
if err != nil {
resHan.InternalServerError(rctx.LoadFileContent, err)

return
}

// Store value
val := buf.String()
// Remove all new lines
val = utils.NewLineMatcherRegex.ReplaceAllString(val, "")
// Check if value is empty or not
if val != "" {
// Store
input.StorageClass = val
}
}

// Check if allow override is enabled
Expand Down
Loading

0 comments on commit 7fe1f2c

Please # to comment.