Skip to content

Commit 089d6f4

Browse files
argos83argos83
committed
feat: support request headers as S3 object metadata in PUT templates
Add RequestHeaders (http.Header) to PutInput so that PUT metadata and storage class templates can access incoming request headers via .Input.RequestHeaders.Get. The lookup is case-insensitive, matching Go's standard http.Header semantics. Multi-value headers can be accessed with index and combined using sprig's join function.
1 parent fc9628f commit 089d6f4

File tree

6 files changed

+123
-25
lines changed

6 files changed

+123
-25
lines changed

‎docs/configuration/example.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -397,8 +397,14 @@ targets:
397397
# # Values can be templated. Empty values will be flushed.
398398
# metadata:
399399
# key: value
400+
# # Request headers can be forwarded as S3 object metadata using .Input.RequestHeaders.Get.
401+
# # The header lookup is case-insensitive
402+
# # (e.g. "x-my-header" and "X-My-Header" both work).
403+
# uploaded-by: '{{ .Input.RequestHeaders.Get "X-Uploaded-By" }}'
404+
# correlation-id: '{{ .Input.RequestHeaders.Get "X-Correlation-Id" }}'
400405
# # System Metadata cases.
401406
# # Values can be templated. Empty values will be flushed.
407+
# # Request headers can also be forwarded using .Input.RequestHeaders.Get.
402408
# systemMetadata:
403409
# # Cache-Control value (will be put as header after)
404410
# cacheControl: ""
@@ -407,6 +413,8 @@ targets:
407413
# # Content-Encoding value (will be put as header after)
408414
# contentEncoding: ""
409415
# # Content-Language value (will be put as header after)
416+
# # Can be forwarded from the incoming request:
417+
# # contentLanguage: '{{ .Input.RequestHeaders.Get "Content-Language" }}'
410418
# contentLanguage: ""
411419
# # Expires value (will be put as header after)
412420
# # Side note: This must have the RFC3339 date format at the end.

‎docs/configuration/structure.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -279,8 +279,8 @@ See more information [here](../feature-guide/key-rewrite.md).
279279

280280
| Key | Type | Required | Default | Description |
281281
| -------------- | ----------------------------------------------------------------------------------------- | -------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
282-
| metadata | Map[String]String | No | None | Metadata key/values that will be put on S3 objects. Map Values can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |
283-
| systemMetadata | [PutActionConfigSystemMetadataConfiguration](#putactionconfigsystemmetadataconfiguration) | No | `nil` | This allow to put system metadata values to uploaded objects. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |
282+
| metadata | Map[String]String | No | None | Metadata key/values that will be put on S3 objects. Map Values can be templated. Empty values will be flushed. Incoming request headers are accessible in templates via `.Input.RequestHeaders`. See [here](../feature-guide/templates.md#put-metadata-and-system-metadata) |
283+
| systemMetadata | [PutActionConfigSystemMetadataConfiguration](#putactionconfigsystemmetadataconfiguration) | No | `nil` | This allow to put system metadata values to uploaded objects. Value can be templated. Empty values will be flushed. Incoming request headers are accessible in templates via `.Input.RequestHeaders`. See [here](../feature-guide/templates.md#put-metadata-and-system-metadata) |
284284
| storageClass | String | No | `""` | Storage class that will be used for uploaded objects. See storage class here: [https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html](https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html). Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-storage-class) |
285285
| allowOverride | Boolean | No | `false` | Will allow override objects if enabled |
286286
| cannedACL | String | No | `nil`  | Canned ACL put on each file uploaded. See official values here [https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl](https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl). |
@@ -290,11 +290,11 @@ See more information [here](../feature-guide/key-rewrite.md).
290290

291291
| Key | Type | Required | Default | Description |
292292
| ------------------ | ------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
293-
| cacheControl | String | `""` | Cache-Control value. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |
294-
| contentDisposition | String | `""` | Content-Disposition value. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |
295-
| contentEncoding | String | `""` | Content-Encoding value. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |
296-
| contentLanguage | String | `""` | Content-Language value. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |
297-
| expires | String | `""` | Expires value This must have the RFC3339 date format at the end. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |
293+
| cacheControl | String | `""` | Cache-Control value. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata-and-system-metadata) |
294+
| contentDisposition | String | `""` | Content-Disposition value. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata-and-system-metadata) |
295+
| contentEncoding | String | `""` | Content-Encoding value. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata-and-system-metadata) |
296+
| contentLanguage | String | `""` | Content-Language value. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata-and-system-metadata) |
297+
| expires | String | `""` | Expires value This must have the RFC3339 date format at the end. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata-and-system-metadata) |
298298

299299
## DeleteActionConfiguration
300300

‎docs/feature-guide/templates.md‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -223,9 +223,9 @@ Available for:
223223

224224
## PUT Metadata and Storage class
225225

226-
### PUT Metadata
226+
### PUT Metadata and System Metadata
227227

228-
This case will be used for all PUT metadata templates.
228+
This case will be used for all PUT metadata and system metadata templates (`metadata` and `systemMetadata` fields).
229229

230230
Available data:
231231

@@ -298,12 +298,13 @@ These are the properties available:
298298

299299
### PutInput
300300

301-
| Name | Type | Description |
302-
| ----------- | ------- | ---------------------------- |
303-
| RequestPath | String | Request path |
304-
| Filename | String | Filename used for upload |
305-
| ContentType | String | File content type for upload |
306-
| ContentSize | Integer | File content size for upload |
301+
| Name | Type | Description |
302+
| -------------- | ----------------- | ---------------------------- |
303+
| RequestPath | String | Request path |
304+
| Filename | String | Filename used for upload |
305+
| ContentType | String | File content type for upload |
306+
| ContentSize | Integer | File content size for upload |
307+
| RequestHeaders | [http.Header](https://golang.org/pkg/net/http/#Header) | Incoming request headers |
307308

308309
### PutData
309310

‎pkg/s3-proxy/bucket/bucket-req-impl_test.go‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1354,6 +1354,92 @@ func Test_requestContext_Put(t *testing.T) {
13541354
times: 1,
13551355
},
13561356
},
1357+
{
1358+
name: "should be ok to do templating on metadata using request headers",
1359+
fields: fields{
1360+
targetCfg: &config.TargetConfig{
1361+
Name: "name",
1362+
Bucket: &config.BucketConfig{Prefix: "/"},
1363+
Actions: &config.ActionsConfig{
1364+
PUT: &config.PutActionConfig{
1365+
Config: &config.PutActionConfigConfig{
1366+
Metadata: map[string]string{
1367+
"first-header-value-only-key": `{{ .Input.RequestHeaders.Get "X-Custom-Header" }}`,
1368+
"all-header-values-key": `{{ join "," (index .Input.RequestHeaders "X-Custom-Header") }}`,
1369+
"header-normalised-test-key": `{{ .Input.RequestHeaders.Get "x-CASE-is-NORMALISED" }}`,
1370+
},
1371+
AllowOverride: true,
1372+
},
1373+
},
1374+
},
1375+
},
1376+
mountPath: "/mount",
1377+
},
1378+
args: args{
1379+
inp: &PutInput{
1380+
RequestPath: "/test",
1381+
Filename: "file",
1382+
Body: nil,
1383+
ContentType: "content-type",
1384+
RequestHeaders: func() http.Header {
1385+
h := http.Header{}
1386+
h["X-Custom-Header"] = []string{"first-value", "second-value"}
1387+
h["X-Case-Is-Normalised"] = []string{"test-value"}
1388+
return h
1389+
}(),
1390+
},
1391+
},
1392+
s3ClientHeadObjectMockResult: s3ClientHeadObjectMockResult{
1393+
times: 0,
1394+
},
1395+
s3ClientPutObjectMockResult: s3ClientPutObjectMockResult{
1396+
input2: &s3client.PutInput{
1397+
Key: "/test/file",
1398+
ContentType: "content-type",
1399+
Metadata: map[string]string{
1400+
"first-header-value-only-key": "first-value",
1401+
"all-header-values-key": "first-value,second-value",
1402+
"header-normalised-test-key": "test-value",
1403+
},
1404+
},
1405+
res: &s3client.ResultInfo{
1406+
Bucket: "bucket",
1407+
Key: "key",
1408+
Region: "region",
1409+
S3Endpoint: "s3endpoint",
1410+
},
1411+
times: 1,
1412+
},
1413+
webhookManagerManagePutHooksMockResult: webhookManagerManagePutHooksMockResult{
1414+
input2: "name",
1415+
input3: "/test",
1416+
input4: &webhook.PutInputMetadata{
1417+
Filename: "file",
1418+
ContentType: "content-type",
1419+
},
1420+
input5: &webhook.S3Metadata{
1421+
Bucket: "bucket",
1422+
Key: "key",
1423+
Region: "region",
1424+
S3Endpoint: "s3endpoint",
1425+
},
1426+
times: 1,
1427+
},
1428+
s3clManagerClientForTargetMockInput: "name",
1429+
responseHandlerPutMockResultTimes: responseHandlerPutMockResult{
1430+
input: &responsehandlermodels.PutInput{
1431+
Key: "/test/file",
1432+
Filename: "file",
1433+
ContentType: "content-type",
1434+
Metadata: map[string]string{
1435+
"first-header-value-only-key": "first-value",
1436+
"all-header-values-key": "first-value,second-value",
1437+
"header-normalised-test-key": "test-value",
1438+
},
1439+
},
1440+
times: 1,
1441+
},
1442+
},
13571443
}
13581444
for _, tt := range tests {
13591445
t.Run(tt.name, func(t *testing.T) {

‎pkg/s3-proxy/bucket/client.go‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ package bucket
33
import (
44
"context"
55
"io"
6+
"net/http"
67
"time"
78

89
"emperror.dev/errors"
@@ -44,11 +45,12 @@ type GetInput struct {
4445

4546
// PutInput represents Put input.
4647
type PutInput struct {
47-
RequestPath string
48-
Filename string
49-
Body io.ReadSeeker
50-
ContentType string
51-
ContentSize int64
48+
RequestPath string
49+
Filename string
50+
Body io.ReadSeeker
51+
ContentType string
52+
ContentSize int64
53+
RequestHeaders http.Header
5254
}
5355

5456
// PutData Put Data represents a put data structure used in put templates rendering.

‎pkg/s3-proxy/server/server.go‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -500,11 +500,12 @@ func (svr *Server) generateRouter() (http.Handler, error) {
500500

501501
// Create input for put request
502502
inp := &bucket.PutInput{
503-
RequestPath: requestPath,
504-
Filename: fileHeader.Filename,
505-
Body: file,
506-
ContentType: fileHeader.Header.Get("Content-Type"),
507-
ContentSize: fileHeader.Size,
503+
RequestPath: requestPath,
504+
Filename: fileHeader.Filename,
505+
Body: file,
506+
ContentType: fileHeader.Header.Get("Content-Type"),
507+
ContentSize: fileHeader.Size,
508+
RequestHeaders: req.Header,
508509
}
509510
// Action
510511
brctx.Put(req.Context(), inp)

0 commit comments

Comments
 (0)