Skip to content

Commit 271a072

Browse files
authored
Merge pull request #10 from comet-ml/CRThaze/dev
Support multiple backends and fix workflow command
2 parents 026b7bd + 0db1d41 commit 271a072

File tree

11 files changed

+592
-199
lines changed

11 files changed

+592
-199
lines changed

.cursorrules

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Cursor Rules for S3Proxy Helm Chart Project
2+
3+
## Project Structure
4+
This is a Helm chart project for deploying S3Proxy to Kubernetes. The chart is located in `charts/s3proxy/`.
5+
6+
## Important File Handling Rules
7+
8+
### README.md Files - DO NOT EDIT DIRECTLY
9+
**CRITICAL:** The following README.md files are auto-generated and must NEVER be edited directly:
10+
- `/README.md` (root project README)
11+
- `/charts/s3proxy/README.md` (chart README, if it exists)
12+
13+
These files are generated from their corresponding template files using helm-docs.
14+
15+
**When asked to modify README content, you MUST:**
16+
1. ONLY edit the template file: `charts/s3proxy/README.md.gotmpl`
17+
2. NEVER directly edit any `README.md` files
18+
3. Inform the user that README.md is auto-generated and changes should be made to the template
19+
4. After editing the template, remind the user to regenerate README.md using helm-docs
20+
21+
**Template file location:**
22+
- Template: `charts/s3proxy/README.md.gotmpl`
23+
- Generated: `README.md` (root) - auto-generated from the template
24+
25+
## Helm Chart Best Practices
26+
- Always validate YAML syntax when editing chart files
27+
- Preserve the existing indentation style (spaces for YAML files as per YAML spec)
28+
- When modifying values.yaml, ensure documentation comments are kept in sync
29+
- Test template rendering with `helm template` after making changes

charts/s3proxy/Chart.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ type: application
1515
# This is the chart version. This version number should be incremented each time you make changes
1616
# to the chart and its templates, including the app version.
1717
# Versions are expected to follow Semantic Versioning (https://semver.org/)
18-
version: 0.0.3
18+
version: 0.0.4
1919

2020
# This is the version number of the application being deployed. This version number should be
2121
# incremented each time you make changes to the application. Versions are not expected to

charts/s3proxy/README.md.gotmpl

Lines changed: 27 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -53,11 +53,14 @@ The following section lists the configurable parameters of the {{ template "char
5353
```yaml
5454
# values-filesystem.yaml
5555
config:
56-
authorization: "aws-v4"
57-
identity: "myaccesskey"
58-
credential: "mysecretkey"
59-
backend:
60-
provider: "filesystem-nio2"
56+
auth:
57+
type: "aws-v4"
58+
identity: "myaccesskey"
59+
secret: "mysecretkey"
60+
backends:
61+
filesystem:
62+
enabled: true
63+
nio2: true
6164
filesystem:
6265
basedir: "/data/s3proxy"
6366

@@ -76,9 +79,10 @@ helm install s3proxy-fs ./{{ template "chart.name" . }} -f values-filesystem.yam
7679
```yaml
7780
# values-aws-s3.yaml
7881
config:
79-
authorization: "aws-v4"
80-
identity: "proxy-access-key" # For clients connecting to s3proxy
81-
credential: "proxy-secret-key"
82+
auth:
83+
type: "aws-v4"
84+
identity: "proxy-access-key" # For clients connecting to s3proxy
85+
secret: "proxy-secret-key"
8286
backend:
8387
provider: "aws-s3"
8488
awsS3:
@@ -100,9 +104,10 @@ helm install s3proxy-s3 ./{{ template "chart.name" . }} -f values-aws-s3.yaml
100104
```yaml
101105
# values-azure.yaml
102106
config:
103-
authorization: "aws-v4"
104-
identity: "myaccesskey"
105-
credential: "mysecretkey"
107+
auth:
108+
type: "aws-v4"
109+
identity: "myaccesskey"
110+
secret: "mysecretkey"
106111
backend:
107112
provider: "azureblob"
108113
azureblob:
@@ -123,9 +128,10 @@ helm install s3proxy-azure ./{{ template "chart.name" . }} -f values-azure.yaml
123128
```yaml
124129
# values-gcs.yaml
125130
config:
126-
authorization: "aws-v4"
127-
identity: "myaccesskey"
128-
credential: "mysecretkey"
131+
auth:
132+
type: "aws-v4"
133+
identity: "myaccesskey"
134+
secret: "mysecretkey"
129135
backend:
130136
provider: "google-cloud-storage"
131137
googleCloudStorage:
@@ -145,9 +151,12 @@ persistence:
145151
```yaml
146152
# values-anonymous.yaml
147153
config:
148-
authorization: "none"
149-
backend:
150-
provider: "transient-nio2" # In-memory storage
154+
auth:
155+
type: "none"
156+
backends:
157+
transient:
158+
enabled: true
159+
nio2: true # In-memory storage
151160

152161
persistence:
153162
enabled: false
@@ -263,7 +272,7 @@ This will remove all resources created by the chart. If using persistence, the P
263272

264273
### Common Issues
265274

266-
1. **Authentication failures**: Ensure `config.identity` and `config.credential` are set correctly for client authentication.
275+
1. **Authentication failures**: Ensure `config.auth.identity` and `config.auth.secret` are set correctly for client authentication.
267276

268277
2. **Backend connection issues**: Verify backend credentials are correctly configured in the appropriate section (e.g., `config.backend.awsS3.*`).
269278

charts/s3proxy/override-values.example.yaml

Lines changed: 109 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,31 @@
11
# Example values for s3proxy Helm chart
22
# This demonstrates a typical configuration using filesystem backend with authentication
33

4+
# Custom image configuration (optional)
5+
# image:
6+
# repository: andrewgaul/s3proxy
7+
# tag: "latest"
8+
# pullPolicy: Always
9+
10+
# Custom config merge container image configuration (optional)
11+
# configMergeImage:
12+
# repository: busybox
13+
# tag: "1.36"
14+
# pullPolicy: IfNotPresent
15+
416
# S3Proxy configuration
517
config:
6-
# Authentication type for clients connecting to S3Proxy
7-
# Options: none, aws-v2, aws-v4, aws-v2-or-v4
8-
authorization: "aws-v4"
9-
10-
# Credentials for clients to authenticate to S3Proxy
11-
# These should be changed to secure values
12-
identity: "admin"
13-
credential: "changeme123"
18+
# Log level for S3Proxy (DEBUG, INFO, WARN, ERROR)
19+
logLevel: "INFO"
20+
21+
auth:
22+
# Authentication type for clients connecting to S3Proxy
23+
# Options: none, aws-v2, aws-v4, aws-v2-or-v4
24+
type: "aws-v4"
25+
# Credentials for clients to authenticate to S3Proxy
26+
# These should be changed to secure values
27+
identity: "admin"
28+
secret: "changeme123"
1429

1530
# Enable CORS for web applications
1631
cors:
@@ -30,24 +45,95 @@ config:
3045
allowCredential: true
3146

3247
# Storage backend configuration
33-
backend:
34-
# Use filesystem backend for local storage
35-
provider: "filesystem-nio2"
48+
# Multiple backends can be enabled simultaneously
49+
# Properties files will be loaded in order: main properties first, then each backend's properties
50+
# Later properties can override earlier ones if there are conflicts
51+
backends:
52+
# Filesystem backend for local storage
3653
filesystem:
54+
enabled: true # Set to true to use filesystem backend
55+
nio2: true # Set to true for NIO2 implementation (filesystem-nio2), false for standard (filesystem)
3756
basedir: "/data/s3proxy"
3857

39-
# Example: AWS S3 backend (uncomment to use)
40-
# provider: "aws-s3"
41-
# awsS3:
42-
# region: "us-west-2"
43-
# accessKeyId: "AKIAIOSFODNN7EXAMPLE"
44-
# secretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
45-
46-
# Example: Azure Blob backend (uncomment to use)
47-
# provider: "azureblob"
48-
# azureblob:
49-
# account: "mystorageaccount"
50-
# key: "base64encodedkey=="
58+
# Transient (in-memory) backend - useful for testing
59+
transient:
60+
enabled: false # Set to true to use transient backend
61+
nio2: true # Set to true for NIO2 implementation (transient-nio2), false for standard (transient)
62+
63+
# S3 backend (AWS S3 or S3-compatible storage)
64+
s3:
65+
enabled: false # Set to true to use S3 backend
66+
aws: true # Set to true for AWS S3 (aws-s3 provider), false for generic S3
67+
region: "us-west-2"
68+
# endpoint: "https://s3.amazonaws.com" # Optional custom endpoint (e.g., MinIO, Ceph)
69+
accessKeyId: "AKIAIOSFODNN7EXAMPLE"
70+
secretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
71+
72+
# Azure Blob Storage backend
73+
azureblob:
74+
enabled: false # Set to true to use Azure Blob backend
75+
provider: "azureblob" # Can be "azureblob" or "azureblob-sdk"
76+
account: "mystorageaccount"
77+
key: "base64encodedkey=="
78+
# endpoint: "https://mystorageaccount.blob.core.windows.net" # Optional
79+
# sasToken: "" # Optional SAS token
80+
81+
# Google Cloud Storage backend
82+
googleCloudStorage:
83+
enabled: false # Set to true to use GCS backend
84+
projectId: "my-project"
85+
86+
# Service account email or user email (required for both authentication methods)
87+
clientEmail: "[email protected]"
88+
89+
# Option 1: Using privateKey directly
90+
privateKey: "-----BEGIN RSA PRIVATE KEY-----\n..."
91+
92+
# Option 2: Using JSON credentials file (preferred for GCP)
93+
# This provides the credential (privateKey) via a mounted file
94+
jsonCredentials:
95+
enabled: false # Set to true to use JSON credentials
96+
97+
# Either provide the JSON content directly:
98+
# jsonContent: |
99+
# {
100+
# "type": "service_account",
101+
# "project_id": "my-project",
102+
# "private_key_id": "key-id",
103+
# "private_key": "-----BEGIN RSA PRIVATE KEY-----\n...",
104+
# "client_email": "[email protected]",
105+
# "client_id": "...",
106+
# "auth_uri": "https://accounts.google.com/o/oauth2/auth",
107+
# "token_uri": "https://oauth2.googleapis.com/token",
108+
# "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
109+
# "client_x509_cert_url": "..."
110+
# }
111+
112+
# Or reference an existing secret containing the JSON:
113+
# existingSecret: "my-gcp-credentials-secret"
114+
# secretKey: "credentials.json" # Key in the secret containing the JSON (default: credentials.json)
115+
116+
# Backblaze B2 backend
117+
b2:
118+
enabled: false # Set to true to use B2 backend
119+
account: "account-id"
120+
applicationKey: "application-key"
121+
122+
# OpenStack Swift backend
123+
openstackSwift:
124+
enabled: false # Set to true to use Swift backend
125+
authUrl: "https://auth.cloud.com/v2.0"
126+
tenantName: "my-tenant"
127+
userName: "my-user"
128+
password: "my-password"
129+
region: "RegionOne"
130+
131+
# Rackspace Cloud Files backend
132+
rackspaceCloudfiles:
133+
enabled: false # Set to true to use Rackspace Cloud Files backend
134+
region: "us" # Region: "us" or "uk"
135+
userName: "my-user"
136+
apiKey: "my-api-key"
51137

52138
# Persistence settings for filesystem backend
53139
persistence:

charts/s3proxy/templates/NOTES.txt

Lines changed: 54 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,41 @@
2222
{{- end }}
2323

2424
2. S3Proxy Configuration:
25-
- Backend Provider: {{ .Values.config.backend.provider }}
26-
- Authorization: {{ .Values.config.authorization }}
25+
- Backend Provider:
26+
{{- if .Values.config.backends.filesystem.enabled }}
27+
{{- if .Values.config.backends.filesystem.nio2 }}
28+
filesystem (filesystem-nio2)
29+
{{- else }}
30+
filesystem (filesystem)
31+
{{- end }}
32+
{{- else if .Values.config.backends.transient.enabled }}
33+
{{- if .Values.config.backends.transient.nio2 }}
34+
transient (transient-nio2)
35+
{{- else }}
36+
transient (transient)
37+
{{- end }}
38+
{{- else if .Values.config.backends.s3.enabled }}
39+
{{- if .Values.config.backends.s3.aws }}
40+
AWS S3 (aws-s3)
41+
{{- else }}
42+
Generic S3 (s3)
43+
{{- end }}
44+
{{- else if .Values.config.backends.azureblob.enabled }}
45+
Azure Blob ({{ .Values.config.backends.azureblob.provider }})
46+
{{- else if .Values.config.backends.googleCloudStorage.enabled }}
47+
Google Cloud Storage
48+
{{- else if .Values.config.backends.b2.enabled }}
49+
Backblaze B2
50+
{{- else if .Values.config.backends.openstackSwift.enabled }}
51+
OpenStack Swift
52+
{{- else if .Values.config.backends.rackspaceCloudfiles.enabled }}
53+
{{- if eq .Values.config.backends.rackspaceCloudfiles.region "uk" }}
54+
Rackspace Cloud Files UK
55+
{{- else }}
56+
Rackspace Cloud Files US
57+
{{- end }}
58+
{{- end }}
59+
- Authorization: {{ .Values.config.auth.type }}
2760
{{- if .Values.config.cors.enabled }}
2861
- CORS: Enabled
2962
{{- end }}
@@ -34,13 +67,13 @@
3467
3. Testing S3Proxy:
3568
Once the service is available, you can test it using AWS CLI or curl:
3669

37-
{{- if eq .Values.config.authorization "none" }}
70+
{{- if eq .Values.config.auth.type "none" }}
3871
# Anonymous access (no authentication required)
3972
curl -X GET http://<SERVICE_URL>/
4073
{{- else }}
4174
# Set up AWS CLI credentials (if authentication is enabled)
4275
export AWS_ACCESS_KEY_ID=<your-s3proxy-identity>
43-
export AWS_SECRET_ACCESS_KEY=<your-s3proxy-credential>
76+
export AWS_SECRET_ACCESS_KEY=<your-s3proxy-secret>
4477

4578
# List buckets
4679
aws --endpoint-url http://<SERVICE_URL> s3 ls
@@ -53,7 +86,7 @@
5386

5487

5588
# You can also use other AWS S3 API compatible clients like MinIO's mc.
56-
mc alias set s3proxy http://<SERVICE_URL> <your-s3proxy-identity> <your-s3proxy-credential>
89+
mc alias set s3proxy http://<SERVICE_URL> <your-s3proxy-identity> <your-s3proxy-secret>
5790

5891
# List buckets
5992
mc ls s3proxy
@@ -70,12 +103,22 @@
70103
kubectl logs -n {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "s3proxy.name" . }},app.kubernetes.io/instance={{ .Release.Name }}"
71104

72105
5. Important Notes:
73-
{{- if or (eq .Values.config.backend.provider "filesystem") (eq .Values.config.backend.provider "filesystem-nio2") }}
74-
- Using filesystem backend at: {{ .Values.config.backend.filesystem.basedir }}
106+
{{- if .Values.config.backends.filesystem.enabled }}
107+
- Using filesystem backend at: {{ .Values.config.backends.filesystem.basedir }}
75108
{{- end }}
76-
{{- if not .Values.config.identity }}
77-
- WARNING: S3Proxy identity not configured. Remember to set s3proxy.identity and s3proxy.credential for authentication.
109+
{{- if not .Values.config.auth.identity }}
110+
- WARNING: S3Proxy identity not configured. Remember to set config.auth.identity and config.auth.secret for authentication.
78111
{{- end }}
79-
{{- if and (ne .Values.config.backend.provider "filesystem") (ne .Values.config.backend.provider "filesystem-nio2") (ne .Values.config.backend.provider "transient") (ne .Values.config.backend.provider "transient-nio2") }}
80-
- Make sure to configure backend credentials for {{ .Values.config.backend.provider }}
112+
{{- if .Values.config.backends.s3.enabled }}
113+
- Make sure to configure S3 backend credentials
114+
{{- else if .Values.config.backends.azureblob.enabled }}
115+
- Make sure to configure Azure Blob backend credentials
116+
{{- else if .Values.config.backends.googleCloudStorage.enabled }}
117+
- Make sure to configure Google Cloud Storage backend credentials
118+
{{- else if .Values.config.backends.b2.enabled }}
119+
- Make sure to configure Backblaze B2 backend credentials
120+
{{- else if .Values.config.backends.openstackSwift.enabled }}
121+
- Make sure to configure OpenStack Swift backend credentials
122+
{{- else if .Values.config.backends.rackspaceCloudfiles.enabled }}
123+
- Make sure to configure Rackspace Cloud Files backend credentials
81124
{{- end }}

0 commit comments

Comments
 (0)