Skip to content

[RFC-0010] Add multi-tenant workload identity support for AWS Bucket #1868

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
+149 −47
Open

[RFC-0010] Add multi-tenant workload identity support for AWS Bucket #1868

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3300705
[RFC-0010] Add multi-tenant workload identity support for AWS Bucket
cappyzawa Aug 13, 2025
794228f
fixup! [RFC-0010] Add multi-tenant workload identity support for AWS …
cappyzawa Aug 14, 2025
1b1fc21
fixup! [RFC-0010] Add multi-tenant workload identity support for AWS …
cappyzawa Aug 14, 2025
ef755b2
fixup! [RFC-0010] Add multi-tenant workload identity support for AWS …
cappyzawa Aug 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions api/v1/bucket_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@ const (
// BucketProviderGeneric for any S3 API compatible storage Bucket.
BucketProviderGeneric string = "generic"
// BucketProviderAmazon for an AWS S3 object storage Bucket.
// Provides support for retrieving credentials from the AWS EC2 service.
// Provides support for retrieving credentials from the AWS EC2 service
// and workload identity authentication.
BucketProviderAmazon string = "aws"
// BucketProviderGoogle for a Google Cloud Storage Bucket.
// Provides support for authentication using a workload identity.
Expand All @@ -51,7 +52,7 @@ const (
// +kubebuilder:validation:XValidation:rule="self.provider != 'generic' || !has(self.sts) || self.sts.provider == 'ldap'", message="'ldap' is the only supported STS provider for the 'generic' Bucket provider"
// +kubebuilder:validation:XValidation:rule="!has(self.sts) || self.sts.provider != 'aws' || !has(self.sts.secretRef)", message="spec.sts.secretRef is not required for the 'aws' STS provider"
// +kubebuilder:validation:XValidation:rule="!has(self.sts) || self.sts.provider != 'aws' || !has(self.sts.certSecretRef)", message="spec.sts.certSecretRef is not required for the 'aws' STS provider"
// +kubebuilder:validation:XValidation:rule="self.provider == 'gcp' || !has(self.serviceAccountName)", message="ServiceAccountName is only supported for the 'gcp' Bucket provider"
// +kubebuilder:validation:XValidation:rule="self.provider == 'gcp' || self.provider == 'aws' || !has(self.serviceAccountName)", message="ServiceAccountName is only supported for the 'gcp' and 'aws' Bucket providers"
// +kubebuilder:validation:XValidation:rule="!has(self.secretRef) || !has(self.serviceAccountName)", message="cannot set both .spec.secretRef and .spec.serviceAccountName"
type BucketSpec struct {
// Provider of the object storage bucket.
Expand Down Expand Up @@ -96,7 +97,8 @@ type BucketSpec struct {
SecretRef *meta.LocalObjectReference `json:"secretRef,omitempty"`

// ServiceAccountName is the name of the Kubernetes ServiceAccount used to authenticate
// the bucket. For more information about workload identity:
// the bucket. This field is only supported for the 'gcp' and 'aws' providers.
// For more information about workload identity:
// https://fluxcd.io/flux/components/source/buckets/#workload-identity
// +optional
ServiceAccountName string `json:"serviceAccountName,omitempty"`
Expand Down
8 changes: 5 additions & 3 deletions config/crd/bases/source.toolkit.fluxcd.io_buckets.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,8 @@ spec:
serviceAccountName:
description: |-
ServiceAccountName is the name of the Kubernetes ServiceAccount used to authenticate
the bucket. For more information about workload identity:
the bucket. This field is only supported for the 'gcp' and 'aws' providers.
For more information about workload identity:
https://fluxcd.io/flux/components/source/buckets/#workload-identity
type: string
sts:
Expand Down Expand Up @@ -238,8 +239,9 @@ spec:
rule: '!has(self.sts) || self.sts.provider != ''aws'' || !has(self.sts.secretRef)'
- message: spec.sts.certSecretRef is not required for the 'aws' STS provider
rule: '!has(self.sts) || self.sts.provider != ''aws'' || !has(self.sts.certSecretRef)'
- message: ServiceAccountName is only supported for the 'gcp' Bucket provider
rule: self.provider == 'gcp' || !has(self.serviceAccountName)
- message: ServiceAccountName is only supported for the 'gcp' and 'aws'
Bucket providers
rule: self.provider == 'gcp' || self.provider == 'aws' || !has(self.serviceAccountName)
- message: cannot set both .spec.secretRef and .spec.serviceAccountName
rule: '!has(self.secretRef) || !has(self.serviceAccountName)'
status:
Expand Down
6 changes: 4 additions & 2 deletions docs/api/v1/source.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,8 @@ string
<td>
<em>(Optional)</em>
<p>ServiceAccountName is the name of the Kubernetes ServiceAccount used to authenticate
the bucket. For more information about workload identity:
the bucket. This field is only supported for the &lsquo;gcp&rsquo; and &lsquo;aws&rsquo; providers.
For more information about workload identity:
<a href="https://fluxcd.io/flux/components/source/buckets/#workload-identity">https://fluxcd.io/flux/components/source/buckets/#workload-identity</a></p>
</td>
</tr>
Expand Down Expand Up @@ -1646,7 +1647,8 @@ string
<td>
<em>(Optional)</em>
<p>ServiceAccountName is the name of the Kubernetes ServiceAccount used to authenticate
the bucket. For more information about workload identity:
the bucket. This field is only supported for the &lsquo;gcp&rsquo; and &lsquo;aws&rsquo; providers.
For more information about workload identity:
<a href="https://fluxcd.io/flux/components/source/buckets/#workload-identity">https://fluxcd.io/flux/components/source/buckets/#workload-identity</a></p>
</td>
</tr>
Expand Down
49 changes: 49 additions & 0 deletions docs/spec/v1/buckets.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,55 @@ data:
secretkey: <BASE64>
```

##### AWS Controller-Level Workload Identity example
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's mention specifically https://fluxcd.io/flux/integrations/aws/#for-amazon-simple-storage-service as well like we did for GCP

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should add the link eventually. However, the current documentation at that URL (specifically the "At the object level" section) states that S3 integration with the Bucket API "does not support configuring authentication through OIDC Federation at the object level" and will be introduced in Flux v2.7.

Since this PR is implementing that functionality, I'll include the link in this PR and update the website documentation afterward to reflect the new capabilities. This way users will have the reference ready once the website is updated.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed via 794228f .


```yaml
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: Bucket
metadata:
name: aws-controller-level-workload-identity
namespace: default
spec:
interval: 5m0s
provider: aws
bucketName: podinfo
endpoint: s3.amazonaws.com
region: us-east-1
timeout: 30s
```

##### AWS Object-Level Workload Identity example

**Note:** To use Object-Level Workload Identity (`.spec.serviceAccountName` with
cloud providers), the controller feature gate `ObjectLevelWorkloadIdentity` must
be enabled.

```yaml
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: Bucket
metadata:
name: aws-object-level-workload-identity
namespace: default
spec:
interval: 5m0s
provider: aws
bucketName: podinfo
endpoint: s3.amazonaws.com
region: us-east-1
serviceAccountName: aws-workload-identity-sa
timeout: 30s
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: aws-workload-identity-sa
namespace: default
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/flux-bucket-role
```

#### Azure

When a Bucket's `.spec.provider` is set to `azure`, the source-controller will
Expand Down
64 changes: 46 additions & 18 deletions internal/bucket/minio/minio.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,17 @@ import (
"fmt"
"net/http"
"net/url"
"os"
"strings"

"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
"github.com/minio/minio-go/v7/pkg/s3utils"
corev1 "k8s.io/api/core/v1"

"github.com/fluxcd/pkg/auth"
awsauth "github.com/fluxcd/pkg/auth/aws"

sourcev1 "github.com/fluxcd/source-controller/api/v1"
)

Expand All @@ -46,6 +50,7 @@ type options struct {
tlsConfig *tls.Config
stsTLSConfig *tls.Config
proxyURL *url.URL
authOpts []auth.Option
}

// Option is a function that configures the Minio client.
Expand Down Expand Up @@ -86,8 +91,15 @@ func WithSTSTLSConfig(tlsConfig *tls.Config) Option {
}
}

// WithAuth sets the auth options for workload identity authentication.
func WithAuth(authOpts ...auth.Option) Option {
return func(o *options) {
o.authOpts = authOpts
}
}

// NewClient creates a new Minio storage client.
func NewClient(bucket *sourcev1.Bucket, opts ...Option) (*MinioClient, error) {
func NewClient(ctx context.Context, bucket *sourcev1.Bucket, opts ...Option) (*MinioClient, error) {
var o options
for _, opt := range opts {
opt(&o)
Expand All @@ -105,7 +117,11 @@ func NewClient(bucket *sourcev1.Bucket, opts ...Option) (*MinioClient, error) {
case o.secret != nil:
minioOpts.Creds = newCredsFromSecret(o.secret)
case bucketProvider == sourcev1.BucketProviderAmazon:
minioOpts.Creds = newAWSCreds(bucket, o.proxyURL)
creds, err := newAWSCreds(ctx, &o)
if err != nil {
return nil, err
}
minioOpts.Creds = creds
case bucketProvider == sourcev1.BucketProviderGeneric:
minioOpts.Creds = newGenericCreds(bucket, &o)
}
Expand Down Expand Up @@ -159,23 +175,35 @@ func newCredsFromSecret(secret *corev1.Secret) *credentials.Credentials {
}

// newAWSCreds creates a new Minio credentials object for `aws` bucket provider.
func newAWSCreds(bucket *sourcev1.Bucket, proxyURL *url.URL) *credentials.Credentials {
stsEndpoint := ""
if sts := bucket.Spec.STS; sts != nil {
stsEndpoint = sts.Endpoint
}

creds := credentials.NewIAM(stsEndpoint)
if proxyURL != nil {
transport := http.DefaultTransport.(*http.Transport).Clone()
transport.Proxy = http.ProxyURL(proxyURL)
client := &http.Client{Transport: transport}
creds = credentials.New(&credentials.IAM{
Client: client,
Endpoint: stsEndpoint,
})
//
// This function is only called when Secret authentication is not available.
//
// Uses AWS SDK's config.LoadDefaultConfig() which supports:
// - Workload Identity (IRSA/EKS Pod Identity)
// - EC2 instance profiles
// - Environment variables
// - Shared credentials files
// - All other AWS SDK authentication methods
func newAWSCreds(ctx context.Context, o *options) (*credentials.Credentials, error) {
var opts auth.Options
opts.Apply(o.authOpts...)

// Set AWS_REGION environment variable from bucket.Spec.Region for pkg/auth/aws
if opts.STSRegion != "" {
os.Setenv("AWS_REGION", opts.STSRegion)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
os.Setenv("AWS_REGION", opts.STSRegion)

I see that this is already being correctly configured with auth.WithSTSRegion() in bucket_controller.go, why is this here too?

(By the way, we should never do something like this, setting an environment variable in the middle of controller code, as there are reconciliations running in parallel and this would leak to other tenants.)

Copy link
Member Author

@cappyzawa cappyzawa Aug 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason this is here is because pkg/auth/aws internally uses config.LoadDefaultConfig(), which requires the AWS_REGION environment variable when no region is explicitly provided through other means. Even though we pass the region via auth.WithSTSRegion(), it seems the AWS SDK still looks for the environment variable in some cases.

I was focused on ensuring existing users with .spec.STS configurations could upgrade seamlessly, but I completely overlooked the multi-tenant implications 😮‍💨 You're right that this approach is problematic.

I think the proper fix would be to modify pkg/auth/aws to prioritize the region passed through auth.WithSTSRegion() over the environment variable. Would you be open to this approach? I could submit a PR to the pkg repository to fix this at the source rather than working around it here.

This would eliminate the need for os.Setenv() entirely and make the implementation cleaner and safer for multi-tenant environments.

I’d like to revisit how the current implementation works 🙇

Copy link
Member Author

@cappyzawa cappyzawa Aug 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upon further reflection, I realize the core issue: there was no Controller-Level authentication previously.

When no Secret was specified, authentication was handled at the Object level (per-Bucket) using each Bucket's specific IAM authentication (leveraging the individual Bucket's region and STS configuration).

Introducing Controller-Level Workload Identity at this point creates an overlap with the existing Object-level IAM authentication conditions. This means existing test cases that expected Object-level behavior are now failing due to Controller-Level WI attempting to authenticate without the required AWS_REGION environment variable.

This makes it difficult to introduce Controller-Level WI without regressions.

I've updated the implementation to defer Controller-Level Workload Identity and focus on:

1. Object-Level Workload Identity (when serviceAccountName is specified)
2. Traditional Object-level AWS credential chain (preserving existing behavior)

This approach maintains backward compatibility while delivering the most requested feature.

Question: What are your thoughts on Controller-Level Workload Identity support? Any ideas on how to introduce it safely without breaking existing users?

}

awsCredsProvider := awsauth.NewCredentialsProvider(ctx, o.authOpts...)
awsCreds, err := awsCredsProvider.Retrieve(ctx)
if err != nil {
return nil, fmt.Errorf("AWS authentication failed: %w", err)
}
return creds

return credentials.NewStaticV4(
awsCreds.AccessKeyID,
awsCreds.SecretAccessKey,
awsCreds.SessionToken,
), nil
}

// newGenericCreds creates a new Minio credentials object for the `generic` bucket provider.
Expand Down
Loading