[RFC-0010] Add multi-tenant workload identity support for AWS Bucket

[cappyzawa]

Part of: fluxcd/flux2#5022

[matheuscscp]

Looking good overall, thanks!

[matheuscscp]

Suggested change

os.Setenv("AWS_REGION", opts.STSRegion)

I see that this is already being correctly configured with auth.WithSTSRegion() in bucket_controller.go, why is this here too?

(By the way, we should never do something like this, setting an environment variable in the middle of controller code, as there are reconciliations running in parallel and this would leak to other tenants.)

[cappyzawa]

The reason this is here is because pkg/auth/aws internally uses config.LoadDefaultConfig(), which requires the AWS_REGION environment variable when no region is explicitly provided through other means. Even though we pass the region via auth.WithSTSRegion(), it seems the AWS SDK still looks for the environment variable in some cases.

I was focused on ensuring existing users with .spec.STS configurations could upgrade seamlessly, but I completely overlooked the multi-tenant implications 😮‍💨 You're right that this approach is problematic.

I think the proper fix would be to modify pkg/auth/aws to prioritize the region passed through auth.WithSTSRegion() over the environment variable. Would you be open to this approach? I could submit a PR to the pkg repository to fix this at the source rather than working around it here.

This would eliminate the need for os.Setenv() entirely and make the implementation cleaner and safer for multi-tenant environments.

I’d like to revisit how the current implementation works 🙇

[cappyzawa]

Upon further reflection, I realize the core issue: there was no Controller-Level authentication previously.

When no Secret was specified, authentication was handled at the Object level (per-Bucket) using each Bucket's specific IAM authentication (leveraging the individual Bucket's region and STS configuration).

Introducing Controller-Level Workload Identity at this point creates an overlap with the existing Object-level IAM authentication conditions. This means existing test cases that expected Object-level behavior are now failing due to Controller-Level WI attempting to authenticate without the required AWS_REGION environment variable.

This makes it difficult to introduce Controller-Level WI without regressions.

I've updated the implementation to defer Controller-Level Workload Identity and focus on:

1. Object-Level Workload Identity (when serviceAccountName is specified)
2. Traditional Object-level AWS credential chain (preserving existing behavior)

This approach maintains backward compatibility while delivering the most requested feature.

Question: What are your thoughts on Controller-Level Workload Identity support? Any ideas on how to introduce it safely without breaking existing users?

[matheuscscp]

Let's mention specifically https://fluxcd.io/flux/integrations/aws/#for-amazon-simple-storage-service as well like we did for GCP

[cappyzawa]

I think we should add the link eventually. However, the current documentation at that URL (specifically the "At the object level" section) states that S3 integration with the Bucket API "does not support configuring authentication through OIDC Federation at the object level" and will be introduced in Flux v2.7.

Since this PR is implementing that functionality, I'll include the link in this PR and update the website documentation afterward to reflect the new capabilities. This way users will have the reference ready once the website is updated.

[cappyzawa]

Fixed via 794228f .