[DO NOT MERGE, WIP]Add support to fetch cacertfiles from Amazon S3 #14572

BeichenZhang-BCZ · 2025-09-18T23:38:20Z

In collaboration with Sunfinite - thanks for kicking off first commit

Proposed Changes

Please describe the big picture of your changes here to communicate to the RabbitMQ team why we should accept this pull request.
If it fixes a bug or resolves a feature request, be sure to link to that issue.

When rabbitmq user configures or operates rabbitmq, user may want to configure rabbitmq to access information such as private CA certs/credentials etc, in AWS service rather than passing those information as local file or plain-text configuration to improve security and usability. For example, if user intends to configure LDAP for authentication and authorization, user as of today needs to define a local certs file and inform rabbitmq with config or environment variable .

With this AWS Resource Fetcher feature, user can pass in Amazon Resource Name(ARN) of the certs through new configuration schema eg: aws.arns.auth_ldap.ssl_options.cacertfile. The fetcher will fetch the relevant cert files from AWS S3 from user's account and assign it to the appropriate LDAP plugin application environment.

For corner case handling:

Change is backward compatible if user doesn't configure any new ARN based attributes.
If the user configurates both arn attribute and normal attribute for the same content, the value fetched according to arn attribute will prevail.
If the fetcher failed to fetch from user resource, an unique error message will be thrown and rabbitmq start-up will fail.

This PR is working in progress to solicit early feedbacks. We are working on to add more functionalities such as allowing customer to configure Secret Manager arn for rabbitmq to retrieve credentials.

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

Bug fix (non-breaking change which fixes issue #NNNN)
[ x ] New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause an observable behavior change in existing systems)
Documentation improvements (corrections, new content, etc)
Cosmetic change (whitespace, formatting, etc)
Build system and/or CI

Checklist

Put an x in the boxes that apply.
You can also fill these out after creating the PR.
This is simply a reminder of what we are going to look for before merging your code.

[ x ] Mandatory: I (or my employer/client) have have signed the CA (see https://github.com/rabbitmq/cla)
[ x ] I have read the CONTRIBUTING.md document
I have added tests that prove my fix is effective or that my feature works
All tests pass locally with my changes
If relevant, I have added necessary documentation to https://github.com/rabbitmq/rabbitmq-website
If relevant, I have added this change to the first version(s) in release-notes that I expect to introduce it

Further Comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution
you did and what alternatives you considered, etc.

lukebakken

Let's chat about cacerts.

lukebakken · 2025-09-19T19:02:56Z

deps/rabbitmq_aws/src/rabbitmq_aws_resource_fetcher.erl

+            lists:foreach(fun({_Key, Arn, Handler}) ->
+                case resolve_arn(Arn) of
+                    {ok, Content} ->
+                        FilePath = write_to_file(Arn, Content),


There is no need to write these files to the filesystem if we use the cacerts configuration option. Please see this discussion for more information:

#14426

lukebakken · 2025-09-19T19:03:41Z

deps/rabbitmq_aws/src/rabbitmq_aws_resource_fetcher.erl

+
+%% Export all for unit tests
+-ifdef(TEST).
+-compile(export_all).


It would be better to be specific about which functions are exported for tests.

deps/rabbitmq_aws/test/rabbitmq_aws_resource_fetcher_tests.erl

lukebakken · 2025-09-19T19:04:55Z

deps/rabbitmq_aws/src/rabbitmq_aws_resource_fetcher.erl

+
+-spec resolve_arn(string()) -> {ok, binary()} | {error, term()}.
+resolve_arn(Arn) ->
+    ?LOG_INFO("Attempting to resolve ARN: ~p", [Arn]),


Suggested change

?LOG_INFO("Attempting to resolve ARN: ~p", [Arn]),

?LOG_DEBUG("aws arn: resolving: ~p", [Arn]),

lukebakken · 2025-09-19T19:05:13Z

deps/rabbitmq_aws/src/rabbitmq_aws_resource_fetcher.erl

+                        end,
+                        ?LOG_INFO("Fetched ARN ~p and stored to ~p", [Arn, FilePath]);
+                    {error, Reason} ->
+                        ?LOG_ERROR("Failed to resolve ARN ~p: ~p", [Arn, Reason])


Suggested change

?LOG_ERROR("Failed to resolve ARN ~p: ~p", [Arn, Reason])

?LOG_ERROR("aws arn: failed to resolve ~p: ~p", [Arn, Reason])

lukebakken · 2025-09-19T19:05:32Z

deps/rabbitmq_aws/src/rabbitmq_aws_resource_fetcher.erl

+                            ldap_ssl_options_cacertfile ->
+                                update_env(rabbitmq_auth_backend_ldap, ssl_options, cacertfile, FilePath)
+                        end,
+                        ?LOG_INFO("Fetched ARN ~p and stored to ~p", [Arn, FilePath]);


Suggested change

?LOG_INFO("Fetched ARN ~p and stored to ~p", [Arn, FilePath]);

?LOG_INFO("aws arn: fetched ~p and stored to ~p", [Arn, FilePath]);

lukebakken · 2025-09-19T19:05:54Z

deps/rabbitmq_aws/src/rabbitmq_aws_resource_fetcher.erl

+        {ok, #{service := "s3"}} ->
+            fetch_s3_object(Arn);
+        {ok, #{service := Service}} ->
+            ?LOG_ERROR("Unsupported service: ~p", [Service]),


Suggested change

?LOG_ERROR("Unsupported service: ~p", [Service]),

?LOG_ERROR("aws arn: unsupported service: ~p", [Service]),

lukebakken · 2025-09-19T19:06:10Z

deps/rabbitmq_aws/src/rabbitmq_aws_resource_fetcher.erl

+            ?LOG_ERROR("Unsupported service: ~p", [Service]),
+            {error, {unsupported_service, Service}};
+        {error, _} = Error ->
+            ?LOG_ERROR("Failed to parse ARN ~p: ~p", [Arn, Error]),


Suggested change

?LOG_ERROR("Failed to parse ARN ~p: ~p", [Arn, Error]),

?LOG_ERROR("aws arn: failed to parse ~p: ~p", [Arn, Error]),

deps/rabbitmq_aws/src/rabbitmq_aws_resource_fetcher.erl

lukebakken · 2025-09-19T19:08:56Z

deps/rabbitmq_aws/src/rabbitmq_aws_resource_fetcher.erl

+    case binary:split(ArnBin, <<":">>, [global]) of
+        [<<"arn">>, Partition, Service, Region, Account, Resource] ->
+            {ok, #{
+                partition => binary_to_list(Partition),


Why are we doing conversions to lists here?

+1 I also noted the conversion here. The input from {mapping, "aws.arns.auth_ldap.ssl_options.cacertfile", "rabbit.aws_arns", [{datatype, string}]}. is a binary.Can we just use binary:split ? @sunfinite

If the reason of using list is because you need to use string:tokens against the output, considering using string:lexemes which takes in either list or binary

lukebakken · 2025-09-19T19:25:25Z

An alternative to using aws arn: to differentiate log messages would be to have an AWS-plugin specific logging domain - https://www.erlang.org/doc/apps/kernel/logger.html#t:metadata/0 cc @SimonUnge @the-mikedavis

michaelklishin · 2025-09-24T17:52:34Z

v4.2.x and v4.1.x now have separate logging domains for all, or at least most, major plugins.

BeichenZhang-BCZ · 2025-09-24T23:41:07Z

deps/rabbitmq_aws/src/rabbitmq_aws_resource_fetcher.erl

+        {ok, ExistingConfig} -> ExistingConfig;
+        undefined -> []
+    end,
+    NewConfig = lists:keystore(Key, 1, Config, {Key, Value}),


This code applies if the configuration value is of type list. However, for many config, they are just a tuple. For example, when you run

rabbitmqctl eval "application:get_all_env(rabbitmq_auth_backend_ldap)."

assuming you have configured the ldap, you will get:

[... {ssl_options,[{verify,verify_none}]}, {user_dn_pattern,"${username}"}, .. ]

some fields are array while others are just tuple.

In next rev, I will create modify the code to call different functions based on the nature of the configuration

* Add comprehensive tests for the `rabbitmq_aws` cuttlefish schema. * Move `aws_prefer_imdsv2` setting to the `rabbitmq_aws` application.

… used with IMDSv2 previously.

* Update ARN handling to not require file system access for CA certs

…it_aws when it should be rabbitmq_aws

…ource_fetcher

…. Solve bug where STS is called more than once. 3. Solve bugs where type mismatches happened during refactoring. 4. Solve bugs where AWS_LOG_ERROR receives more than 2 args

lukebakken self-requested a review September 19, 2025 18:57

lukebakken self-assigned this Sep 19, 2025

lukebakken force-pushed the aws-resource-fetcher branch from 031d6b0 to e711234 Compare September 19, 2025 18:58

lukebakken requested changes Sep 19, 2025

View reviewed changes

BeichenZhang-BCZ changed the title ~~Add support to fetch cacertfiles from Amazon S3~~ [DO NOT MERGE, WIP]Add support to fetch cacertfiles from Amazon S3 Sep 19, 2025

lukebakken marked this pull request as draft September 23, 2025 15:49

lukebakken force-pushed the aws-resource-fetcher branch from e711234 to 07d22a9 Compare September 23, 2025 19:49

lukebakken mentioned this pull request Sep 23, 2025

AWS plugin: enable Cuttlefish schema tests #14595

Merged

mergify bot mentioned this pull request Sep 24, 2025

AWS plugin: enable Cuttlefish schema tests (backport #14595) #14596

Merged

lukebakken force-pushed the aws-resource-fetcher branch from 07d22a9 to 789669f Compare September 24, 2025 14:56

lukebakken force-pushed the aws-resource-fetcher branch from 7df552b to baf7519 Compare September 24, 2025 22:20

BeichenZhang-BCZ commented Sep 24, 2025

View reviewed changes

lukebakken force-pushed the aws-resource-fetcher branch 2 times, most recently from 2777062 to 903ef14 Compare September 26, 2025 13:41

mergify bot added the make label Sep 26, 2025

BeichenZhang-BCZ force-pushed the aws-resource-fetcher branch from 4763a64 to 5a9e4d4 Compare September 29, 2025 17:53

lukebakken force-pushed the aws-resource-fetcher branch from ab397b0 to 03f0591 Compare September 29, 2025 21:42

sunfinite and others added 11 commits September 30, 2025 09:26

Add support to fetch cacertfiles from Amazon S3

aab1cb3

* Add comprehensive tests for the `rabbitmq_aws` cuttlefish schema. * Move `aws_prefer_imdsv2` setting to the `rabbitmq_aws` application.

Use AWS session token when present in env or config file. It was only…

18ee003

… used with IMDSv2 previously.

First pass of changes for assume role support in resource fetcher

5e46a8f

Fix use of application:get_env/2 to get from the correct application

f11ea54

Minor Makefile re-org

5dcb3cd

* Update logging to use aws arn: prefix

68ea67e

* Update ARN handling to not require file system access for CA certs

Refactor to use function head matching instead of nested case

785138c

Use string:split in a couple places where it is more appropriate.

3040a03

Add Support to Fetch LDAP Bind Passwords from AWS Secret Manager

b03fc3e

Make Secret Manager Fetcher to Assume Role first before Calling.

19f5494

re:split/3 can use charlists, and return charlists

bb07dee

lukebakken and others added 11 commits September 30, 2025 09:26

Fix bug in what is passed to fetch_s3_object/1

e52bc5f

Change all LOG_INFO to LOG DEBUG, other refactorings, fix use of rabb…

d9ba63b

…it_aws when it should be rabbitmq_aws

Reduce code copying via function passing.

e741920

Use custom log macros to ensure log lines start with rabbitmq_aws_res…

e5b4d88

…ource_fetcher

Tighten up logging and error detection

17b659a

Simplify code for setting LDAP password

54d4100

Fix error propagation

f141dc4

Simplify LDAP config further, error propagation again

55d7e8d

Use existing cacertfile if exists and can be decoded

2cba1d5

Only call assume_role/1 once, before ARNs are processed

b1ea6e2

1.Configure and Use Confused Deputy Header for STS assume role call 2…

4e96997

…. Solve bug where STS is called more than once. 3. Solve bugs where type mismatches happened during refactoring. 4. Solve bugs where AWS_LOG_ERROR receives more than 2 args

lukebakken force-pushed the aws-resource-fetcher branch from 03f0591 to 4e96997 Compare September 30, 2025 16:26

lukebakken closed this Oct 2, 2025

lukebakken deleted the aws-resource-fetcher branch October 24, 2025 15:02

	?LOG_INFO("Attempting to resolve ARN: ~p", [Arn]),
	?LOG_DEBUG("aws arn: resolving: ~p", [Arn]),

	?LOG_ERROR("Failed to resolve ARN ~p: ~p", [Arn, Reason])
	?LOG_ERROR("aws arn: failed to resolve ~p: ~p", [Arn, Reason])

	?LOG_INFO("Fetched ARN ~p and stored to ~p", [Arn, FilePath]);
	?LOG_INFO("aws arn: fetched ~p and stored to ~p", [Arn, FilePath]);

	?LOG_ERROR("Unsupported service: ~p", [Service]),
	?LOG_ERROR("aws arn: unsupported service: ~p", [Service]),

	?LOG_ERROR("Failed to parse ARN ~p: ~p", [Arn, Error]),
	?LOG_ERROR("aws arn: failed to parse ~p: ~p", [Arn, Error]),

[DO NOT MERGE, WIP]Add support to fetch cacertfiles from Amazon S3 #14572

[DO NOT MERGE, WIP]Add support to fetch cacertfiles from Amazon S3 #14572

Uh oh!

Conversation

BeichenZhang-BCZ commented Sep 18, 2025

Proposed Changes

Types of Changes

Checklist

Further Comments

Uh oh!

lukebakken left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lukebakken commented Sep 19, 2025

Uh oh!

michaelklishin commented Sep 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants