Allow importing secrets with a full alternative key

[naumannt]

Type: Feature

Is your feature request related to a problem? Please describe.
I am currently setting up mostly automated secret rotations in a complex project infrastructure run on AWS. We use Infrastructure-as-Code to manage our AWS resources. To make the configurations as concise and understandable as possible, we want to have a single secret for each usage, e.g. a single database password secret used to connect to a database, used by different services. Due to how the services are set up, they use different keys for the passwords, which creates problems on secret import.

Describe the solution you'd like
Similar to the original prefix addition that was added in this PR, it would be great if it was possible to configure a full alternativ key used by the service property on importing a secret. This way, each consumer can decide their own key when importing secrets, moving that configuration to the consumer themselves.

Describe alternatives you've considered
Defining the secret import with custom properties file (using @Valid and the likes) sadly is not an option for implicitly set environment properties such as database connection passwords and username, and if it was, it would be an suboptimal amount of more or less boilerplate code to do something that spring boot already abstracted for developers.
We currently use an extra secret as an abstraction layer to fix this problem (each service module creates its own secret by aggregating all secrets required for the secret), but then each secret rotation process would need to update multiple secrets (the original source secrets and all aggregation secrets for each consumer), which explodes in complexity.

Additional context
The solution could be as simple as these changes (am not authorized to push a branch): https://pastebin.com/raw/1uachHjE

[MatejNedic]

Hey @naumannt ,

sadly this breaks core feature of spring.config.import which states that imported properties should be treated as they were copy pasted on the end of the file (taking precedence and overriding key if it is already present). One way how to solve your issue, would be to reorder spring.config.import keys, when you are loading and this way control the priority.

Now we could think of something but it won't be aligned with other spring.config.import implementations.

Could you please provide me a sample like secrets what you are trying to do so I can understand better. ->

spring.config.import: whatToLoad?

Secret dbPassword

passowrd: someValue

Secret dbPasswordStaging

passowrd: somethingElse

[naumannt]

Yes, here is an example that is creating the most obvious problem, but there is others:

• Service A and B are both Spring Boot Services exposing a REST API secured with Basic Auth
• Service A creates a User for Service B; the password for this User B is imported from Secretsmanager Secret /secret/servicea/basic-auth/serviceb/password
• Service B imports the password to connect to Service A from Secretsmanager Secret with the key /secret/serviceb/servicea/password
• Both services referencing the same secret is not possible, we need to save the secret with redundance

Granted, in this example, writing a custom BasicAuthProperties-Bean for the config would be possible.
There are other secrets imported into Spring without the usage of custom code at all (e.g. database credentials don't even show up in our properties files (key e.g. db.datasource.password)

Also; why does this

sadly this breaks core feature of spring.config.import which states that imported properties should be treated as they were copy pasted on the end of the file (taking precedence and overriding key if it is already present).

not also apply to the prefix function?

[naumannt]

@MatejNedic ping. It would be great if you have suggestions for alternative solutions, or if this could be implemented.

[MatejNedic]

Hey @naumannt , I am currently on vacation so I will reply slower sorry for that.

sadly this breaks core feature of spring.config.import which states that imported properties should be treated as they were copy pasted on the end of the file (taking precedence and overriding key if it is already present).

not also apply to the prefix function?

It does. What prefix does is before enriching properties just adding prefix. So if you in properties have

spring.config.import...
some_value=bla

#It is like we just copy pasted this below which overrides everything above
prefixed.secret=someth

Just to grasp if I got it correctly based on your example:

Service A creates a User for Service B; the password for this User B is imported from Secretsmanager Secret /secret/servicea/basic-auth/serviceb/password
Service B imports the password to connect to Service A from Secretsmanager Secret with the key /secret/serviceb/servicea/password

You are loading both of these secrets in Service B? If yes why not prefixing them with basic_auth. or connection_details.service_a. ?

If I understand it now, issue is that you want have a power of when you get a key password to decide to take one already loaded or new one which was just fetched?

You can also use spring.config.import order of loading for example.

spring.config.import= firstSecret; secondSecret

secondSecret will override first one. I tbh use this approach when I want to mark that certain secret has higher priority in my projects.
So if you had password in both firstSecret and secondSecret one from secondSecret will be in context.

If you don't want to use prefix nor order you can implement your own SecretsManagerPropertySource as a workaround.

I have in mind few ways how to solve this, maybe we can implement it for 4.0.0 release, will mark this for team discussion.

[naumannt]

Both Service A and B provide basic auth users which are directly inserted into the spring context via their keys. The terraform snippet looks like this, for example:
"basic-auth.users[0].username" = { plain = "admin" }
"basic-auth.users[0].password" = { secret = "esprit_basicauth_admin_password" }
"basic-auth.users[0].roles" = { plain = "ADMIN" }

We have no code mapping those secrets to the users; the basic-auth properties file is written in a way where these keys (basic-auth.users[0].username) end up in the user array directly.
But the admin user for example, is a different user for Service A and B, as each one provides such a user, and consumes one of the other service. Prefixing doesn't cut it, because the key needs to be exactly basic-auth.users[0].password. For the consume-case I can change the variable name, sure; that is for example already "servicea.basicauth.password", but that isn't the problem.

Other attributes, like the sql database password supplied to spring, also do not use a custom properties file, so it would be a lot of work to implement all the relevant PropertySource-Files; to the point where implementing a lambda aggregating and mapping the secrets in a redundant secret is less work.

I hope that explains the problem better. Thank you for answeing, and enjoy your vacation. Because of time pressure I will start implementing this with said abstraction layer via a lambda function, but I am still interesting in making this as sleek as possible.

[MatejNedic]

Okey I see what you are getting at now, tnx.

So for databse since spring boot does not support multiple datasources beans by default sadly we cannot do anything there.

Now regarding your issue something that comes quickly on top of my head, if I got it right.

Create a secret for a password which is shared.

Create per a service secret or parameter store param with encryption to save on costs for role and username.

But are you sure you want to use a single password shared for multiple usernames?

I will take some time to think about it and if I get some ideas will write!

[naumannt]

But are you sure you want to use a single password shared for multiple usernames?

optimally, yes. Because the secret rotation is configured on the secret in AWS. If the secret rotation has to change multiple secrets, as is the case when you have an additional redundancy/abstraction layer like a aggregating secret, it becomes quite complicated. Especially because multiple secrets in the aggregating secrets have to be rotated seperately.

[MatejNedic]

Hey, @naumannt was thinking little bit about the issue but apart from overriding the username in another secret/parameter or splitting username and passwords apart I am not seeing easy solution.

I will talk with Maciej and Tomaz about this feature and should we implement something for 4.0.0