Handling of custom AWS Secrets Manager data formats

[ogjkoch]

Describe the feature

I would like to have a way for the library to be able to retrieve credentials from AWS Secrets Manager entries that contain data which is not in the standard format.

Use Case

One of our organization's products has hundreds of production databases, and each of these has multiple sets of credentials we are managing (a read only as well as a read write connection as an example). Each database has one associated entry in AWS Secrets Manager which contains multiple sets of database credentials as well as other information specific to that data set.

It would be interesting to us to be able to use this plugin so that we were not handling credentials ourselves (currently we read the secrets and parse what we need from them at runtime, including for handling of password rotation under load), but creating an AWS Secrets Manager entry for each database user we are managing would be a significant lift and an additional cost that we would prefer to avoid. Authenticating as an IAM Role is unfortunately not an option for us due to limits and the need to maintain isolation between databases.

Providing the library with the ARN for an AWS Secrets Manager entry to read is very achievable, but our Secrets Manager data is not in the format that this library expects.

Proposed Solution

Presumably there are multiple ways this might be achieved:

• Setting of Properties which would allow consumers to provide a JSON path specification that identifies the location of username/password and any other desired properties within the secret value
• Passing a delegate function from the consumer to the library which would parse the secrets manager value for the desired properties

Other Information

Presumably we could fork the repository and implement this ourselves but we would prefer to avoid taking on ongoing maintenance of a fork. However, this has not been fully investigated and it is possible that modification of this repository alone would not be sufficient to achieve the desired goal.

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

The AWS Advanced .NET Data Provider Wrapper version used

We have only recently discovered the existence of this project are not yet consuming it

dotnet version used

9.0.308

Operating System and version

Microsoft Windows Server 2022

[kenrickyap]

Thanks for the detailed feature request! While we can’t provide a timeline for implementation, we agree this would be a valuable addition to the .NET wrapper. We’ve addressed a similar request in the Python wrapper in the past (see aws/aws-advanced-python-wrapper#843). That solution, however, assumes a relatively flat JSON structure and does not currently support deeply nested objects or arrays.

Would the Python approach be compatible with the structure of your Secrets Manager entries, or do you rely on nested? If you’re able to share a representative (sanitized) sample of the JSON structure you’re working with, that would be helpful in identifying a solution.

[ogjkoch]

Our expectation is that we would have a flat JSON structure in our AWS Secrets Manager entries and would be hoping to access top-level attribute names only, so the approach used for the Python implementation should be suitable for what we were hoping to achieve, yes.

We will be storing multiple credentials in a secret along with other data, so our values would be something along the lines of

{ "username_rw": "", "password_rw": "", "username_ro": "", "password_ro": "", other_value: "", ... }

As the current .NET implementation does not specify read-only vs. read-write connection handling we were already expecting to need to implement something for that ourselves, but we keep information for both connections in one secrets manager entry as they are both scoped to the same database instance for our use case.

[kenrickyap]

Got it, it seems unlikely that we would be able to detect which credential pair is needed per connection. For now we will look into providing parameters to specify which JSON properties should be used for the credential pair as we have done in Python.

As you mentioned, this would still require some additional logic on your end to identify which credential pair to use.

If this is sufficient we will aim to include this in our next release.

[ogjkoch]

That would be great, thank you for considering this!

[kenrickyap]

Just a heads up that we have merged in a PR for the suggested changes: #209.

This change should be reflected in the published NuGet packages in the next release.