Skip to content

Commit 5663804

Browse files
authored
fix: add helper for AWS (#17)
* fix: add helper for AWS * fix: pr ci * feat: add hashivault AWS IAM Auth minor CI tweak * fix: hashivault tests and implementation * fix: add more tests to hashivault * fix: add docs to document additional hashicorp features +semver: feat
1 parent 156f6b6 commit 5663804

File tree

20 files changed

+776
-149
lines changed

20 files changed

+776
-149
lines changed

.github/workflows/build.yml

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ on:
55
branches: [ master, main ]
66

77
jobs:
8-
set-deps-dotnet-poop-here:
8+
set-version:
99
runs-on: ubuntu-latest
1010
container:
1111
image: mcr.microsoft.com/dotnet/sdk:6.0
@@ -31,10 +31,10 @@ jobs:
3131
runs-on: ubuntu-latest
3232
container:
3333
image: golang:1.19-bullseye
34-
needs: set-deps-dotnet-poop-here
34+
needs: set-version
3535
env:
36-
SEMVER: ${{ needs.set-deps-dotnet-poop-here.outputs.semVer }}
37-
GIT_TAG: ${{ needs.set-deps-dotnet-poop-here.outputs.semVer }}
36+
SEMVER: ${{ needs.set-version.outputs.semVer }}
37+
GIT_TAG: ${{ needs.set-version.outputs.semVer }}
3838
GOVCS: false
3939
steps:
4040
- uses: actions/checkout@v3
@@ -65,6 +65,6 @@ jobs:
6565
# Additional arguments for the sonarcloud scanner
6666
args:
6767
# mandatory
68-
-Dsonar.projectVersion=${{ needs.set-deps-dotnet-poop-here.outputs.semVer }}
68+
-Dsonar.projectVersion=${{ needs.set-version.outputs.semVer }}
6969
-Dsonar.go.coverage.reportPaths=/github/workspace/.coverage/out
7070
-Dsonar.go.tests.reportPaths=/github/workspace/.coverage/report-junit.xml

.github/workflows/pr.yml

Lines changed: 9 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ on:
55
branches: [ master, main ]
66

77
jobs:
8-
set-deps-dotnet-poop-here:
8+
set-version:
99
runs-on: ubuntu-latest
1010
container:
1111
image: mcr.microsoft.com/dotnet/sdk:6.0
@@ -22,25 +22,22 @@ jobs:
2222
- name: Set SemVer Version
2323
uses: gittools/actions/gitversion/[email protected]
2424
id: gitversion
25-
26-
- name: echo VERSIONS
27-
run: |
28-
echo "REVISION -> $GITHUB_SHA"
29-
echo "VERSION -> $GITVERSION_SEMVER"
3025
pr:
3126
runs-on: ubuntu-latest
3227
container:
3328
image: golang:1.19-bullseye
34-
needs: set-deps-dotnet-poop-here
29+
needs: set-version
3530
env:
3631
REVISION: $GITHUB_SHA
37-
SEMVER: ${{ needs.set-deps-dotnet-poop-here.outputs.semVer }}
32+
SEMVER: ${{ needs.set-version.outputs.semVer }}
3833
steps:
3934
- uses: actions/checkout@v3
4035
- name: install deps
4136
run: |
42-
apt update && apt install jq -y
43-
make REVISION=$GITHUB_SHA install
37+
apt-get update && apt-get install -y jq git
38+
git config --global --add safe.directory "$GITHUB_WORKSPACE"
39+
git config user.email ${{ github.actor }}[email protected]
40+
git config user.name ${{ github.actor }}
4441
- name: make test
4542
run: |
4643
make REVISION=$GITHUB_SHA test
@@ -57,6 +54,6 @@ jobs:
5754
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} # Generate a token on Sonarcloud.io, add it to the secrets of this repo with the name SONAR_TOKEN (Settings > Secrets > Actions > add new repository secret)
5855
with:
5956
args:
60-
-Dsonar.projectVersion=${{ needs.set-deps-dotnet-poop-here.outputs.semVer }}
57+
-Dsonar.projectVersion=${{ needs.set-version.outputs.semVer }}
6158
-Dsonar.go.coverage.reportPaths=/github/workspace/.coverage/out
62-
-Dsonar.go.tests.reportPaths=/github/workspace/.coverage/report-junit.xml
59+
-Dsonar.go.tests.reportPaths=/github/workspace/.coverage/report-junit.xml

.github/workflows/release.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ on:
77
- completed
88

99
jobs:
10-
set-deps-dotnet-poop-here:
10+
set-version:
1111
runs-on: ubuntu-latest
1212
if: ${{ github.event.workflow_run.conclusion == 'success' }}
1313
container:
@@ -36,9 +36,9 @@ jobs:
3636
image: golang:1.19-bullseye
3737
env:
3838
FOO: Bar
39-
needs: set-deps-dotnet-poop-here
39+
needs: set-version
4040
env:
41-
SEMVER: ${{ needs.set-deps-dotnet-poop-here.outputs.semVer }}
41+
SEMVER: ${{ needs.set-version.outputs.semVer }}
4242
steps:
4343
- uses: actions/checkout@v3
4444
with:

README.md

Lines changed: 45 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -10,17 +10,32 @@
1010
Package used for retrieving application settings from various sources.
1111

1212
Currently supported variable and secrets implementations:
13+
<!--
14+
"AWSSECRETS"
15+
// AWS Parameter Store prefix
16+
ParamStorePrefix ImplementationPrefix = "AWSPARAMSTR"
17+
// Azure Key Vault Secrets prefix
18+
AzKeyVaultSecretsPrefix ImplementationPrefix = "AZKVSECRET"
19+
// Hashicorp Vault prefix
20+
HashicorpVaultPrefix ImplementationPrefix = "VAULT"
21+
// GcpSecrets
22+
GcpSecretsPrefix ImplementationPrefix = "GCPSECRETS" -->
1323

1424
- [AWS SecretsManager](https://aws.amazon.com/secrets-manager/)
25+
- Implementation Indicator: `AWSSECRETS`
1526
- [AWS ParameterStore](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html)
27+
- Implementation Indicator: `AWSPARAMSTR`
1628
- [AzureKeyvault Secrets](https://azure.microsoft.com/en-gb/products/key-vault/)
29+
- Implementation Indicator: `AZKVSECRET`
1730
- see [Special consideration for AZKVSECRET](#special-consideration-for-azkvsecret) around how to structure the token in this case.
1831
- [GCP Secrets](https://cloud.google.com/secret-manager)
32+
- Implementation Indicator: `GCPSECRETS`
1933
- [Hashicorp Vault](https://developer.hashicorp.com/vault/docs/secrets/kv)
20-
- using the V2 endpoint
34+
- Implementation Indicator: `VAULT`
35+
- using the KvV2 engine endpoint
2136
- see [special consideration hashivault](#special-consideration-for-hashicorpvault)
2237

23-
The main driver is to use component level configuration objects, if stored in a `"namespaced"` manner e.g. in AWS ParamStore as `/nonprod/component-service-a/configVar`, however this is not a requirement and the param name can be whatever. Though whilst using some sort of a organised manner it will be more straight forward to allow other services to consume certain secrets/params based on resource/access policies.
38+
The main driver is to use component level configuration objects, if stored in a `"namespaced"` manner e.g. in AWS ParamStore as `/nonprod/component-service-a/configVar`, however this is not a requirement and the param name can be whatever. Though whilst using some sort of a organised manner it will be more straight forward to allow other services to consume certain secrets/params based on resource/access policies.
2439

2540
> Beware size limitation with certain config/vault implementations. In which case it's best to split certain items up e.g. TLS certs `/nonprod/component-service-a/pub-cert`, `/nonprod/component-service-a/private-cert`, `/nonprod/component-service-a/chain1-cert`, etc...
2641
@@ -46,13 +61,17 @@ ConfigManager comes packaged as a CLI for all major platforms, to see [download/
4661

4762
For more detailed usage you can run -h with each subcommand and additional info can be found [here](./docs/commands.md)
4863

49-
## Config Tokens
64+
## __Config Tokens__
5065

5166
The token is made up of 3 parts:
5267

53-
- `AWSSECRETS` the strategy identifier to choose at runtime
68+
### Implementation indicator
5469

55-
- `#` separator - used for separating the implementation indicator and the look up value.
70+
e.g. `AWSSECRETS` the strategy identifier to choose at runtime
71+
72+
### __Token Separator__
73+
74+
e.g. `#` - used for separating the implementation indicator and the look up value.
5675

5776
> The default is currently `#` - it will change to `://` to allow for a more natural reading of the "token". you can achieve this behaviour now by either specifying the `-s` to the CLI or ConfigManager public methods, like below.
5877
@@ -72,11 +91,12 @@ Alternatively you can use the helper methods for Yaml or Json tagged structs - s
7291

7392
If contents of the `AWSSECRETS#/appxyz/service1-password` are a string then `service1-password` will be the key and converted to UPPERCASE e.g. `SERVICE1_PASSWORD=som3V4lue`
7493

75-
### KeySeparator
94+
### __Key Separator__
7695

7796
Specifying a key seperator on token items that can be parsed as a K/V map will result in only retrieving the specific key from the map.
7897

7998
e.g. if contents of the `AWSSECRETS#/appxyz/service1-db-config` are parseable into the below object
99+
80100
```json
81101
{
82102
"host": "db.internal",
@@ -113,7 +133,6 @@ db:
113133
114134
If your config parameter matches the config interface, you can also leave the entire token to point to the `db` key
115135

116-
117136
```yaml
118137
app:
119138
name: xyz
@@ -132,6 +151,20 @@ db: {
132151
}
133152
```
134153

154+
### Additional Token Config
155+
156+
Suffixed `[]` with `role:` or `version:` specified inside the brackets and comma separated
157+
158+
order is not important, but the `role:` keyword must be followed by the role string
159+
160+
e.g. `VAULT://baz/bar/123|d88[role:arn:aws:iam::1111111:role/i-orchestration,version:1082313]`
161+
162+
Currently only supporting version and role but may be extended in the future.
163+
164+
- role is used with `VAULT` `aws_iam` auth type. Specifying it on a token level as opposed to globally will ensure that multiple roles can be used provided that the caller has the ability to assume them.
165+
166+
- version can be used within all implementations that support versioned config items e.g. `VAULT`, `GCPSECRETS` , `AWSSECRETS`, `AZKVSECRET`. If omitted it will default to the `LATEST`.
167+
135168
### Special consideration for AZKVSECRET
136169

137170
For Azure KeyVault the first part of the token needs to be the name of the vault.
@@ -147,12 +180,15 @@ For Azure KeyVault the first part of the token needs to be the name of the vault
147180
### Special consideration for HashicorpVault
148181

149182
For HashicorpVault the first part of the token needs to be the name of the mountpath. In Dev Vaults this is `"secret"`,
150-
e.g.:
183+
e.g.: `VAULT://secret___demo/configmanager|test`
151184

152-
`VAULT://secret___demo/configmanager|test`
185+
or if the secrets are at another location: `VAULT://another/mount/path__config/app1/db`
153186

154187
The hardcoded separator cannot be modified and you must separate your `mountPath` with `___` (3x `_`) followed by the key to the secret.
155188

189+
#### AWS IAM auth to vault
190+
191+
when using Vault in AWS - you can set the value of the `VAULT_TOKEN=aws_iam` this will trigger the AWS Auth login as opposed to using the local token.
156192

157193
The Hashicorp Vault functions in the same exact way as the other implementations. It will retrieve the JSON object and can be looked up within it by using a key separator.
158194

go.mod

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ require (
2424
github.com/AzureAD/microsoft-authentication-library-for-go v0.7.0 // indirect
2525
github.com/armon/go-metrics v0.3.9 // indirect
2626
github.com/armon/go-radix v1.0.0 // indirect
27+
github.com/aws/aws-sdk-go v1.30.27 // indirect
2728
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.0 // indirect
2829
github.com/cenkalti/backoff/v3 v3.0.0 // indirect
2930
github.com/fatih/color v1.7.0 // indirect
@@ -42,6 +43,7 @@ require (
4243
github.com/hashicorp/go-plugin v1.4.5 // indirect
4344
github.com/hashicorp/go-retryablehttp v0.6.6 // indirect
4445
github.com/hashicorp/go-rootcerts v1.0.2 // indirect
46+
github.com/hashicorp/go-secure-stdlib/awsutil v0.1.6 // indirect
4547
github.com/hashicorp/go-secure-stdlib/mlock v0.1.1 // indirect
4648
github.com/hashicorp/go-secure-stdlib/parseutil v0.1.6 // indirect
4749
github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 // indirect
@@ -61,6 +63,7 @@ require (
6163
github.com/oklog/run v1.0.0 // indirect
6264
github.com/pierrec/lz4 v2.5.2+incompatible // indirect
6365
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
66+
github.com/pkg/errors v0.9.1 // indirect
6467
github.com/ryanuber/go-glob v1.0.0 // indirect
6568
go.opencensus.io v0.24.0 // indirect
6669
go.uber.org/atomic v1.9.0 // indirect
@@ -93,6 +96,7 @@ require (
9396
github.com/axw/gocov v1.1.0
9497
github.com/googleapis/gax-go/v2 v2.7.0
9598
github.com/hashicorp/vault/api v1.8.2
99+
github.com/hashicorp/vault/api/auth/aws v0.3.0
96100
github.com/inconshreveable/mousetrap v1.1.0 // indirect
97101
github.com/jmespath/go-jmespath v0.4.0 // indirect
98102
github.com/jstemmer/go-junit-report v1.0.0

0 commit comments

Comments
 (0)