Skip to content

Commit f9a1bdd

Browse files
dnitschdnitsch
authored
Feat/azure app config (#26)
* fix: rename keyvault inline with other az services * fix: add errors to central place in package * fix: clean up * fix: genvars config * fix: ParseMetadata use common generic * fix: az app conf unit tests +semver: feature +semver: feat * fix: update docs
1 parent 78f4439 commit f9a1bdd

24 files changed

+1165
-451
lines changed

‎README.md‎

Lines changed: 64 additions & 161 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,8 @@ Currently supported variable and secrets implementations:
3232
- [Azure TableStorage](https://azure.microsoft.com/en-gb/products/storage/tables/)
3333
- Implementation Indicator: `AZTABLESTORE`
3434
- see [Special consideration for AZTABLESTORE](#special-consideration-for-aztablestore) around how to structure the token in this case.
35+
- [Azure AppConfig](https://azure.microsoft.com/en-gb/products/app-configuration/)
36+
- Implementation Indicator: `AZAPPCONF`
3537
- [GCP Secrets](https://cloud.google.com/secret-manager)
3638
- Implementation Indicator: `GCPSECRETS`
3739
- [Hashicorp Vault](https://developer.hashicorp.com/vault/docs/secrets/kv)
@@ -65,109 +67,106 @@ ConfigManager comes packaged as a CLI for all major platforms, to see [download/
6567

6668
For more detailed usage you can run -h with each subcommand and additional info can be found [here](./docs/commands.md)
6769

68-
## __Config Tokens__
70+
## __Token Config__
6971

70-
The token is made up of 3 parts:
72+
The token is made up of the following parts:
73+
74+
_An example token would look like this_
75+
76+
#### `AWSSECRETS#/path/to/my/key|lookup.Inside.Object[meta=data]`
7177

7278
### Implementation indicator
7379

74-
e.g. `AWSSECRETS` the strategy identifier to choose at runtime
80+
The `AWSSECRETS` the strategy identifier to choose the correct provider at runtime. Multiple providers can be referenced in a single run via a CLI or with the API.
81+
82+
This is not overrideable and must be exactly as it is in the provided list of providers.
7583

7684
### __Token Separator__
7785

78-
e.g. `#` - used for separating the implementation indicator and the look up value.
86+
The `#` symbol from the [example token](#awssecretspathtomykeylookupinsideobjectmetadata) - used for separating the implementation indicator and the look up value.
7987

80-
> The default is currently `#` - it will change to `://` to allow for a more natural reading of the "token". you can achieve this behaviour now by either specifying the `-s` to the CLI or ConfigManager public methods, like below.
88+
> The default is currently `#` - it will change to `://` to allow for a more natural reading of the "token". you can achieve this behaviour now by either specifying the `-s` to the CLI or ConfigManager Go API.
8189
8290
```go
83-
rawStr := `somePAss: AWSPARAMSTR:///int-test/pocketbase/admin-pwd`
84-
cm := configmanager.ConfigManager{}
85-
// use custom token separator
86-
// inline with v2 coming changes
8791
cnf := generator.NewConfig().WithTokenSeparator("://")
88-
// replaced will be a string which needs unmarshalling
89-
replaced, err := cm.RetrieveWithInputReplaced(rawStr, *cnf)
9092
```
9193

92-
Alternatively you can use the helper methods for Yaml or Json tagged structs - see [examples](./examples/examples.go) for more details
94+
### __Provider Secret/Config Path__
9395

94-
- `/path/to/parameter` the actual path to the secret or parameter in the target system e.g. AWS SecretsManager or ParameterStore (it does assume a path like pattern might throw a runtime error if not found)
96+
The `/path/to/my/key` part from the [example token](#awssecretspathtomykeylookupinsideobjectmetadata) is the actual path to the item in the backing store.
9597

96-
If contents of the `AWSSECRETS#/appxyz/service1-password` are a string then `service1-password` will be the key and converted to UPPERCASE e.g. `SERVICE1_PASSWORD=som3V4lue`
98+
See the different special considerations per provider as it different providers will require different implementations.
9799

98100
### __Key Separator__
99101

100-
Specifying a key seperator on token items that can be parsed as a K/V map will result in only retrieving the specific key from the map.
102+
__THIS IS OPTIONAL__
103+
104+
The `|` symbol from the [example token](#awssecretspathtomykeylookupinsideobjectmetadata) is used to specify the key seperator.
105+
106+
If an item retrieved from a store is JSON parseable map it can be interrogated for further properties inside.
107+
108+
### __Look up key__
101109

102-
e.g. if contents of the `AWSSECRETS#/appxyz/service1-db-config` are parseable into the below object
110+
__THIS IS OPTIONAL__
111+
112+
The `lookup.Inside.Object` from the [example token](#awssecretspathtomykeylookupinsideobjectmetadata) is used to perform a lookup inside the retrieved item IF it is parseable into a `map[string]any` structure.
113+
114+
Given the below response from a backing store
103115

104116
```json
105117
{
106-
"host": "db.internal",
107-
"port": 3306,
108-
"pass": "sUp3$ecr3T!",
118+
"lookup": {
119+
"Inside": {
120+
"Object": {
121+
"host": "db.internal",
122+
"port": 3306,
123+
"pass": "sUp3$ecr3T!",
124+
}
125+
}
126+
}
109127
}
110128
```
111129

112-
Then you can access the single values like this `AWSSECRETS#/appxyz/service1-db-config|host` ==> `export SERVICE1_DB_CONFIG__HOST='db.internal'`
113-
114-
Alternatively if you are `configmanager`-ing a file via the fromstr command and the input is something like this:
115-
116-
(YAML)
130+
The value returned for the [example token](#awssecretspathtomykeylookupinsideobjectmetadata) would be:
117131

118-
```yaml
119-
app:
120-
name: xyz
121-
db:
122-
host: AWSSECRETS#/appxyz/service1-db-config|host
123-
port: AWSSECRETS#/appxyz/service1-db-config|port
124-
pass: AWSSECRETS#/appxyz/service1-db-config|pass
132+
```json
133+
{
134+
"host": "db.internal",
135+
"port": 3306,
136+
"pass": "sUp3$ecr3T!",
137+
}
125138
```
126139

127-
which would result in this
140+
See [examples of working with files](docs/examples.md#working-with-files) for more details.
128141

129-
```yaml
130-
app:
131-
name: xyz
132-
db:
133-
host: db.internal
134-
port: 3306
135-
pass: sUp3$ecr3T!
136-
```
142+
### Token Metadata Config
137143

138-
If your config parameter matches the config interface, you can also leave the entire token to point to the `db` key
144+
The `[meta=data]` from the [example token](#awssecretspathtomykeylookupinsideobjectmetadata) - is the optional metadata about the target in the backing provider
139145

140-
```yaml
141-
app:
142-
name: xyz
143-
db: AWSSECRETS#/appxyz/service1-db-config
144-
```
146+
IT must have this format `[key=value]` - IT IS OPTIONAL
145147

146-
result:
148+
The `key` and `value` would be provider specific. Meaning that different providers support different config, these values _CAN_ be safely omitted configmanager would just use the defaults where applicable or not specify the additional
147149

148-
```yaml
149-
app:
150-
name: xyz
151-
db: {
152-
"host": "db.internal",
153-
"port": 3306,
154-
"pass": "sUp3$ecr3T!",
155-
}
156-
```
150+
- Hashicorp Vault (VAULT)
151+
- `iam_role` - would be the value of an IAM role ARN to use with AWSClient Authentication.
152+
- `version` - is the version of the secret/configitem to get (should be in an integer format)
157153

158-
### Additional Token Config
154+
e.g. `VAULT://baz/bar/123|d88[role=arn:aws:iam::1111111:role/i-orchestration,version=1082313]`
159155

160-
Suffixed `[]` with `role:` or `version:` specified inside the brackets and comma separated
156+
- Azure AppConfig (AZAPPCONF)
157+
- `label` - the label to use whilst retrieving the item
158+
- `etag` - etag value
161159

162-
order is not important, but the `role:` keyword must be followed by the role string
160+
e.g. `AZAPPCONF://baz/bar/123|d88[label=dev,etag=aaaaa1082313]`
163161

164-
e.g. `VAULT://baz/bar/123|d88[role:arn:aws:iam::1111111:role/i-orchestration,version:1082313]`
162+
- GCP secrets, AWS SEcrets, AZ KeyVault (`GCPSECRETS` , `AWSSECRETS`, `AZKVSECRET`)
163+
they all support the `version` metadata property
165164

166-
Currently only supporting version and role but may be extended in the future.
165+
e.g. `GCPSECRETS://baz/bar/123|d88[version=verUUID0000-1123zss]`
167166

168-
- role is used with `VAULT` `aws_iam` auth type. Specifying it on a token level as opposed to globally will ensure that multiple roles can be used provided that the caller has the ability to assume them.
167+
## Special considerations
169168

170-
- version can be used within all implementations that support versioned config items e.g. `VAULT`, `GCPSECRETS` , `AWSSECRETS`, `AZKVSECRET`. If omitted it will default to the `LATEST`.
169+
This section outlines the special consideration in token construction on a per provider basis
171170

172171
### Special consideration for AZKVSECRET
173172

@@ -218,105 +217,9 @@ when using Vault in AWS - you can set the value of the `VAULT_TOKEN=aws_iam` thi
218217

219218
The Hashicorp Vault functions in the same exact way as the other implementations. It will retrieve the JSON object and can be looked up within it by using a key separator.
220219

221-
## Go API
222-
223-
latest api [here](https://pkg.go.dev/github.com/dnitsch/configmanager)
224-
225-
### Sample Use case
226-
227-
One of the sample use cases includes implementation in a K8s controller.
228-
229-
E.g. your Custom CRD stores some values in plain text that should really be secrets/nonpublic config parameters - something like this can be invoked from inside the controller code using the generator pkg API.
230-
231-
See [examples](./examples/examples.go) for more examples and tests for sample input/usage
220+
## [Go API](https://pkg.go.dev/github.com/dnitsch/configmanager)
232221

233-
```go
234-
package main
235-
236-
import (
237-
"context"
238-
"fmt"
239-
240-
"github.com/dnitsch/configmanager/pkg/generator"
241-
"github.com/dnitsch/configmanager"
242-
)
243-
244-
func main() {
245-
cm := &configmanager.ConfigManager{}
246-
cnf := generator.NewConfig()
247-
// JSON Marshal K8s CRD into
248-
exampleK8sCrdMarshalled := `apiVersion: crd.foo.custom/v1alpha1
249-
kind: CustomFooCrd
250-
metadata:
251-
name: foo
252-
namespace: bar
253-
spec:
254-
name: baz
255-
secret_val: AWSSECRETS#/customfoo/secret-val
256-
owner: [email protected]
257-
`
258-
pm, err := cm.RetrieveWithInputReplaced(exampleK8sCrdMarshalled, *cnf)
259-
260-
if err != nil {
261-
panic(err)
262-
}
263-
fmt.Println(pm)
264-
}
265-
```
266-
267-
Above example would ensure that you can safely store config/secret values on a CRD in plain text.
268-
269-
Or using go1.19+ [generics example](https://github.com/dnitsch/reststrategy/blob/d14ccec2b29bff646678ab9cf1775c0e93308569/controller/controller.go#L353).
270-
271-
> Beware logging out the CRD after tokens have been replaced.
272-
273-
Samlpe call to retrieve from inside an app/serverless function to only grab the relevant values from config.
274-
275-
```go
276-
package main
277-
278-
import (
279-
"context"
280-
"fmt"
281-
"log"
282-
"os"
283-
284-
"github.com/dnitsch/configmanager"
285-
"github.com/dnitsch/configmanager/pkg/generator"
286-
)
287-
288-
var (
289-
DB_CONNECTION_STRING string = "someuser:%v@tcp(%s:3306)/someschema"
290-
DB_PASSWORD_SECRET_PATH string = os.Getenv("DB_PASSWORD_TOKEN")
291-
DB_HOST_URL string = os.Getenv("DB_URL_TOKEN")
292-
)
293-
294-
func main() {
295-
connString, err := credentialString(context.TODO, DB_PASSWORD_SECRET_PATH, DB_HOST_URL)
296-
if err != nil {
297-
log.Fatal(err)
298-
}
299-
300-
}
301-
302-
func credentialString(ctx context.Context, pwdToken, hostToken string) (string, error) {
303-
304-
cnf := generator.NewConfig()
305-
306-
pm, err := configmanager.Retrieve([]string{pwdToken, hostToken}, *cnf)
307-
308-
if err != nil {
309-
return "", err
310-
}
311-
if pwd, ok := pm[pwdToken]; ok {
312-
if host, ok := pm[hostToken]; ok {
313-
return fmt.Sprintf(DB_CONNECTION_STRING, pwd, host), nil
314-
}
315-
}
316-
317-
return "", fmt.Errorf("unable to find value via token")
318-
}
319-
```
222+
## [Examples](docs/examples.md)
320223

321224
## Help
322225

‎configmanager.go‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,8 @@ import (
77
"sort"
88
"strings"
99

10-
yaml "gopkg.in/yaml.v3"
11-
1210
"github.com/dnitsch/configmanager/pkg/generator"
11+
yaml "gopkg.in/yaml.v3"
1312
)
1413

1514
const (

‎docs/adding-provider.md‎

Lines changed: 37 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# adding provider
1+
# Adding a provider
22

33
Add Token Prefix
44

@@ -30,10 +30,43 @@ ensure your implementation satisfy the `genVarsStrategy` interface
3030

3131
```go
3232
type genVarsStrategy interface {
33-
getTokenValue(rs *retrieveStrategy) (s string, e error)
34-
setToken(s string)
35-
setValue(s string)
33+
tokenVal(rs *retrieveStrategy) (s string, e error)
34+
setTokenVal(s string)
3635
}
3736
```
3837

3938
Even if the native type is K/V return a marshalled version of the JSON as the rest of the flow will decide how to present it back to the final consumer.
39+
40+
Custom properties inside the GetValue request, you could specify your own Config struct for the provider, e.g. HashiVault implementation
41+
42+
```go
43+
// VaultConfig holds the parseable metadata struct
44+
type VaultConfig struct {
45+
Version string `json:"version"`
46+
Role string `json:"iam_role"`
47+
}
48+
```
49+
50+
You could then use it on the backingStore object
51+
52+
```go
53+
type VaultStore struct {
54+
svc hashiVaultApi
55+
ctx context.Context
56+
config *VaultConfig
57+
token string
58+
}
59+
```
60+
61+
On initialize of the instance or in the setTokenVal method (see GCPSecrets or AWSSecrets/ParamStore examples).
62+
63+
```go
64+
storeConf := &VaultConfig{}
65+
initialToken := ParseMetadata(token, storeConf)
66+
imp := &VaultStore{
67+
ctx: ctx,
68+
config: storeConf,
69+
}
70+
```
71+
72+
Where the initialToken is the original Token without the metadata in brackets and the `storeConf` pointer will have been filled with any of the parsed metadata and used in the actual provider implementation, see any of the providers for a sample implementation.

0 commit comments

Comments
 (0)