Add SecretProvider support to RestateCloudEnvironment

[amccague]

Can we add SecretProvider support for the RestateCloudEnvironment secret?

Akin to RestateCluster's requestSigningPrivateKey

[jackkleeman]

yes, good idea

[jackkleeman]

hmm, lets think about how would this work in practice. the operator currently needs to read this secret in order to do registration, in addition to mounting it to the tunnel pods (where csi secret would work fine). there's no way we can dynamically give the operator access to the csi secret, afaik, it has to be deployed with a particular set of csi secrets.

if the tunnel pods always served an unauthenticated admin api for Cloud on :9070, then maybe the operator could register through the tunnel and it wouldn't need the secret. however, the remoteProxy feature currently defaults to false, and with good reason imo, as every pod in the cluster being able to hit :9070 seems something you should have to opt in to. maybe we could default to a network policy that limits access to only the operator, but not everyone has network policies enabled.

could it make sense to just use the csi secret store 'secret sync' functionality for these secrets? maybe you can tell me more about the security constraints and I can think about how to satisfy them

[amccague]

Our current methods of sourcing secrets (from Vault) are:

• vault injector, annotation based. Requires a container to be able to read a file containing the secret value.
• Vault secret driver. Requires a pod volume at least to have the secret created. We will often utilise a chart's additionalVolumes value to specify this then configure the secret provider to sync a secret and then reference the secret in the env, which is often the only path to provide the secret like in this case.

I've seen the recent sync operator to sync secrets without a volume, there will be friction to introduce this as the promise the secret will only exist if it's mounted/required is a desirable quality, it's also yet another operator to maintain.

[jackkleeman]

Option 2 sounds viable, with the current crd, I think? As the tunnel pods mount that secret, leading it to be created
I'm assuming you're referring to this? https://secrets-store-csi-driver.sigs.k8s.io/topics/sync-as-kubernetes-secret

[joebowbeer]

External Secrets Operator (ESO) will sync k8s Secrets from various sources.

@amccague You could use ESO to provide the needed secret?

Generally speaking, k8s Secrets are not secure and are best avoided, which is why SSCSI driver prefers to mount the sensitive content in the pod's filesystem, and only creates k8s Secrets as a convenient afterthought.

[amccague]

@joebowbeer indeed the preferred option is to load the secret by volume only but this is quite rarely supported.

Currently ESO is off the table, and env sourced secret from a CSI provisioned secret on demand is a compromise.

[joebowbeer]

indeed the preferred option is to load the secret by volume only but this is quite rarely supported

Ah, ok, I know what you mean. I wish this were more widely supported.

[jackkleeman]

so @amccague are any changes needed in the operator? it seems to me that we should support using sscsi secrets for the tunnel pods with a synced k8s secret for reading by the operator pods

[amccague]

so @amccague are any changes needed in the operator? it seems to me that we should support using sscsi secrets for the tunnel pods with a synced k8s secret for reading by the operator pods

Yes it's the tunnel pod support I'm looking for, which I believe is absent? Currently only taking a secret ref without the ability to specify a custom volume to trigger the SSCSI to fetch the secret.

Even more ideal if the secret could be read from the volume at a conventional path without it being in the env, and therefore no secret to sync but that's not a hard need.

[jackkleeman]

ok, tunnel pods using a volume source i can definitely do!

Even more ideal if the secret could be read from the volume at a conventional path without it being in the env

does this refer to the fact that the operator needs to read the k8s secret from the api in order to do registration? if so, i think this will be much harder to solve, so lets defer for now if its not a hard need

[amccague]

Even more ideal if the secret could be read from the volume at a conventional path without it being in the env

does this refer to the fact that the operator needs to read the k8s secret from the api in order to do registration? if so, i think this will be much harder to solve, so lets defer for now if its not a hard need

Yeah indeed that's a larger change.

[jackkleeman]

ive created #58, please give it a try and let me know if it fits your needs and if the API is ok!