K8SPSMDB-1418: Allow setting CA for backup storages#2164
Conversation
| cat /etc/s3/certs-in/*.crt > /etc/s3/certs/ca-bundle.crt | ||
| chmod 0644 /etc/s3/certs/ca-bundle.crt |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| cat /etc/s3/certs-in/*.crt > /etc/s3/certs/ca-bundle.crt | |
| chmod 0644 /etc/s3/certs/ca-bundle.crt | |
| cat /etc/s3/certs-in/*.crt >/etc/s3/certs/ca-bundle.crt | |
| chmod 0644 /etc/s3/certs/ca-bundle.crt |
| EOF | ||
|
|
||
| kubectl_bin wait --for=condition=Ready certificate/minio-cert \ | ||
| -n ${namespace} --timeout=120s |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| -n ${namespace} --timeout=120s | |
| -n ${namespace} --timeout=120s |
| -n ${namespace} --timeout=120s | ||
|
|
||
| kubectl_bin get secret minio-tls-temp -n ${namespace} \ | ||
| -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/minio-cert.pem |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/minio-cert.pem | |
| -o jsonpath='{.data.tls\.crt}' | base64 -d >/tmp/minio-cert.pem |
| -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/minio-cert.pem | ||
|
|
||
| kubectl_bin get secret minio-tls-temp -n ${namespace} \ | ||
| -o jsonpath='{.data.tls\.key}' | base64 -d > /tmp/minio-key.pem |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| -o jsonpath='{.data.tls\.key}' | base64 -d > /tmp/minio-key.pem | |
| -o jsonpath='{.data.tls\.key}' | base64 -d >/tmp/minio-key.pem |
| --from-file=public.crt=/tmp/minio-cert.pem \ | ||
| --from-file=private.key=/tmp/minio-key.pem |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| --from-file=public.crt=/tmp/minio-cert.pem \ | |
| --from-file=private.key=/tmp/minio-key.pem | |
| --from-file=public.crt=/tmp/minio-cert.pem \ | |
| --from-file=private.key=/tmp/minio-key.pem |
| --from-file=private.key=/tmp/minio-key.pem | ||
|
|
||
| kubectl_bin create secret generic minio-ca-bundle -n ${namespace} \ | ||
| --from-file=ca.crt=/tmp/minio-cert.pem |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| --from-file=ca.crt=/tmp/minio-cert.pem | |
| --from-file=ca.crt=/tmp/minio-cert.pem |
…ver-mongodb-operator into K8SPSMDB-1418_add_ca
…ver-mongodb-operator into K8SPSMDB-1418_add_ca
pkg/apis/psmdb/v1/psmdb_types.go
Outdated
| MaxUploadParts int32 `json:"maxUploadParts,omitempty"` | ||
| StorageClass string `json:"storageClass,omitempty"` | ||
| InsecureSkipTLSVerify bool `json:"insecureSkipTLSVerify,omitempty"` | ||
| CABundle *corev1.SecretKeySelector `json:"caBundle,omitempty"` |
There was a problem hiding this comment.
Please update PR description if we plan to add it to S3 as well
There was a problem hiding this comment.
no, we want to keep it only for minio type.
There was a problem hiding this comment.
Then should we remove it from here? It is specified in the S3 type as well
pkg/psmdb/statefulset.go
Outdated
| if storage.Type == api.BackupStorageMinio { | ||
| if storage.Minio.CABundle != nil && | ||
| storage.Minio.CABundle.Name != "" && | ||
| storage.Minio.CABundle.Key != "" { |
There was a problem hiding this comment.
Should we default the key to ca.crt if unspecified?
There was a problem hiding this comment.
Ye, I think it's a good suggestion. Added.
Co-authored-by: Mayank Shah <mayank.shah@percona.com>
pkg/psmdb/statefulset.go
Outdated
| return result | ||
| } | ||
|
|
||
| type caRef struct{ Name, Key string } |
There was a problem hiding this comment.
Do we really need a new type for this? Can't we use SecretKeySelector, since it contains the same fields?
pkg/psmdb/statefulset.go
Outdated
| cas := collectStorageCABundles(cr) | ||
| if len(cas) == 1 { | ||
| c.VolumeMounts = append(c.VolumeMounts, corev1.VolumeMount{ | ||
| Name: naming.BackupStorageCAFileVolumeName, | ||
| MountPath: naming.BackupStorageCAFileDirectory, | ||
| ReadOnly: true, | ||
| }) | ||
|
|
||
| } else if len(cas) > 1 { | ||
| c.VolumeMounts = append(c.VolumeMounts, | ||
| corev1.VolumeMount{ | ||
| Name: "ca-bundle-in", | ||
| MountPath: "/etc/s3/certs-in", | ||
| ReadOnly: true, | ||
| }, | ||
| corev1.VolumeMount{ | ||
| Name: naming.BackupStorageCAFileVolumeName, | ||
| MountPath: naming.BackupStorageCAFileDirectory, | ||
| ReadOnly: false, | ||
| }, | ||
| ) | ||
| } |
There was a problem hiding this comment.
Can we extract this to a helper function like getCAVolumes and getCAVolumeMounts? Might help reduce the overall complexity of the outer function and also allow us to add a basic unit test..
We can also do a bit of re-write to eleminate the else if by doing an early return, wdyt?
pkg/psmdb/statefulset.go
Outdated
| if cr.CompareVersion("1.22.0") >= 0 && cr.Spec.Backup.Enabled { | ||
| cas := collectStorageCABundles(cr) | ||
|
|
||
| if len(cas) == 1 { |
There was a problem hiding this comment.
On another thought, I wonder if we really need to check the length of the provided CAs..
What if the user-provided CAs always go to the ca-bundle-in volume (even if 1 CA is specified), and the init container append everything to the ca-bundle.crt in the emptydir volume like it does now.. That way we don't have to check the length here and can simplify the code, wdyt?
pkg/apis/psmdb/v1/psmdb_types.go
Outdated
| MaxUploadParts int32 `json:"maxUploadParts,omitempty"` | ||
| StorageClass string `json:"storageClass,omitempty"` | ||
| InsecureSkipTLSVerify bool `json:"insecureSkipTLSVerify,omitempty"` | ||
| CABundle *corev1.SecretKeySelector `json:"caBundle,omitempty"` |
There was a problem hiding this comment.
Then should we remove it from here? It is specified in the S3 type as well
…ator into K8SPSMDB-1418_add_ca
e7d1f5c
| desc 'Verify CA bundle infrastructure' | ||
|
|
||
| kubectl_bin exec ${cluster}-rs0-0 -n ${namespace} -c backup-agent -- \ | ||
| test -f /etc/s3/certs/ca-bundle.crt |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| test -f /etc/s3/certs/ca-bundle.crt | |
| test -f /etc/s3/certs/ca-bundle.crt |
| test -f /etc/s3/certs/ca-bundle.crt | ||
|
|
||
| kubectl_bin exec ${cluster}-rs0-0 -n ${namespace} -c backup-agent -- \ | ||
| env | grep SSL_CERT_FILE |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| env | grep SSL_CERT_FILE | |
| env | grep SSL_CERT_FILE |
| env | grep SSL_CERT_FILE | ||
|
|
||
| cert_count=$(kubectl_bin exec ${cluster}-rs0-0 -n ${namespace} -c backup-agent -- \ | ||
| grep -c "BEGIN CERTIFICATE" /etc/s3/certs/ca-bundle.crt) |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| grep -c "BEGIN CERTIFICATE" /etc/s3/certs/ca-bundle.crt) | |
| grep -c "BEGIN CERTIFICATE" /etc/s3/certs/ca-bundle.crt) |
| EOF | ||
|
|
||
| kubectl_bin wait --for=condition=Ready certificate/minio2-cert \ | ||
| -n ${namespace} --timeout=120s |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| -n ${namespace} --timeout=120s | |
| -n ${namespace} --timeout=120s |
| -n ${namespace} --timeout=120s | ||
|
|
||
| kubectl_bin get secret minio2-tls-temp -n ${namespace} \ | ||
| -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/minio2-cert.pem |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/minio2-cert.pem | |
| -o jsonpath='{.data.tls\.crt}' | base64 -d >/tmp/minio2-cert.pem |
| --from-file=private.key=/tmp/minio2-key.pem | ||
|
|
||
| kubectl_bin create secret generic minio2-ca-bundle -n ${namespace} \ | ||
| --from-file=ca.crt=/tmp/minio2-cert.pem |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| --from-file=ca.crt=/tmp/minio2-cert.pem | |
| --from-file=ca.crt=/tmp/minio2-cert.pem |
|
|
||
| desc "Verify merged CA bundle" | ||
| cert_count=$(kubectl_bin exec ${cluster}-rs0-0 -n ${namespace} -c backup-agent -- \ | ||
| grep -c "BEGIN CERTIFICATE" /etc/s3/certs/ca-bundle.crt) |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| grep -c "BEGIN CERTIFICATE" /etc/s3/certs/ca-bundle.crt) | |
| grep -c "BEGIN CERTIFICATE" /etc/s3/certs/ca-bundle.crt) |
| 'db.createUser({user:"myApp",pwd:"myPass",roles:[{db:"myApp",role:"readWrite"}]})' \ | ||
| "userAdmin:userAdmin123456@${cluster}-rs0.${namespace}" |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| 'db.createUser({user:"myApp",pwd:"myPass",roles:[{db:"myApp",role:"readWrite"}]})' \ | |
| "userAdmin:userAdmin123456@${cluster}-rs0.${namespace}" | |
| 'db.createUser({user:"myApp",pwd:"myPass",roles:[{db:"myApp",role:"readWrite"}]})' \ | |
| "userAdmin:userAdmin123456@${cluster}-rs0.${namespace}" |
| 'db.getSiblingDB("myApp").test.insert({ x: 100500 })' \ | ||
| "myApp:myPass@${cluster}-rs0.${namespace}" |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| 'db.getSiblingDB("myApp").test.insert({ x: 100500 })' \ | |
| "myApp:myPass@${cluster}-rs0.${namespace}" | |
| 'db.getSiblingDB("myApp").test.insert({ x: 100500 })' \ | |
| "myApp:myPass@${cluster}-rs0.${namespace}" |
| 'db.getSiblingDB("myApp").test.drop()' \ | ||
| "myApp:myPass@${cluster}-rs0.${namespace}" |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| 'db.getSiblingDB("myApp").test.drop()' \ | |
| "myApp:myPass@${cluster}-rs0.${namespace}" | |
| 'db.getSiblingDB("myApp").test.drop()' \ | |
| "myApp:myPass@${cluster}-rs0.${namespace}" |
| desc 'Verify CA bundle infrastructure' | ||
|
|
||
| kubectl_bin exec ${cluster}-rs0-0 -n ${namespace} -c backup-agent -- \ | ||
| test -f /etc/s3/certs/ca-bundle.crt |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| test -f /etc/s3/certs/ca-bundle.crt | |
| test -f /etc/s3/certs/ca-bundle.crt |
| test -f /etc/s3/certs/ca-bundle.crt | ||
|
|
||
| kubectl_bin exec ${cluster}-rs0-0 -n ${namespace} -c backup-agent -- \ | ||
| env | grep SSL_CERT_FILE |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| env | grep SSL_CERT_FILE | |
| env | grep SSL_CERT_FILE |
| env | grep SSL_CERT_FILE | ||
|
|
||
| cert_count=$(kubectl_bin exec ${cluster}-rs0-0 -n ${namespace} -c backup-agent -- \ | ||
| grep -c "BEGIN CERTIFICATE" /etc/s3/certs/ca-bundle.crt) |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| grep -c "BEGIN CERTIFICATE" /etc/s3/certs/ca-bundle.crt) | |
| grep -c "BEGIN CERTIFICATE" /etc/s3/certs/ca-bundle.crt) |
| EOF | ||
|
|
||
| kubectl_bin wait --for=condition=Ready certificate/minio2-cert \ | ||
| -n ${namespace} --timeout=120s |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| -n ${namespace} --timeout=120s | |
| -n ${namespace} --timeout=120s |
| -n ${namespace} --timeout=120s | ||
|
|
||
| kubectl_bin get secret minio2-tls-temp -n ${namespace} \ | ||
| -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/minio2-cert.pem |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/minio2-cert.pem | |
| -o jsonpath='{.data.tls\.crt}' | base64 -d >/tmp/minio2-cert.pem |
| --from-file=private.key=/tmp/minio2-key.pem | ||
|
|
||
| kubectl_bin create secret generic minio2-ca-bundle -n ${namespace} \ | ||
| --from-file=ca.crt=/tmp/minio2-cert.pem |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| --from-file=ca.crt=/tmp/minio2-cert.pem | |
| --from-file=ca.crt=/tmp/minio2-cert.pem |
|
|
||
| desc "Verify merged CA bundle" | ||
| cert_count=$(kubectl_bin exec ${cluster}-rs0-0 -n ${namespace} -c backup-agent -- \ | ||
| grep -c "BEGIN CERTIFICATE" /etc/s3/certs/ca-bundle.crt) |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| grep -c "BEGIN CERTIFICATE" /etc/s3/certs/ca-bundle.crt) | |
| grep -c "BEGIN CERTIFICATE" /etc/s3/certs/ca-bundle.crt) |
| 'db.createUser({user:"myApp",pwd:"myPass",roles:[{db:"myApp",role:"readWrite"}]})' \ | ||
| "userAdmin:userAdmin123456@${cluster}-rs0.${namespace}" |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| 'db.createUser({user:"myApp",pwd:"myPass",roles:[{db:"myApp",role:"readWrite"}]})' \ | |
| "userAdmin:userAdmin123456@${cluster}-rs0.${namespace}" | |
| 'db.createUser({user:"myApp",pwd:"myPass",roles:[{db:"myApp",role:"readWrite"}]})' \ | |
| "userAdmin:userAdmin123456@${cluster}-rs0.${namespace}" |
| 'db.getSiblingDB("myApp").test.insert({ x: 100500 })' \ | ||
| "myApp:myPass@${cluster}-rs0.${namespace}" |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| 'db.getSiblingDB("myApp").test.insert({ x: 100500 })' \ | |
| "myApp:myPass@${cluster}-rs0.${namespace}" | |
| 'db.getSiblingDB("myApp").test.insert({ x: 100500 })' \ | |
| "myApp:myPass@${cluster}-rs0.${namespace}" |
| 'db.getSiblingDB("myApp").test.drop()' \ | ||
| "myApp:myPass@${cluster}-rs0.${namespace}" |
There was a problem hiding this comment.
[shfmt] reported by reviewdog 🐶
| 'db.getSiblingDB("myApp").test.drop()' \ | |
| "myApp:myPass@${cluster}-rs0.${namespace}" | |
| 'db.getSiblingDB("myApp").test.drop()' \ | |
| "myApp:myPass@${cluster}-rs0.${namespace}" |
commit: 698b94e |
9c2c4e2
CHANGE DESCRIPTION
Problem:
Currently, when using MinIO with corporate PKI, users must set insecureSkipTLSVerify: true, which disables TLS validation entirely. This is a security concern for production environments.
Cause:
Short explanation of the root cause of the issue if applicable.
Solution:
This PR adds support for custom CA bundles when using MinIO as backup storage with corporate PKI certificates. The operator now mounts CA certificates into backup-agent containers and configures the SSL_CERT_FILE environment variable for TLS validation.
Added caBundle field to BackupStorageMinioSpec
Volume Management:
For 1+ CA bundles: Creates ProjectedVolume that merges certificates
Mounts to /etc/s3/certs-in/ (read-only) and /etc/s3/certs/ (read-write emptyDir)
Entrypoint script merges certificates: cat /etc/s3/certs-in/*.crt > /etc/s3/certs/ca-bundle.crt
Key Features:
Auto-merge: Multiple CA bundles are automatically merged into single file
Default key: If key is not specified, defaults to ca.crt
Items filtering: Only specified key is mounted from Secret (prevents mounting tls.crt, tls.key from cert-manager Secrets)
Version gated: Only applies to CR version >= 1.22.0
CHECKLIST
Jira
Needs Doc) and QA (Needs QA)?Tests
compare/*-oc.yml)?Config/Logging/Testability