feat: add global certificate management for ZTVP applications #83

minmzzhang · 2025-12-16T03:19:07Z

Add ztvp-certificates chart for managing trusted CA certificates across all ZTVP applications with support for custom certificates via Kubernetes secret references.

Key features:

Initial job for immediate certificate extraction on install
Cronjob for periodic certificate rotation (daily at 2AM)
Automatic rollout restart for consuming applications (labeled strategy)
Auto-detects OpenShift ingress, service and proxy CA certificates
Supports custom CA certificates via secretRef (primary + additional)
Distributes certificates to target namespaces via ACM Policy

Certificate Sources:

Primary custom CA via secretRef (customCA.secretRef)
Additional certificates via extraValueFiles (overrides/values-ztvp-certificates.yaml)
Auto-detected proxy CA from trusted-ca-bundle (openshift-config-managed)
Auto-detected ingress CA from all IngressControllers (not just default)
Auto-detected service CA from openshift-service-ca

Configuration:

Use extraValueFiles for complex nested structures (additionalCertificates, rollout)
Simple overrides via values-hub.yaml for flat key-value pairs

Chart deploys:

Job/cronjob to extract and combine CA certificates
ConfigMap with combined CA bundle (ztvp-trusted-ca)
ACM Policy to distribute to target namespaces
RBAC permissions for CA extraction

Applications can consume the CA bundle by:

Mounting the ztvp-trusted-ca ConfigMap at /etc/pki/ca-trust/extracted/pem/
Setting ZTVP_CA_BUNDLE environment variable
Using standard SSL/TLS libraries that check system CA paths

sabre1041

Looking really good. A few comments

sabre1041 · 2025-12-17T11:48:08Z

charts/ztvp-certificates/values.yaml

+      memory: 128Mi
+    limits:
+      cpu: 200m
+      memory: 256Mi


Recommend increasing these values. My cluster OOMKilled the container

increased, this size mismatch was due to the addition of proxy CA extraction.

sabre1041 · 2025-12-17T12:06:34Z

.github/workflows/superlinter.yml

          # These are the validation we disable atm
          VALIDATE_ANSIBLE: false
          VALIDATE_BASH: false
+          VALIDATE_BASH_EXEC: false


Is this because we do not set the execute bit on the file?

The shell script was a helm template, which has the helm syntax inside, bash_exec cannot pass it. I've updated the filename to avoid this single file's bash_exec linter execution.

What I usually do in these cases is have a simple script and control the variables and conditions using environment variables. Another option is to have source files and import them depending on the environment

The FILTER_REGEX_EXCLUDE option also works for me

charts/ztvp-certificates/templates/ca-extraction-cronjob.yaml

mlorenzofr · 2025-12-17T21:13:36Z

I think it might be interesting to extend the functionality to 2 additional components

External Secrets Operator

Currently, the ClusterSecretStore vault-backend has a configuration section where the CA it uses is specified:

       caProvider:
          key: ca.crt
          name: kube-root-ca.crt
          namespace: golang-external-secrets
          type: ConfigMap

This can be configured via an override in values-hub.yaml:

    golang-external-secrets:
      name: golang-external-secrets
      namespace: golang-external-secrets
      project: hub
      chart: golang-external-secrets
      chartVersion: 0.1.*
      overrides:
        - name: golangExternalSecrets.caProvider.hostCluster.name
          value: ztvp-trusted-ca
        - name: golangExternalSecrets.caProvider.hostCluster.key
          value: tls-ca-bundle.pem
        - name: golangExternalSecrets.caProvider.hostCluster.namespace
          value: openshift-config

Should we change it permanently if we adopt this solution?

Image registries (Quay)

The cluster can use image registries that use internal certificates. In this case, the configuration is done in the images.config.openshift.io resource.

apiVersion: config.openshift.io/v1
kind: Image
  name: cluster
spec:
  additionalTrustedCA:
    name: ztvp-registry-trusted-ca

The additionalTrustedCA attribute references a ConfigMap in the openshift-config namespace. The ConfigMap contains a list of the registry hostnames and trusted CAs:

apiVersion: v1
data:
  quay-registry-quay-quay-enterprise.apps.example.domain: |
    -----BEGIN CERTIFICATE-----
    ... ...
kind: ConfigMap
metadata:
  name: ztvp-registry-trusted-ca
  namespace: openshift-config

This functionality would allow configuring CRI-O on the nodes to download images from registries with internal certificates.

…Files Implements comprehensive certificate management for ZTVP: Certificate Sources: - Primary custom CA via secretRef (customCA.secretRef) - Additional certificates via extraValueFiles (overrides/values-ztvp-certificates.yaml) - Auto-detected proxy CA from trusted-ca-bundle (openshift-config-managed) - Auto-detected ingress CA from all IngressControllers (not just default) - Auto-detected service CA from openshift-service-ca Features: - Initial Job for immediate certificate extraction on install - CronJob for periodic certificate rotation (daily at 2 AM) - Warning and continue behavior for missing additional certificates - Automatic rollout restart for consuming applications (labeled strategy) - ACM Policy distribution to target namespaces Configuration: - Use extraValueFiles for complex nested structures (additionalCertificates, rollout) - Simple overrides via values-hub.yaml for flat key-value pairs Signed-off-by: Min Zhang <[email protected]>

mlorenzofr · 2026-01-13T17:41:26Z

charts/qtodo/templates/app-deployment.yaml

+        # Configure Java to use custom truststore via system properties
+        # This applies to ALL Java SSL connections including Vert.x/Netty
+        - name: JAVA_TOOL_OPTIONS
+          value: "-Djavax.net.ssl.trustStore=/truststore/cacerts -Djavax.net.ssl.trustStoreType=JKS -Djavax.net.ssl.trustStorePassword=$(TRUSTSTORE_PASSWORD)"


Instead of configuring the variables at the JVM level, I would do it using the environment variables that Quarkus has for this: QUARKUS_OIDC_TLS_TRUST_STORE_FILE and QUARKUS_OIDC_TLS_TRUST_STORE_PASSWORD.

We could add them to the charts/qtodo/templates/app-config-env.yaml file, like other Quarkus variables.

Thank you @mlorenzofr , updated based on your suggestion that using Quarkus env variables is cleaner.

mlorenzofr · 2026-01-13T17:45:38Z

charts/qtodo/templates/app-deployment.yaml

          - name: db-credentials
            mountPath: /run/secrets/db-credentials
+          - name: truststore
+            mountPath: /truststore


Perhaps I would create it under /run/secrets or /etc/pki instead of creating a directory in the root

Thank you @mlorenzofr , updated the truststore to a more standard location based on your suggestion.

mlorenzofr

Good job, it works properly.

I'm adding a couple of small changes that I think would better align the solution with the style of other qtodo components.

mlorenzofr · 2026-01-14T12:37:32Z

charts/qtodo/values.yaml

    vaultPath: "secret/data/global/oidc-client-secret"
+
+  # Truststore configuration for Java CA certificates
+  truststore:


Suggested change

truststore:

truststore:

enabled: true

- Add init container to convert CA bundle to Java JKS truststore - Add ExternalSecret for truststore password from Vault - Configure JAVA_TOOL_OPTIONS for JVM-level SSL truststore - Mount ztvp-trusted-ca ConfigMap for CA certificates - Enable truststore by default when SPIRE is enabled Signed-off-by: Min Zhang <[email protected]>

Replace deprecated JKS truststore with modern PKCS12 format: - Use keytool with -storetype PKCS12 to create truststore - Configure via JAVA_TOOL_OPTIONS (Quarkus TLS registry doesn't work with OIDC) - Update Hibernate ORM env var to non-deprecated name - Add wait-for-keycloak init container to prevent OIDC 503 warnings PKCS12 is the modern standard recommended over Java-specific JKS format. Signed-off-by: Min Zhang <[email protected]>

minmzzhang force-pushed the ztvp-trusted-ca branch 27 times, most recently from fd1ada7 to 6407ec3 Compare December 17, 2025 05:08

minmzzhang requested review from mlorenzofr and sabre1041 December 17, 2025 05:20

sabre1041 requested changes Dec 17, 2025

View reviewed changes

minmzzhang force-pushed the ztvp-trusted-ca branch 5 times, most recently from cd77935 to 4349ad5 Compare December 17, 2025 15:45

minmzzhang force-pushed the ztvp-trusted-ca branch from 4349ad5 to 73e20c0 Compare January 12, 2026 15:38

mlorenzofr reviewed Jan 13, 2026

View reviewed changes

minmzzhang force-pushed the ztvp-trusted-ca branch 3 times, most recently from 73e20c0 to ac3a0ce Compare January 14, 2026 02:56

mlorenzofr reviewed Jan 14, 2026

View reviewed changes

minmzzhang force-pushed the ztvp-trusted-ca branch 7 times, most recently from a364831 to cbb9415 Compare January 14, 2026 14:58

minmzzhang force-pushed the ztvp-trusted-ca branch from cbb9415 to 9c5a3ef Compare January 14, 2026 15:25

feat: add global certificate management for ZTVP applications #83

Are you sure you want to change the base?

feat: add global certificate management for ZTVP applications #83

Uh oh!

Conversation

minmzzhang commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sabre1041 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

minmzzhang Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mlorenzofr commented Dec 17, 2025

External Secrets Operator

Image registries (Quay)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mlorenzofr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

minmzzhang commented Dec 16, 2025 •

edited

Loading

minmzzhang Dec 17, 2025 •

edited

Loading