Validate HTTPS connection from NiFi to service with custom certificate

[dmasice]

Hi,

I'm trying to configure NiFi on our k8s cluster (with nifi-operator) to use our keycloak server for authentication. This server uses a custom HTTPS certificate from our own self-signed CA and is outside of the k8s cluster. I've already configured the NiFi cluster to use the oidc configuration, but fails to start when fetching the discovery url as it fails to verify the server certificate.

Is there a way to add our CA certificate to the keystore used by NiFi or the one from the JVM so that it can verify the server certificate? I see that the NiFi keystore seems to be managed by the secrets-operator, but I don't know if there's some option to manage the JVM cacerts and how to add more certificates.

Thanks.

Best regards.

[dmasice]

Another option is to create a custom image and add the CA certificate there, but I rather not have to do that. I could mount the certificate with a configmap or a secret, but I need to be able to update the cacerts file (via update-ca-trust) before starting the NiFi process.

A third option is to replace the cacerts file and mount my own, but that requires more maintenance to keep it updated than to update the one in the image when starting NiFi.

[soenkeliebau]

Hi @dmasice
normally we'd recommend to use the extral volumes feature to mount CA certs into pods: https://docs.stackable.tech/home/stable/nifi/usage_guide/extra-volumes.html

I am not 100% sure if you can configure the location of the keystore or pem file for your use case or if it absolutely has to be in the system keystore? But if you can specify where the cert is located you could use this to mount the CA from a secret/configmap and then use it.

[dmasice]

That was my first option, but for this case it won't work, as the nifi.security.user.oidc.truststore.strategy specifies that it has to be either the java cacerts file or the NiFi keystore defined in nifi.security.truststore. That means that I need to add the cert to one of those stores.

Is it possible to specify extra volumes outside of /stackable/userdata? Maybe using podOverrides? I'm thinking about creating a one time job to add the certificate to cacerts and copy that to another volume, then mount that in NiFi to replace the original cacerts file.

[dmasice]

I got it working with a volume and a batch job. First I created a persistentVolumeClaim to store the modified cacerts file:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nifi-cacerts-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Mi
  storageClassName: ceph-filesystem

and a configMap for the remote server CA certificate:

kubectl create configmap --from-file remote-server-ca.crt remote-server-ca-certificate -o yaml

Then, using the same NiFi image, I created a one time job that updates the cacerts file and makes a copy of the file into the volume:

apiVersion: batch/v1
kind: Job
metadata:
  name: update-nifi-cacerts
spec:
  template:
    spec:
      securityContext:
        fsGroup: 0
        runAsGroup: 0
        runAsUser: 0
      containers:
      - name: update-nifi-cacerts
        image: docker.stackable.tech/stackable/nifi:1.21.0-stackable23.7.0
        command:
          - "/bin/bash"
          - "-c"
          - "-euo"
          - "pipefail"
        args:
          - "echo Updating cacerts && /usr/bin/update-ca-trust && echo Copying cacerts to volume && cp /etc/pki/ca-trust/extracted/java/cacerts /data/nifi-cacerts/"
        volumeMounts:
        - name: nifi-cacerts
          mountPath: /data/nifi-cacerts
        - name: remote-server-ca-certificate
          mountPath: /etc/pki/ca-trust/source/anchors
      volumes:
      - name: nifi-cacerts
        persistentVolumeClaim:
          claimName: nifi-cacerts-pvc
      - name: remote-server-ca-certificate
        configMap:
          name: remote-server-ca-certificate
      restartPolicy: Never
  backoffLimit: 4

The securityContext configuration is needed to run the commands as root in order to be able to update the cacerts file.

The last step is to modify the NiFi cluster definition to mount the new cacerts file. I've used podOverrides to be able to mount the file at the required path inside the container instead of using the extraVolumes option:

spec:
  nodes:
    podOverrides:
      spec:
        containers:
          - name: nifi
            volumeMounts:
            - name: nifi-cacerts
              mountPath: /etc/pki/ca-trust/extracted/java/cacerts
              subPath: cacerts
        volumes:
        - name: nifi-cacerts
          persistentVolumeClaim:
            claimName: nifi-cacerts-pvc

With this, I'm able to add custom certificates without having to rebuild the image. If I update the image, I just need to run the job again to regenerate the cacerts file.

If there's a better/simpler way to add custom certificates in the future, that'll be great. But, for now, this will do.

[dmasice]

After using the method avobe, I've been able to get NiFi to validate the url of the Keycloak server. But after Keycloak validates the user I get the following error:

Insufficient Permissions

Untrusted proxy CN=generated certificate for pod

In the log I see the following:

2023-07-20 14:17:39,819 INFO [NiFi Web Server-176] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 10.233.67.181 [<[email protected]><CN=generated certificate for pod>] GET https://simple-nifi-node-default-0.simple-nifi-node-default.default.svc.cluster.local:8443/nifi-api/flow/current-user
2023-07-20 14:17:39,825 WARN [NiFi Web Server-176] o.a.n.w.s.NiFiAuthenticationFilter Authentication Failed 10.233.67.181 GET https://simple-nifi-node-default-0.simple-nifi-node-default.default.svc.cluster.local:8443/nifi-api/flow/current-user [Untrusted proxy CN=generated certificate for pod]
2023-07-20 14:17:39,826 INFO [NiFi Web Server-176] org.apache.nifi.web.server.RequestLog 10.233.67.181 - - [20/Jul/2023:14:17:39 +0000] "GET /nifi-api/flow/current-user HTTP/1.1" 403 49 "https://nifi.k8s.myserver.local/nifi/" "Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0"

It seems that NiFi is unable to validate the certificate for https://simple-nifi-node-default-0.simple-nifi-node-default.default.svc.cluster.local:8443. This certificate (/stackable/keystore/tls.crt) seems to be generated everytime that the pod starts and needs to be added to /etc/pki/ca-trust/extracted/java/cacerts for NiFi to validate the request. Also, it may be necessary to add the certificates from the other pods of the cluster to each pod, as one cluster node might need to make a request to another cluster node.

[dmasice]

I'm thinking about replacing the autogenerated certificate with my own and thus be able to add it to the cacerts file. Is this certificate used elsewhere? Right now I'm reading the documentation about secretClass and thinking about creating a new one with k8sSearch as backend and using that instead of the default for NiFi.

Also, I'm thinking about adding another initcontainer to the cluster to update the cacerts file on pod start instead of having the batch job that I need to run manually.

[sbernauer]

Both seems like valid approaches to me. Would you mind sharing your whole config?
i think we did also run into the Untrusted proxy CN=generated certificate for pod and added the following two settings to trust them: https://github.com/stackabletech/nifi-operator/blob/9795944bd73cb116f7953769a049c5ab7fc99569/rust/crd/src/authentication.rs#L628 and https://github.com/stackabletech/nifi-operator/blob/9795944bd73cb116f7953769a049c5ab7fc99569/rust/crd/src/authentication.rs#L642

[sbernauer]

Regarding a initContainer, this should be an working example:
Please note that we set

kubectl get cm simple-nifi-node-default -o yaml | grep 'nifi.security.keystore'
    nifi.security.keystore=/stackable/keystore/keystore.p12
    nifi.security.keystorePasswd=secret
    nifi.security.keystoreType=PKCS12

apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
  name: simple-nifi
spec:
  image:
    productVersion: 1.21.0
    stackableVersion: "0.0.0-dev"
  clusterConfig:
    authentication:
      method:
        singleUser:
          adminCredentialsSecret: nifi-admin-credentials-simple
          autoGenerate: true
    listenerClass: external-unstable
    sensitiveProperties:
      keySecret: nifi-sensitive-property-key
      autoGenerate: true
    zookeeperConfigMapName: simple-nifi-znode
  nodes:
    podOverrides:
      spec:
        initContainers:
          - name: prepare-add-custom-cert
            args:
              - |
                ls -la /stackable/keystore
                ls -la /custom-cert-secret
                keytool -list -keystore /stackable/keystore/truststore.p12 -storetype PKCS12 -storepass secret
                keytool -importcert -file /custom-cert-secret/ca.crt -keystore /stackable/keystore/truststore.p12 -storetype pkcs12 -storepass secret -noprompt -alias custom_ca_cert
                keytool -list -keystore /stackable/keystore/truststore.p12 -storetype PKCS12 -storepass secret
            command:
              - /bin/bash
              - -c
              - -euo
              - pipefail
            image: docker.stackable.tech/stackable/nifi:1.21.0-stackable0.0.0-dev
            imagePullPolicy: IfNotPresent
            volumeMounts:
              - mountPath: /stackable/keystore
                name: keystore
              - mountPath: /custom-cert-secret
                name: custom-cert-secret
        volumes:
          - name: custom-cert-secret
            secret:
              secretName: secret-provisioner-tls-ca # change to custom-cert that contains ca.crt to add
    roleGroups:
      default:
        replicas: 1

Does it work for you as well?

[dmasice]

Well, adding those two lines did it, so that means that I don't need to add those certificates to cacerts. I did try adding those lines with the names of the nodes, but it didn't work because it didn't match the DN of the certificate.

Thank you.

I need to clean it up a bit, but will share the config later.

BTW, those certificates are valid for just 1 day. Does it mean that the pod needs to be restarted every day to regenerate them? For a development environment it does not matter that much, but for any other kind of (more stable) environment this could be a problem.

[sbernauer]

Yes, the lifetime of 24h is intentional. The good news is that we have code in place that adds the annotation restarter.stackable.tech/expires-at.b584f0a2e4834d7e0d9733280f570b25: "2023-07-22T08:07:50.598587667+00:00" to the Pods.
The result is that the commons-operator restarts the Pod before the certificate expires, so you should not have Pods using expired certificates

[sbernauer]

We might make the cert lifetime configurable at some point of time, especially for products without HA support