fluentd pods crashing specified cert path does not exist:/var/run/ocp-collector/secrets/openshift-logforwarding-splunk/ca-bundle.crt"

[walidshaari]

Hi Andrew,

I follow your README file, installed the local splunk, however when I try to deploy the openshift-logforwarding-splunk

helm upgrade -i --namespace=openshift-logging openshift-logforwarding-splunk charts/openshift-logforwarding-splunk/ --set forwarding.splunk.token=4a8a737d-5452-426c-a6f7-106dca4e813f

fluentd pods start crashing loop and I get the error "specified cert path does not exist:/var/run/ocp-collector/secrets/openshift-logforwarding-splunk/ca-bundle.crt" so I thought maybe I am not using the right value so I specify the values.yaml file using -f option, same error, I do not do upgrade, but I uninstall and deploy

helm upgrade -i --namespace=openshift-logging openshift-logforwarding-splunk charts/openshift-logforwarding-splunk/ -f charts/openshift-logforwarding-splunk/values.yaml --set forwarding.splunk.token=4a8a737d-5452-426c-a6f7-106dca4e813f 

then I use both -f and --set

helm upgrade -i --namespace=openshift-logging openshift-logforwarding-splunk charts/openshift-logforwarding-splunk/ -f charts/openshift-logforwarding-splunk/values.yaml --set forwarding.splunk.token=4a8a737d-5452-426c-a6f7-106dca4e813f  --set forwarding.fluentd.caFile=./charts/openshift-logforwarding-splunk/files/default-openshift-logging-fluentd.crt

#oc logs fluentd-4s9xl

2020-08-11 06:54:28 +0000 [error]: config error file="/etc/fluent/fluent.conf" error_class=Fluent::ConfigError error="specified cert path does not exist:/var/run/ocp-collector/secrets/openshift-logforwarding-splunk/ca-bundle.crt"

# oc logs fluentd-mx548

2020-08-11 14:10:27 +0000 [error]: config error file="/etc/fluent/fluent.conf" error_class=Fluent::ConfigError error="specified cert path does not exist:/var/run/ocp-collector/secrets/openshift-logforwarding-splunk/ca-bundle.crt"

confession: I have not read your book yet, I promise I will do soon, any hints where I have gone wrong?

[sabre1041]

Just tested it on OpenShift 4.5 using the steps described in this issue. Deployed without issues. Can you share more about your environment and any additional information that you can provide?

[walidshaari]

its disconnected 4.5 installation, will check again tomorrow, it is at work
its when I select SSL value for Fluentd to be false.

[sabre1041]

@walidshaari any update?

[walidshaari]

I have to enable the TLS, if I enable TLS I do not have any issues.

[sabre1041]

@walidshaari Is this in fluentd as part of the integrated OCP or as part of the forwarder?

[walidshaari]

I believe it is the fluentd communication with each other as part of the OCP, not the forwarder

[sabre1041]

@walidshaari I looked into the issue. there should be a secret called openshift-logforwarding-splunk in the openshift namespace with a key called ca-bundle.crt.

https://github.com/sabre1041/openshift-logforwarding-splunk/blob/master/charts/openshift-logforwarding-splunk/templates/log-forwarding-splunk-secret.yaml#L21

This is what the fluentd pod is trying to locate, but is failing

[walidshaari]

yes, I can try again next week, but no matter how I did it last time, worked