HELP! Traefik pod is failling in one of the kubernetes node

[Manikanta-Reddy-Pasala]

Hello Team,

Need support in solving issue in production.

I see one of the production is node not selected I went through the pods there is see Traefik job is falling continuously.

I tried to add another node to the cluster even then it is the same issue. traefik pod is not running on the new node.

attached the log file.

please help me to resolve the issue.

helm-install-traefik-27hv7.log

[mysticaltech]

@Manikanta-Reddy-Pasala Most probably traefik not running on a few nodes is normal, it just depends on the default deployment setup, as it is not a daemonset (which would run on every node). You can definitely configure that, please see the traefik helm values here and copy the portion you need (in your case):

deployment:
  enabled: true
  # Can be either Deployment or DaemonSet
  kind: Deployment

Change the kind to DaemontSet, it will give you the behavior you desire, without worrying about the number of replicas.

And those helm values, you need to apply them via a HelmChartConfig definition. Luckily, the cluster already has one. Just pull it via kubectl get helmchartconfig -n kube-system -o yaml, and under the valuesContent at the top of the section, add the values above.

---

Now about the upgrade, it is failing, because somehow it needs to reinstall, and this we do not allow without user intervention, as it could change the value of the LB IP. So we have in the HelmChartConfig this param set failurePolicy: abort, just delete that line (a few minutes to do the upgrades a least, then you can add it back for security).

And apply this manifest with kubectl; it will force the upgrade on the spot: https://github.com/k3s-io/k3s/blob/master/manifests/traefik.yaml.

Then you should no longer have helm jobs failing and you will, if you have chosen a deployment kind of DaemonSet in the HelmChartConfig, see traefik running on all nodes!

Basically, this last manifest contains the new upgraded HelmChart definition, and your HelmChartConfig holds all the needed custom values. See more on the dance between HelmChart and HelmChartConfig definitions here https://docs.k3s.io/helm (if you are curious).

Good luck, and if somehow this does not work, just paste your HelmChartConfig in here.

[Manikanta-Reddy-Pasala]

Thanks for the reply..

I tried to update the deployment yaml. but it didn't work. finally, I have deleted and recreated the cluster once again..

[mysticaltech]

That was overkill... You should have posted your HelmChartConfig in here; it probably was just an indentation problem. But ok, glad it's now ok.