|
| 1 | +--- |
| 2 | +title: Adopting Sidecar Containers |
| 3 | +content_type: tutorial |
| 4 | +weight: 40 |
| 5 | +min-kubernetes-server-version: 1.29 |
| 6 | +--- |
| 7 | + |
| 8 | +<!-- overview --> |
| 9 | + |
| 10 | +This section is relevant for people adopting a new built-in [sidecar containers](/docs/concepts/workloads/pods/sidecar-containers/) feature for their workloads. |
| 11 | + |
| 12 | +Sidecar containers is not a new concept as posted in the [blog post](https://kubernetes.io/blog/2015/06/the-distributed-system-toolkit-patterns/). |
| 13 | +Kubernetes allowed to run multiple containers in a Pod to implement this concept. However, running sidecar container as a regular container |
| 14 | +has a lot of limitations being fixed with the new built-in sidecar containers support. |
| 15 | + |
| 16 | +{{< feature-state feature_gate_name="SidecarContainers" >}} |
| 17 | + |
| 18 | +## {{% heading "objectives" %}} |
| 19 | + |
| 20 | +* Understand the need for sidecar containers |
| 21 | +* Be able to troubleshoot issues with the sidecar containers |
| 22 | +* Understand options to universally "inject" sidecar containers to any workload |
| 23 | + |
| 24 | + |
| 25 | +## {{% heading "prerequisites" %}} |
| 26 | + |
| 27 | +{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} |
| 28 | + |
| 29 | + |
| 30 | + |
| 31 | +<!-- lessoncontent --> |
| 32 | + |
| 33 | +## Sidecar containers overview |
| 34 | + |
| 35 | +Sidecar containers are the secondary containers that run along with the main |
| 36 | +application container within the same {{< glossary_tooltip text="Pod" term_id="pod" >}}. |
| 37 | +These containers are used to enhance or to extend the functionality of the primary _app |
| 38 | +container_ by providing additional services, or functionality such as logging, monitoring, |
| 39 | +security, or data synchronization, without directly altering the primary application code. |
| 40 | +You can read more in the [Sidecar containers](/docs/concepts/workloads/pods/sidecar-containers/) |
| 41 | +concept page. |
| 42 | + |
| 43 | +The concept of sidecar containers is not new and there are multiple implementation of this concept. |
| 44 | +As well as sidecar containers that you, the person defining the Pod, want to run, you can also find |
| 45 | +that some {{< glossary_tooltip text="addons" term_id="addons" >}} modify Pods - before the Pods |
| 46 | +start running - so that there are extra sidecar containers. The mechanisms to _inject_ those extra |
| 47 | +sidecars are often [mutating webhooks](/docs/reference/access-authn-authz/admission-controllers/ |
| 48 | +#mutatingadmissionwebhook). |
| 49 | +For example, a service mesh addon might inject a sidecar that configures mutual TLS and encryption |
| 50 | +in transit between different Pods. |
| 51 | + |
| 52 | +While the concept of sidecar containers is not new, |
| 53 | +the native implementation of this feature in Kubernetes, however, is new. And as with every new feature, |
| 54 | +adopting this feature may present certain challenges. |
| 55 | + |
| 56 | +This tutorial explore challenges and solution can be experienced by end users as well as |
| 57 | +by authors of sidecar containers. |
| 58 | + |
| 59 | +## Benefits of a built-in sidecar containers |
| 60 | + |
| 61 | +Using Kubernetes' native support for sidecar containers provides several benefits: |
| 62 | + |
| 63 | +1. You can configure a native sidecar container to start ahead of {{< glossary_tooltip text="init containers" term_id="init-container" >}}. |
| 64 | +2. The built-in sidecar containers can be authored to guarantee that they are terminated last. |
| 65 | + Sidecar containers are terminated with a `SIGTERM` signal once all the regular containers |
| 66 | + are completed and terminated. If the sidecar container isn’t gracefully shut down, a |
| 67 | + `SIGKILL` signal will be used to terminate it. |
| 68 | +3. With Jobs, when Pod's `restartPolicy: OnFailure` or `restartPolicy: Never`, |
| 69 | + native sidecar containers do not block Pod completion. With legacy sidecar containers, |
| 70 | + special care is needed to handle this situation. |
| 71 | +4. Also, with Jobs, built-in sidecar containers would keep being restarted once they are done, even if regular containers would not with Pod's `restartPolicy: Never`. |
| 72 | + |
| 73 | +## Adopting built-in sidecar containers |
| 74 | + |
| 75 | +The `SidecarContainers` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is in beta state starting from Kubernetes version 1.29 and is enabled by default. |
| 76 | +Some clusters may have this feature disabled or have software installed that is incompatible with the feature. |
| 77 | + |
| 78 | +When this happens, the Pod may be rejected or the sidecar containers may block Pod startup, rendering the Pod useless. |
| 79 | +This condition is easy to detect as Pod simply get stuck on initialization. However, it is rarely clear what caused the problem. |
| 80 | + |
| 81 | +Here are the considerations and troubleshooting steps that one can take while adopting sidecar containers for their workload. |
| 82 | + |
| 83 | +### Ensure the feature gate is enabled |
| 84 | + |
| 85 | +As a very first step, make sure that both - API server and Nodes are at Kubernetes version v1.29 or |
| 86 | +later. |
| 87 | +The feature will break on clusters where Nodes are running earlier versions where it is not enabled. |
| 88 | + |
| 89 | +{{< alert title="Note" color="info" >}} |
| 90 | + |
| 91 | +The feature can be enabled on nodes with the version 1.28. The behavior of built-in sidecar |
| 92 | +container termination was different in version 1.28 and it is not recommended to adjust |
| 93 | +behavior of a sidecar to that behavior. However if that only concern is the startup order, the |
| 94 | +above statement can be changed to Nodes, running version 1.28 with the feature gate enabled. |
| 95 | + |
| 96 | +{{< /alert >}} |
| 97 | + |
| 98 | +You should ensure that the feature gate is enabled for the API server(s) within the control plane |
| 99 | +**and** for all nodes. |
| 100 | + |
| 101 | +One of the ways to check the feature gate enablement is running a command like this: |
| 102 | + |
| 103 | +- For API Server |
| 104 | + `kubectl get --raw /metrics | grep kubernetes_feature_enabled | grep SidecarContainers` |
| 105 | +- For the individual node: |
| 106 | + `kubectl get --raw /api/v1/nodes/<node-name>/proxy/metrics | grep kubernetes_feature_enabled | grep SidecarContainers` |
| 107 | + |
| 108 | +If you see something like: `kubernetes_feature_enabled{name="SidecarContainers",stage="BETA"} 1`, |
| 109 | +it means that the feature is enabled. |
| 110 | + |
| 111 | +### Check for 3rd party tooling and mutating webhooks |
| 112 | + |
| 113 | +If you experience issues when validating the feature, it may be an indication that one of the |
| 114 | +3rd party tools or mutating webhooks are broken. |
| 115 | + |
| 116 | +When the `SidecarContainers` feature gate is enabled, Pods gain a new field in their API. |
| 117 | +Some tools or mutating webhooks might have been built with an earlier version of Kubernetes API. |
| 118 | + |
| 119 | +If tools pass the unknown fields as-is using various patching strategies to mutate a Pod object, |
| 120 | +this will not be a problem. However there are tools that will strip out unknown fields; |
| 121 | +if you have those, they must be recompiled with the v1.28+ version of Kubernetes API client code. |
| 122 | + |
| 123 | +The way to check this is to use the `kubectl describe pod` command with your Pod that has passed through |
| 124 | +mutating admission. |
| 125 | +If any tools stripped out the new field (`restartPolicy:Always`), you will not see it in the command output. |
| 126 | + |
| 127 | +If you hit an issue like this, please advise the author of the tools or the webhooks use one of the patching strategies of modifying objects instead of a full object update. |
| 128 | + |
| 129 | +{{< alert title="Note" color="info" >}} |
| 130 | + |
| 131 | +Mutating webhook may update Pods based on some conditions. So sidecar containers may work for some Pods and fail for others. |
| 132 | + |
| 133 | +{{< /alert >}} |
| 134 | + |
| 135 | +### Automatic injection of sidecars |
| 136 | + |
| 137 | +If you are using software that injects sidecars automatically, |
| 138 | +there are a few possible strategies you may follow to |
| 139 | +ensure that native sidecar container can be used. |
| 140 | +All of the strategies are generally options you may choose to decide whether |
| 141 | +the Pod the sidecar will be injected to will land on a Node supporting the feature or not. |
| 142 | + |
| 143 | +As an example, you can follow [this conversation in Istio community](https://github.com/istio/istio/issues/48794). The discussion is exploring the options listed below. |
| 144 | + |
| 145 | +1. Mark Pods that lands to nodes supporting sidecars. You can use node labels |
| 146 | + and node affinity to mark nodes supporting sidecar containers and Pods landing on those nodes. |
| 147 | +2. Check Nodes compatibility on injection. During sidecar injection you may use the following strategies to check node compatibility: |
| 148 | + - query node version and assume the feature gate is enabled on the version 1.29+ |
| 149 | + - query node prometheus metrics and check feature enablement status |
| 150 | + - assume the nodes are running with a [supported version skew](/releases/version-skew-policy/#supported-version-skew) |
| 151 | + from the API server |
| 152 | + - there may be other custom ways to detect nodes compatibility. |
| 153 | +3. Develop a universal sidecar injector. The idea of a universal sidecar container is to inject a sidecar container |
| 154 | + as a regular container as well as a native sidecar container. And have a runtime logic to decide which one will work. |
| 155 | + The universal sidecar injector is wasteful as it will account for requests twice, but may be considered as a workable solution for special cases. |
| 156 | + - One way would be on start of a native sidecar container |
| 157 | + detect the node version and exit immediately if the version does not support the sidecar feature. |
| 158 | + - Consider runtime feature detection design: |
| 159 | + - Define an empty dir so containers can communicate with each other |
| 160 | + - Inject init container, let's call it `NativeSidecar` with `restartPolicy=Always`. |
| 161 | + - `NativeSidecar` must write a file to an empty dir indicating the first run and exists immediately with exit code `0`. |
| 162 | + - `NativeSidecar` on restart (when native sidecars are supported) checks that file already exists in the empty dir and changes it - indicating that the built-in sidecar containers are supported and running. |
| 163 | + - Inject regular container, let's call it `OldWaySidecar`. |
| 164 | + - `OldWaySidecar` on start checks the presence of a file in an empty dir. |
| 165 | + - If the file indicates that the `NativeSidecar` is NOT running - it assumes that the sidecar feature is not supported and works assuming it is the sidecar. |
| 166 | + - If the file indicates that the `NativeSidecar` is running - it either does nothing and sleeps forever (in case when Pod’s `restartPolicy=Always`) or exists immediately with exit code `0` (in case when Pod’s `restartPolicy!=Always`). |
| 167 | + |
| 168 | + |
| 169 | +## {{% heading "whatsnext" %}} |
| 170 | + |
| 171 | + |
| 172 | +* Learn more about [sidecar containers](/docs/concepts/workloads/pods/sidecar-containers/). |
0 commit comments