feat: init container instrumentation (#4635)

* feat: initContainer injection

Signed-off-by: Alan Clucas <alan@clucas.org>

* chore: delete telemetry checks

Signed-off-by: Alan Clucas <alan@clucas.org>

* chore: review feedback

Signed-off-by: Alan Clucas <alan@clucas.org>

---------

Signed-off-by: Alan Clucas <alan@clucas.org>

Author: Joibel

.chloggen/init-container.yaml

@@ -0,0 +1,18 @@
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: enhancement
+
+# The name of the component, or a single word describing the area of concern, (e.g. collector, target allocator, auto-instrumentation, opamp, github action)
+component: auto-instrumentation
+
+# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: "Add support for initContainers to instrumentation injector"
+
+# One or more tracking issues related to the change
+issues: [3308]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext: |
+  Add support for instrumenting init containers.
+  Init container support is available for Java, Python, Node.js, .NET and SDK-only, and works using the same annotation as for regular containers.

README.md

@@ -361,12 +361,12 @@ The possible values for the annotation can be
 
 #### Multi-container pods with single instrumentation
 
-If nothing else is specified, instrumentation is performed on the first container available in the pod spec.
+If nothing else is specified, instrumentation is performed on the first container available in the pod spec (from `.spec.containers`, not init containers).
 In some cases (for example in the case of the injection of an Istio sidecar) it becomes necessary to specify on which container(s) this injection must be performed.
 
 For this, it is possible to fine-tune the pod(s) on which the injection will be carried out.
 
-For this, we will use the `instrumentation.opentelemetry.io/container-names` annotation for which we will indicate one or more container names (`.spec.containers.name`) on which the injection must be made:
+For this, we will use the `instrumentation.opentelemetry.io/container-names` annotation for which we will indicate one or more container names (from `.spec.containers.name` or `.spec.initContainers.name`) on which the injection must be made:
 
 ```yaml
 apiVersion: apps/v1
@@ -399,11 +399,59 @@ In the above case, `myapp` and `myapp2` containers will be instrumented, `myapp3
 
 > 🚨 **NOTE**: Go auto-instrumentation **does not** support multicontainer pods. When injecting Go auto-instrumentation the first pod should be the only pod you want instrumented.
 
+#### Instrumenting Init Containers
+
+Init containers can be instrumented by including their names in the `container-names` annotation. When an init container is targeted for instrumentation, the operator automatically inserts the instrumentation init container **before** the target init container in the pod's init container sequence. This ensures the instrumentation agent files are available when the target init container runs.
+
+Supported instrumentations for init containers:
+- Java
+- Python
+- Node.js
+- .NET
+- SDK-only injection
+
+**Not supported** for init containers:
+- Go (does not support multicontainer pods)
+- Apache HTTPD
+- Nginx
+
+> **Note**: Kubernetes guarantees that container names are unique across both the `initContainers` and `containers` lists within a pod spec. This allows the operator to unambiguously identify whether a container name refers to an init container or a regular container.
+
+Example with both init container and regular container instrumentation:
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: my-deployment-with-init-container
+spec:
+  selector:
+    matchLabels:
+      app: my-app
+  replicas: 1
+  template:
+    metadata:
+      labels:
+        app: my-app
+      annotations:
+        instrumentation.opentelemetry.io/inject-python: "true"
+        instrumentation.opentelemetry.io/container-names: "my-init-job,myapp"
+    spec:
+      initContainers:
+        - name: my-init-job
+          image: my-python-init-image
+      containers:
+        - name: myapp
+          image: my-python-app-image
+```
+
+In this example, both `my-init-job` (an init container) and `myapp` (a regular container) will be instrumented with Python auto-instrumentation.
+
 #### Multi-container pods with multiple instrumentations
 
 Works only when `enable-multi-instrumentation` flag is `true`.
 
-Annotations defining which language instrumentation will be injected are required. When feature is enabled, specific for Instrumentation language containers annotations are used:
+Annotations defining which language instrumentation will be injected are required. When feature is enabled, specific for Instrumentation language containers annotations are used (these also support init container names for Java, Python, Node.js, .NET, and SDK):
 
 Java:
 
@@ -453,11 +501,11 @@ SDK:
 instrumentation.opentelemetry.io/sdk-container-names: "app1,app2"
 ```
 
-If language instrumentation specific container names are not specified, instrumentation is performed on the first container available in the pod spec (only if single instrumentation injection is configured).
+If language instrumentation specific container names are not specified, instrumentation is performed on the first regular container available in the pod spec (only if single instrumentation injection is configured).
 
 In some cases containers in the pod are using different technologies. It becomes necessary to specify language instrumentation for container(s) on which this injection must be performed.
 
-For this, we will use language instrumentation specific container names annotation for which we will indicate one or more container names (`.spec.containers.name`) on which the injection must be made:
+For this, we will use language instrumentation specific container names annotation for which we will indicate one or more container names (`.spec.containers.name` or `.spec.initContainers.name`) on which the injection must be made:
 
 ```yaml
 apiVersion: apps/v1

internal/instrumentation/dotnet.go

@@ -39,27 +39,27 @@ const (
 	dotNetRuntimeLinuxMusl  = "linux-musl-x64"
 )
 
-func injectDotNetSDK(dotNetSpec v1alpha1.DotNet, pod corev1.Pod, container *corev1.Container, runtime string, instSpec v1alpha1.InstrumentationSpec) (corev1.Pod, error) {
+func injectDotNetSDKToContainer(dotNetSpec v1alpha1.DotNet, container *corev1.Container, runtime string) error {
 	volume := instrVolume(dotNetSpec.VolumeClaimTemplate, dotnetVolumeName, dotNetSpec.VolumeSizeLimit)
 
 	err := validateContainerEnv(container.Env, envDotNetStartupHook, envDotNetAdditionalDeps, envDotNetSharedStore)
 	if err != nil {
-		return pod, err
+		return err
 	}
 
 	// check if OTEL_DOTNET_AUTO_HOME env var is already set in the container
 	// if it is already set, then we assume that .NET Auto-instrumentation is already configured for this container
 	if getIndexOfEnv(container.Env, envDotNetOTelAutoHome) > -1 {
-		return pod, errors.New("OTEL_DOTNET_AUTO_HOME environment variable is already set in the container")
+		return errors.New("OTEL_DOTNET_AUTO_HOME environment variable is already set in the container")
 	}
 
 	// check if OTEL_DOTNET_AUTO_HOME env var is already set in the .NET instrumentation spec
 	// if it is already set, then we assume that .NET Auto-instrumentation is already configured for this container
 	if getIndexOfEnv(dotNetSpec.Env, envDotNetOTelAutoHome) > -1 {
-		return pod, errors.New("OTEL_DOTNET_AUTO_HOME environment variable is already set in the .NET instrumentation spec")
+		return errors.New("OTEL_DOTNET_AUTO_HOME environment variable is already set in the .NET instrumentation spec")
 	}
 	if runtime != "" && runtime != dotNetRuntimeLinuxGlibc && runtime != dotNetRuntimeLinuxMusl {
-		return pod, fmt.Errorf("provided instrumentation.opentelemetry.io/dotnet-runtime annotation value '%s' is not supported", runtime)
+		return fmt.Errorf("provided instrumentation.opentelemetry.io/dotnet-runtime annotation value '%s' is not supported", runtime)
 	}
 
 	// inject .NET instrumentation spec env vars.
@@ -69,11 +69,17 @@ func injectDotNetSDK(dotNetSpec v1alpha1.DotNet, pod corev1.Pod, container *core
 		Name:      volume.Name,
 		MountPath: dotnetInstrMountPath,
 	})
+	return nil
+}
+
+func injectDotNetSDKToPod(dotNetSpec v1alpha1.DotNet, pod corev1.Pod, firstContainerName string, instSpec v1alpha1.InstrumentationSpec) corev1.Pod {
+	volume := instrVolume(dotNetSpec.VolumeClaimTemplate, dotnetVolumeName, dotNetSpec.VolumeSizeLimit)
 
 	// We just inject Volumes and init containers for the first processed container.
 	if isInitContainerMissing(pod, dotnetInitContainerName) {
 		pod.Spec.Volumes = append(pod.Spec.Volumes, volume)
-		pod.Spec.InitContainers = append(pod.Spec.InitContainers, corev1.Container{
+
+		initContainer := corev1.Container{
 			Name:      dotnetInitContainerName,
 			Image:     dotNetSpec.Image,
 			Command:   []string{"cp", "-r", "/autoinstrumentation/.", dotnetInstrMountPath},
@@ -83,9 +89,25 @@ func injectDotNetSDK(dotNetSpec v1alpha1.DotNet, pod corev1.Pod, container *core
 				MountPath: dotnetInstrMountPath,
 			}},
 			ImagePullPolicy: instSpec.ImagePullPolicy,
-		})
+		}
+
+		pod.Spec.InitContainers = insertInitContainer(&pod, initContainer, firstContainerName)
+	}
+	return pod
+}
+
+// injectDotNetSDK injects .NET instrumentation into the specified containers.
+// Containers must point into the provided pod and be ordered with init containers first.
+func injectDotNetSDK(dotNetSpec v1alpha1.DotNet, pod *corev1.Pod, containers []*corev1.Container, runtime string, instSpec v1alpha1.InstrumentationSpec) error {
+	for _, container := range containers {
+		if err := injectDotNetSDKToContainer(dotNetSpec, container, runtime); err != nil {
+			return err
+		}
+	}
+	if len(containers) > 0 {
+		*pod = injectDotNetSDKToPod(dotNetSpec, *pod, containers[0].Name, instSpec)
 	}
-	return pod, nil
+	return nil
 }
 
 func injectDefaultDotNetEnvVars(container *corev1.Container, runtime string) {

internal/instrumentation/dotnet_test.go

@@ -533,20 +533,120 @@ func TestInjectDotNetSDK(t *testing.T) {
 			},
 			err: errors.New("provided instrumentation.opentelemetry.io/dotnet-runtime annotation value 'not-supported' is not supported"),
 		},
+		{
+			name:   "inject into init container",
+			DotNet: v1alpha1.DotNet{Image: "foo/bar:1", Env: []corev1.EnvVar{}},
+			pod: corev1.Pod{
+				Spec: corev1.PodSpec{
+					InitContainers: []corev1.Container{
+						{
+							Name: "my-init",
+						},
+					},
+				},
+			},
+			runtime: "",
+			expected: corev1.Pod{
+				Spec: corev1.PodSpec{
+					Volumes: []corev1.Volume{
+						{
+							Name: "opentelemetry-auto-instrumentation-dotnet",
+							VolumeSource: corev1.VolumeSource{
+								EmptyDir: &corev1.EmptyDirVolumeSource{
+									SizeLimit: &defaultVolumeLimitSize,
+								},
+							},
+						},
+					},
+					InitContainers: []corev1.Container{
+						{
+							Name:    "opentelemetry-auto-instrumentation-dotnet",
+							Image:   "foo/bar:1",
+							Command: []string{"cp", "-r", "/autoinstrumentation/.", "/otel-auto-instrumentation-dotnet"},
+							VolumeMounts: []corev1.VolumeMount{{
+								Name:      "opentelemetry-auto-instrumentation-dotnet",
+								MountPath: "/otel-auto-instrumentation-dotnet",
+							}},
+						},
+						{
+							Name: "my-init",
+							VolumeMounts: []corev1.VolumeMount{
+								{
+									Name:      "opentelemetry-auto-instrumentation-dotnet",
+									MountPath: "/otel-auto-instrumentation-dotnet",
+								},
+							},
+							Env: []corev1.EnvVar{
+								{
+									Name:  envDotNetCoreClrEnableProfiling,
+									Value: dotNetCoreClrEnableProfilingEnabled,
+								},
+								{
+									Name:  envDotNetCoreClrProfiler,
+									Value: dotNetCoreClrProfilerID,
+								},
+								{
+									Name:  envDotNetCoreClrProfilerPath,
+									Value: dotNetCoreClrProfilerGlibcPath,
+								},
+								{
+									Name:  envDotNetStartupHook,
+									Value: dotNetStartupHookPath,
+								},
+								{
+									Name:  envDotNetAdditionalDeps,
+									Value: dotNetAdditionalDepsPath,
+								},
+								{
+									Name:  envDotNetOTelAutoHome,
+									Value: dotNetOTelAutoHomePath,
+								},
+								{
+									Name:  envDotNetSharedStore,
+									Value: dotNetSharedStorePath,
+								},
+							},
+						},
+					},
+				},
+			},
+			err: nil,
+		},
 	}
 
 	injector := sdkInjector{}
 	for _, test := range tests {
 		t.Run(test.name, func(t *testing.T) {
-			pod, err := injectDotNetSDK(test.DotNet, test.pod, &test.pod.Spec.Containers[0], test.runtime, v1alpha1.InstrumentationSpec{})
-			assert.Equal(t, test.err, err)
-			if err == nil {
-				pod = injector.injectDefaultDotNetEnvVarsWrapper(pod, &pod.Spec.Containers[0], test.runtime)
+			pod := test.pod
+
+			// Collect all containers (regular first, then init)
+			var containers []*corev1.Container
+			for i := range pod.Spec.Containers {
+				containers = append(containers, &pod.Spec.Containers[i])
+			}
+			for i := range pod.Spec.InitContainers {
+				containers = append(containers, &pod.Spec.InitContainers[i])
+			}
+
+			err := injectDotNetSDK(test.DotNet, &pod, containers, test.runtime, v1alpha1.InstrumentationSpec{})
+			if err != nil {
 				assert.Equal(t, test.expected, pod)
 				assert.Equal(t, test.err, err)
-			} else {
-				assert.Equal(t, test.expected, pod)
+				return
+			}
+
+			for i := range pod.Spec.Containers {
+				pod = injector.injectDefaultDotNetEnvVarsWrapper(pod, &pod.Spec.Containers[i], test.runtime)
 			}
+			for i := range pod.Spec.InitContainers {
+				// Skip the instrumentation init container we added
+				if pod.Spec.InitContainers[i].Name == dotnetInitContainerName {
+					continue
+				}
+				pod = injector.injectDefaultDotNetEnvVarsWrapper(pod, &pod.Spec.InitContainers[i], test.runtime)
+			}
+			assert.Equal(t, test.expected, pod)
+			assert.Equal(t, test.err, err)
 		})
 	}
 }