Skip to content

Commit 27cace5

Browse files
eedugoneedugon
authored
K8s auto-instrumentation details and example (Java) (#54)
* instrumenting java added * changes in generic instrumenting-applications.md file
1 parent 41da320 commit 27cace5

File tree

2 files changed

+191
-4
lines changed

2 files changed

+191
-4
lines changed

‎docs/kubernetes/operator/instrumenting-applications.md‎

Lines changed: 37 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,15 @@ This section provides guidance and examples for applications instrumentation in
1212

1313
In Kubernetes environments with the OpenTelemetry Operator, [**automatic (or zero-code) instrumentation**](https://opentelemetry.io/docs/kubernetes/operator/automatic/) simplifies the process by injecting and configuring instrumentation libraries into the targeted Pods.
1414

15-
On the other hand, **manual instrumentation** with OpenTelemetry involves adding specific OpenTelemetry SDKs and APIs directly into your application’s code. This approach provides more granular control over what and how data is captured, allowing you to customize trace spans, metrics, and logging based on your application’s logic.
15+
On the other hand, **manual instrumentation** with OpenTelemetry allows you to customize trace spans, metrics, and logging directly in your application’s code. This approach provides more granular control over what and how data is captured.
1616

1717
## Table of contents
1818

1919
- [Supported languages](#supported-languages)
2020
- [Prerequisites](#prerequisites)
2121
- [Auto-instrumentation basics](#auto-instrumentation-basics)
2222
- [Configuring auto-instrumentation](#configuring-auto-instrumentation)
23+
- [How auto-instrumentation works](#how-auto-instrumentation-works)
2324
- [Advanced configuration](#advanced-configuration)
2425
- [Manual instrumentation](#manual-instrumentation)
2526

@@ -110,9 +111,9 @@ where ``<LANGUAGE>`` is one of: `go` , `java`, `nodejs`, `python`, `dotnet`
110111

111112
> [!NOTE]
112113
> Ensure you add the annotations at Pod level and not directly at the workload `spec` level (Deployment, Job, etc.).
113-
> Ensure the annotation value must points to an existing `Instrumentation` object.
114+
> Ensure the annotation value points to an existing `Instrumentation` object.
114115

115-
Alternatively, you can enable auto-instrumentation by adding the annotation at the **namespace level**. This approach automatically applies instrumentation to all Pods within the specified namespace.
116+
Alternatively, you can enable auto-instrumentation by adding the annotation at **namespace level**. This approach automatically applies instrumentation to all Pods within the specified namespace.
116117

117118
```yaml
118119
apiVersion: v1
@@ -133,11 +134,17 @@ In case you have multiple Instrumentation objects with different settings or ima
133134

134135
The possible values for the annotation are detailed in the [Operator documentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#add-annotations-to-existing-deployments). For reference purposes, the values are:
135136

136-
- `"true"`: to inject Instrumentation resource with default name from the current namespace.
137+
- `"true"`: to inject Instrumentation instance with `default` name from the current namespace.
137138
- `"my-instrumentation"`: to inject Instrumentation instance with name `"my-instrumentation"` in the current namespace.
138139
- `"my-other-namespace/my-instrumentation"`: to inject Instrumentation instance with name `"my-instrumentation"` from another namespace `"my-other-namespace"`.
139140
- `"false"`: do not inject.
140141

142+
For details on instrumenting specific languages, refer to:
143+
144+
- [Instrumenting Java](./instrumenting-java.md)
145+
- [Instrumenting Python](./instrumenting-python.md)
146+
- [Instrumenting Dotnet](./instrumenting-dotnet.md)
147+
141148
### Namespace based annotations example
142149

143150
The following example creates a namespace with an annotation to instrument all Pods of the namespace with `java` libraries.
@@ -152,6 +159,32 @@ kubectl annotate namespace java-apps instrumentation.opentelemetry.io/inject-jav
152159
kubectl run otel-test -n java-apps --env OTEL_INSTRUMENTATION_METHODS_INCLUDE="test.Testing[methodB]" --image docker.elastic.co/demos/apm/k8s-webhook-test
153160
```
154161
162+
## Verify auto-instrumentation
163+
164+
After adding the annotation and restarting the Pods, run `kubectl describe` on your application Pod to verify the SDK has been properly attached.
165+
166+
Ensure that the `init container`, `volume`, and `environment variables` described in [how auto-instrumentation works](#how-auto-instrumentation-works) have been successfully injected into the Pod.
167+
168+
## How auto-instrumentation works
169+
170+
The OpenTelemetry Operator automates the process of instrumenting applications by injecting the necessary libraries and configuration into the application Pods.
171+
The process may vary slightly depending on the language, but it generally involves the following steps:
172+
173+
- **Adding an init container**:
174+
175+
The operator adds an init container into the Pod. This container is responsible for copying the OpenTelemetry instrumentation library and make it accessible to the main application container.
176+
177+
- **Creating a shared volume**:
178+
179+
The operator creates an `emptyDir` shared volume within the Pod, and mounts it in both containers. This volume serves as the medium for sharing the instrumentation library between the init container and the application container.
180+
181+
- **Configuring the main container**:
182+
183+
The operator injects environment variables into the main application container to configure OpenTelemetry settings (for example, `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_TRACES_SAMPLER`). Additionally, it links the instrumentation library to the application using mechanisms specific to the language runtime, such as:
184+
- **For Java**: The library is linked through the `javaagent` option using the JAVA_TOOL_OPTIONS environment variable.
185+
- **For Node.js**: The library is linked through the `NODE_OPTIONS` environment variable.
186+
- **For Python**: The operator uses the `PYTHONPATH` environment variable to load the library [sitecustomize](https://docs.python.org/es/dev/library/site.html#module-sitecustomize) module.
187+
155188
## Advanced configuration
156189
157190
You can apply OTEL specific configuration to your applications at two different levels:

‎docs/kubernetes/operator/instrumenting-java.md‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# Instrumenting Java applications with EDOT SDKs on Kubernetes
2+
3+
This document focuses on instrumenting Java applications on Kubernetes, using the OpenTelemetry Operator, Elastic Distribution of OpenTelemetry (EDOT) Collectors, and the [EDOT Java](https://github.com/elastic/elastic-otel-java) SDK.
4+
5+
- For general knowledge about the EDOT Java SDK, refer to the [getting started guide](https://github.com/elastic/elastic-otel-java/blob/main/docs/get-started.md).
6+
- For Java auto-instrumentation specifics, refer to [OpenTelemetry Operator Java auto-instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#java).
7+
- For general information about instrumenting applications on kubernetes, refer to [instrumenting applications](./instrumenting-applications.md).
8+
9+
## Java agent extensions consideration
10+
11+
The operator supports a configuration that installs [Java agent extensions](https://opentelemetry.io/docs/zero-code/java/agent/extensions/) in `Instrumentation` objects. The extension needs to be available in an image. Refer to [using extensions with the OpenTelemetry Java agent](https://www.elastic.co/observability-labs/blog/using-the-otel-operator-for-injecting-elastic-agents#using-an-extension-with-the-opentelemetry-java-agent) for an example of adding an extension to an agent.
12+
13+
## Instrument a Java app with EDOT Java SDK on Kubernetes
14+
15+
In this example, you'll learn how to:
16+
17+
- Enable auto-instrumentation of a Java application using one of the following supported methods:
18+
- Adding an annotation to the deployment Pods.
19+
- Adding an annotation to the namespace.
20+
- Verify that auto-instrumentation libraries are injected and configured correctly.
21+
- Confirm data is flowing to **Kibana Observability**.
22+
23+
For this example, we assume the application you're instrumenting is a deployment named `java-app` running in the `java-ns` namespace.
24+
25+
1. Ensure you have successfully [installed the OpenTelemetry Operator](./README.md), and confirm that the following `Instrumentation` object exists in the system:
26+
27+
```bash
28+
$ kubectl get instrumentation -n opentelemetry-operator-system
29+
NAME AGE ENDPOINT
30+
elastic-instrumentation 107s http://opentelemetry-kube-stack-daemon-collector.opentelemetry-operator-system.svc.cluster.local:4318
31+
```
32+
> [!NOTE]
33+
> If your `Instrumentation` object has a different name or is created in a different namespace, you will have to adapt the annotation value in the next step.
34+
35+
2. Enable auto-instrumentation of your Java application using one of the following methods:
36+
37+
- Edit your application workload definition and include the annotation under `spec.template.metadata.annotations`:
38+
39+
```yaml
40+
kind: Deployment
41+
metadata:
42+
name: java-app
43+
namespace: java-ns
44+
spec:
45+
...
46+
template:
47+
metadata:
48+
...
49+
annotations:
50+
instrumentation.opentelemetry.io/inject-java: opentelemetry-operator-system/elastic-instrumentation
51+
...
52+
```
53+
54+
- Alternatively, add the annotation at namespace level to apply auto-instrumentation in all Pods of the namespace:
55+
56+
```bash
57+
kubectl annotate namespace java-ns instrumentation.opentelemetry.io/inject-java=opentelemetry-operator-system/elastic-instrumentation
58+
```
59+
60+
3. Restart application
61+
62+
Once the annotation has been set, restart the application to create new Pods and inject the instrumentation libraries:
63+
64+
```bash
65+
kubectl rollout restart deployment java-app -n java
66+
```
67+
68+
4. Verify the [auto-instrumentation resources](./instrumenting-applications.md#how-auto-instrumentation-works) are injected in the Pod:
69+
70+
Run a `kubectl describe` of one of your application Pods and check:
71+
72+
- There should be an init container named `opentelemetry-auto-instrumentation-java` in the Pod:
73+
74+
```bash
75+
$ kubectl describe pod java-app-8d84c47b8-8h5z2 -n java
76+
Name: java-app-8d84c47b8-8h5z2
77+
Namespace: java-ns
78+
...
79+
...
80+
Init Containers:
81+
opentelemetry-auto-instrumentation-java:
82+
Container ID: containerd://cbf67d7ca1bd62c25614b905a11e81405bed6fd215f2df21f84b90fd0279230b
83+
Image: docker.elastic.co/observability/elastic-otel-javaagent:1.0.0
84+
Image ID: docker.elastic.co/observability/elastic-otel-javaagent@sha256:28d65d04a329c8d5545ed579d6c17f0d74800b7b1c5875e75e0efd29e210566a
85+
Port: <none>
86+
Host Port: <none>
87+
Command:
88+
cp
89+
/javaagent.jar
90+
/otel-auto-instrumentation-java/javaagent.jar
91+
State: Terminated
92+
Reason: Completed
93+
Exit Code: 0
94+
Started: Wed, 13 Nov 2024 15:47:02 +0100
95+
Finished: Wed, 13 Nov 2024 15:47:03 +0100
96+
Ready: True
97+
Restart Count: 0
98+
Environment: <none>
99+
Mounts:
100+
/otel-auto-instrumentation-java from opentelemetry-auto-instrumentation-java (rw)
101+
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-swhn5 (ro)
102+
```
103+
104+
- The main container of the deployment is using the SDK as `javaagent`:
105+
106+
```bash
107+
...
108+
Containers:
109+
java-app:
110+
Environment:
111+
...
112+
JAVA_TOOL_OPTIONS: -javaagent:/otel-auto-instrumentation-java/javaagent.jar
113+
OTEL_SERVICE_NAME: java-app
114+
OTEL_EXPORTER_OTLP_ENDPOINT: http://opentelemetry-kube-stack-daemon-collector.opentelemetry-operator-system.svc.cluster.local:4318
115+
...
116+
```
117+
118+
- The Pod has an `EmptyDir` volume named `opentelemetry-auto-instrumentation-java` mounted in both the main and the init containers in path `/otel-auto-instrumentation-java`.
119+
120+
```bash
121+
Init Containers:
122+
opentelemetry-auto-instrumentation-java:
123+
...
124+
Mounts:
125+
/otel-auto-instrumentation-java from opentelemetry-auto-instrumentation-java (rw)
126+
Containers:
127+
java-app:
128+
...
129+
Mounts:
130+
/otel-auto-instrumentation-java from opentelemetry-auto-instrumentation-java (rw)
131+
...
132+
Volumes:
133+
...
134+
opentelemetry-auto-instrumentation-java:
135+
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
136+
```
137+
138+
Ensure the environment variable `OTEL_EXPORTER_OTLP_ENDPOINT` points to a valid endpoint and there's network communication between the Pod and the endpoint.
139+
140+
5. Confirm data is flowing to **Kibana**:
141+
142+
- Open **Observability** -> **Applications** -> **Service inventory**, and determine if:
143+
- The application appears in the list of services.
144+
- The application shows transactions and metrics.
145+
146+
- For application container logs, open **Kibana Discover** and filter for your Pods' logs. In the provided example, we could filter for them with either of the following:
147+
- `k8s.deployment.name: "java-app"` (**adapt the query filter to your use case**)
148+
- `k8s.pod.name: java-app*` (**adapt the query filter to your use case**)
149+
150+
Note that the container logs are not provided by the instrumentation library, but by the DaemonSet collector deployed as part of the [operator installation](./README.md).
151+
152+
## Troubleshooting
153+
154+
- Refer to [troubleshoot auto-instrumentation](./troubleshoot-auto-instrumentation.md) for further analysis.

0 commit comments

Comments
 (0)