diff --git a/docs/kubernetes/operator/instrumenting-applications.md b/docs/kubernetes/operator/instrumenting-applications.md index ea75aca7..f44e9f32 100644 --- a/docs/kubernetes/operator/instrumenting-applications.md +++ b/docs/kubernetes/operator/instrumenting-applications.md @@ -143,7 +143,7 @@ For details on instrumenting specific languages, refer to: - [Instrumenting Java](./instrumenting-java.md) - [Instrumenting Python](./instrumenting-python.md) -- [Instrumenting Dotnet](./instrumenting-dotnet.md) + ### Namespace based annotations example diff --git a/docs/kubernetes/operator/instrumenting-python.md b/docs/kubernetes/operator/instrumenting-python.md new file mode 100644 index 00000000..18bd3725 --- /dev/null +++ b/docs/kubernetes/operator/instrumenting-python.md @@ -0,0 +1,163 @@ +# Instrumenting Python applications with EDOT SDKs on Kubernetes + +This document focuses on instrumenting Python applications on Kubernetes, using the OpenTelemetry Operator, Elastic Distribution of OpenTelemetry (EDOT) Collectors, and the [EDOT Python](https://github.com/elastic/elastic-otel-python) SDK. + +- For general knowledge about the EDOT Python SDK, refer to the [getting started guide](https://github.com/elastic/elastic-otel-python/blob/main/docs/get-started.md). + +- For Python auto-instrumentation specifics, refer to [OpenTelemetry Operator Python auto-instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#python). + +- To manually instrument your Python application code (by customizing spans and metrics), refer to [EDOT Python manual instrumentation](https://github.com/elastic/elastic-otel-python/blob/main/docs/manual-instrumentation.md#manually-instrument-your-auto-instrumented-python-application). + +- For general information about instrumenting applications on kubernetes, refer to [instrumenting applications](./instrumenting-applications.md). + +## Supported environments and configuration + +- EDOT Python container image supports `glibc` and `musl` based auto-instrumentation for Python 3.12. + +- `musl` based containers instrumentation requires an [extra annotation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#annotations-python-musl) and operator v0.113.0+. + +- To enable logs auto-instrumentation, refer to [auto-instrument python logs](https://opentelemetry.io/docs/kubernetes/operator/automatic/#auto-instrumenting-python-logs). + +- To disable specific instrumentation libraries, refer to [excluding auto-instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#python-excluding-auto-instrumentation). + +- For a full list of configuration options, refer to [Python specific configuration](https://opentelemetry.io/docs/zero-code/python/configuration/#python-specific-configuration). + +- For Python specific limitations when using the OpenTelemetry operator, refer to [Python-specific topics](https://opentelemetry.io/docs/zero-code/python/operator/#python-specific-topics). + +## Instrument a Python app with EDOT Python SDK on Kubernetes + +In this example, you'll learn how to: + +- Enable auto-instrumentation of a Python application using one of the following supported methods: + - Adding an annotation to the deployment Pods. + - Adding an annotation to the namespace. +- Verify that auto-instrumentation libraries are injected and configured correctly. +- Confirm data is flowing to **Kibana Observability**. + +For this example, we assume the application you're instrumenting is a deployment named `python-app` running in the `python-ns` namespace. + +1. Ensure you have successfully [installed the OpenTelemetry Operator](./README.md), and confirm that the following `Instrumentation` object exists in the system: + +```bash +$ kubectl get instrumentation -n opentelemetry-operator-system +NAME AGE ENDPOINT +elastic-instrumentation 107s http://opentelemetry-kube-stack-daemon-collector.opentelemetry-operator-system.svc.cluster.local:4318 +``` +> [!NOTE] +> If your `Instrumentation` object has a different name or is created in a different namespace, you will have to adapt the annotation value in the next step. + +2. Enable auto-instrumentation of the Python application using one of the following methods: + + - Edit your application workload definition and include the annotation under `spec.template.metadata.annotations`: + + ```yaml + spec: + ... + template: + metadata: + labels: + app: python-app + annotations: + instrumentation.opentelemetry.io/inject-python: opentelemetry-operator-system/elastic-instrumentation + ... + ``` + + - Alternatively, add the annotation at namespace level to apply auto-instrumentation in all Pods of the namespace: + + ```bash + kubectl annotate namespace python-ns instrumentation.opentelemetry.io/inject-python=opentelemetry-operator-system/elastic-instrumentation + ``` + +3. Restart application: + + Once the annotation has been set, restart the application to create new Pods and inject the instrumentation libraries: + + ```bash + kubectl rollout restart deployment python-app -n python + ``` + +4. Verify the [auto-instrumentation resources](./instrumenting-applications.md#how-auto-instrumentation-works) are injected in the Pod: + + Run a `kubectl describe` of one of your application pods and check: + + - There should be an init container named `opentelemetry-auto-instrumentation-python` in the Pod: + + ```bash + $ kubectl describe pod python-app-8d84c47b8-8h5z2 -n python + ... + ... + Init Containers: + opentelemetry-auto-instrumentation-python: + Container ID: containerd://fdc86b3191e34ef5ec872853b14a950d0af1e36b0bc207f3d59bd50dd3caafe9 + Image: docker.elastic.co/observability/elastic-otel-python:0.3.0 + Image ID: docker.elastic.co/observability/elastic-otel-python@sha256:de7b5cce7514a10081a00820a05097931190567ec6e18a384ff7c148bad0695e + Port: + Host Port: + Command: + cp + -r + /autoinstrumentation/. + /otel-auto-instrumentation-python + State: Terminated + Reason: Completed + ... + ``` + + - The main container has new environment variables, including `PYTHONPATH`: + + ```bash + ... + Containers: + python-app: + ... + Environment: + ... + PYTHONPATH: /otel-auto-instrumentation-python/opentelemetry/instrumentation/auto_instrumentation:/otel-auto-instrumentation-python + OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf + OTEL_TRACES_EXPORTER: otlp + OTEL_METRICS_EXPORTER: otlp + OTEL_SERVICE_NAME: python-app + OTEL_EXPORTER_OTLP_ENDPOINT: http://opentelemetry-kube-stack-daemon-collector.opentelemetry-operator-system.svc.cluster.local:4318 + ... + ``` + + - The Pod has an `EmptyDir` volume named `opentelemetry-auto-instrumentation-python` mounted in both the main and the init containers in path `/otel-auto-instrumentation-python`: + + ```bash + Init Containers: + opentelemetry-auto-instrumentation-python: + ... + Mounts: + /otel-auto-instrumentation-python from opentelemetry-auto-instrumentation-python (rw) + Containers: + python-app: + ... + Mounts: + /otel-auto-instrumentation-python from opentelemetry-auto-instrumentation-python (rw) + ... + Volumes: + ... + opentelemetry-auto-instrumentation-python: + Type: EmptyDir (a temporary directory that shares a pod's lifetime) + ``` + + Ensure the environment variable `OTEL_EXPORTER_OTLP_ENDPOINT` points to a valid endpoint and there's network communication between the Pod and the endpoint. + +5. Confirm data is flowing through in **Kibana**: + +5. Confirm data is flowing to **Kibana**: + + - Open **Observability** -> **Applications** -> **Service inventory**, and determine if: + - The application appears in the list of services. + - The application shows transactions and metrics. + - If [python logs instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#auto-instrumenting-python-logs) is enabled, the application logs should appear in the Logs tab. + + - For application container logs, open **Kibana Discover** and filter for your Pods' logs. In the provided example, we could filter for them with either of the following: + - `k8s.deployment.name: "python-app"` (**adapt the query filter to your use case**) + - `k8s.pod.name: python-app*` (**adapt the query filter to your use case**) + + Note that the container logs are not provided by the instrumentation library, but by the DaemonSet collector deployed as part of the [operator installation](./README.md). + +## Troubleshooting + +- Refer to [troubleshoot auto-instrumentation](./troubleshoot-auto-instrumentation.md) for further analysis.