From fd310ed93ff01dcb8405fac57cbf8ce042ab01ac Mon Sep 17 00:00:00 2001 From: Riccardo Magliocchetti Date: Wed, 20 Nov 2024 15:31:04 +0100 Subject: [PATCH 1/9] WIP manual instrumentation --- docs/manual-instrumentation.md | 123 +++++++++++++++++++++++++++++++++ 1 file changed, 123 insertions(+) create mode 100644 docs/manual-instrumentation.md diff --git a/docs/manual-instrumentation.md b/docs/manual-instrumentation.md new file mode 100644 index 0000000..b43825f --- /dev/null +++ b/docs/manual-instrumentation.md @@ -0,0 +1,123 @@ + + +# Manual instrumentation + +This guide shows you how to use the Elastic Distribution of OpenTelemetry Python (EDOT Python) to manually instrument your Python application and send OpenTelemetry data to an Elastic Observability deployment. + +**Already familiar with OpenTelemetry?** It's an explicit goal of this distribution to introduce _no new concepts_ outside those defined by the wider OpenTelemetry community. + +**New to OpenTelemetry?** If your are new to OpenTelemetry we encourage you to take a look at our [get started documentation](./get-started.md) instead that will introduce you to autoinstrumentation. + +**Already added autoinstrumentation with OpenTelemetry to your application?** Skip to the [Manually instrument your Python application chapter](#Manually-instrument-your-Python-application). + + +## Prerequisites + +Before getting started, you'll need somewhere to send the gathered OpenTelemetry data, so it can be viewed and analyzed. EDOT Python supports sending data to any OpenTelemetry protocol (OTLP) endpoint, but this guide assumes you are sending data to an [Elastic Observability](https://www.elastic.co/observability) cloud deployment. You can use an existing one or set up a new one. + +
+Expand for setup instructions + +To create your first Elastic Observability deployment: + +1. Sign up for a [free Elastic Cloud trial](https://cloud.elastic.co/registration) or sign into an existing account. +1. Go to . +1. Click **Create deployment**. +1. When the deployment is ready, click **Open** to visit your Kibana home page (for example, `https://{DEPLOYMENT_NAME}.kb.{REGION}.cloud.es.io/app/home#/getting_started`). +
+ + +## Install + +### Install the distribution + +Install EDOT Python: + +```bash +pip install elastic-opentelemetry +``` + +### Install the available instrumentation + +EDOT Python does not install any instrumentation package by default, instead it relies on the +`opentelemetry-bootstrap` command to scan the installed packages and install the available instrumentation. +The following command will install all the instrumentations available for libraries found installed +in your environment: + +```bash +opentelemetry-bootstrap --action=install +``` + +> [!NOTE] +> Add this command every time you deploy an updated version of your application (in other words, add it to your container image build process). + + +## Send data to Elastic + +After installing EDOT Python, configure and initialize it to start sending data to Elastic. + + +### Configure EDOT Python + +To configure EDOT Python, at a minimum you'll need your Elastic Observability cloud deployment's OTLP endpoint and +authorization data to set a few `OTLP_*` environment variables that will be available when running EDOT Python: + +* `OTEL_RESOURCE_ATTRIBUTES`: Use this to add a service name that will make it easier to recognize your application when reviewing data sent to Elastic. +* `OTEL_EXPORTER_OTLP_ENDPOINT`: The full URL of the endpoint where data will be sent. +* `OTEL_EXPORTER_OTLP_HEADERS`: A comma-separated list of `key=value` pairs that will +be added to the headers of every request. This is typically used for authentication information. + +You can find the values of the endpoint and header variables in Kibana's APM tutorial. In Kibana: + +1. Go to **Setup guides**. +1. Select **Observability**. +1. Select **Monitor my application performance**. +1. Scroll down and select the **OpenTelemetry** option. +1. The appropriate values for `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_HEADERS` are shown there. + +Here's an example: + +```sh +export OTEL_RESOURCE_ATTRIBUTES=service.name= +export OTEL_EXPORTER_OTLP_ENDPOINT=https://my-deployment.apm.us-west1.gcp.cloud.es.io +export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer P....l" +``` + +> [!NOTE] +> Alternatively, you can use an [APM agent key](https://www.elastic.co/guide/en/observability/current/apm-api-key.html) to authorize requests to an Elastic Observability endpoint. APM agent keys are revocable, you can have more than one of them, and you can add or remove them without restarting APM Server. +> +> To create and manage APM Agent keys in Kibana: +> +> 1. Go to **APM Settings**. +> 1. Select the **Agent Keys** tab. +> +> When using an APM Agent key, the `OTEL_EXPORTER_OTLP_HEADERS` is set using different auth schema (`ApiKey` rather than `Bearer`). For example: +> ```sh +> export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey TkpXUkx...dVZGQQ==" +> ``` + + +### Manually instrument your Python application + +FIXME + + +## Confirm that EDOT Python is working + +To confirm that EDOT Python has successfully connected to Elastic: + +1. Go to **APM** → **Traces**. +1. You should see the name of the service to which you just added EDOT Python. It can take several minutes after initializing EDOT Python for the service to show up in this list. +1. Click on the name in the list to see trace data. + +> [!NOTE] +> There may be no trace data to visualize unless you have _used_ your application since initializing EDOT Python. + + +## Next steps + +* Reference all available [configuration options](./configure.md). +* Learn more about viewing and interpreting data in the [Observability guide](https://www.elastic.co/guide/en/observability/current/apm.html). From f76ee09dff609e6f79433718a19d8199b6e1c9f3 Mon Sep 17 00:00:00 2001 From: Riccardo Magliocchetti Date: Wed, 20 Nov 2024 16:50:30 +0100 Subject: [PATCH 2/9] Add manual instrumentation to an automatically instrumented application --- docs/manual-instrumentation.md | 63 ++++++++++++++++++++++++++++++++-- 1 file changed, 61 insertions(+), 2 deletions(-) diff --git a/docs/manual-instrumentation.md b/docs/manual-instrumentation.md index b43825f..83a17b0 100644 --- a/docs/manual-instrumentation.md +++ b/docs/manual-instrumentation.md @@ -100,9 +100,68 @@ export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer P....l" > ``` -### Manually instrument your Python application +### Manually instrument your auto instrumented Python application -FIXME +In this section we'll show how to add manual instrumentation to an already automatically instrumented application. A use case for +this setup would be to trace something in particular while keeping the benefits of the simplicity of the automatic instrumentation doing +the hard work for us. + +As an example we'll use an application using the Flask framework that implements an endpoint mounted on `/hello` returning a friendly +salute. This application is saved in a file named `app.py` that is the default module for Flask applications. + +``` +import random + +from flask import Flask +from opentelemetry import trace + +tracer = trace.get_tracer(__name__) + +app = Flask(__name__) + +@app.route("/hello") +def hello(): + choices = ["there", "world", "folks", "hello"] + with tracer.start_as_current_span("choice") as span: + choice = random.choice(choices) + span.set_attribute("choice.value", choice) + return f"Hello {choice}!" +``` + +And then we can run this application with the following command: + +```bash +opentelemetry-instrument flask run +``` + +We may not only need to add custom span to our application we may also want to use a custom metric, like in the example below where we +are tracking how many times we are getting one of the possible choices for our salutes. + +``` +import random + +from flask import Flask +from opentelemetry import trace + +tracer = trace.get_tracer(__name__) +meter = metrics.get_meter(__name__) + +hello_counter = meter.create_counter( + "hello.choice", + description="The number of times a salute is chosen", +) + +app = Flask(__name__) + +@app.route("/hello") +def hello(): + choices = ["there", "world", "folks", "hello"] + with tracer.start_as_current_span("choice") as span: + choice = random.choice(choices) + span.set_attribute("choice.value", choice) + hello_counter.add(1, {"choice.value": choice}) + return f"Hello {choice}!" +``` ## Confirm that EDOT Python is working From 1f70f9a1d1616dd13354c5649ba55f20e6b84cd0 Mon Sep 17 00:00:00 2001 From: Riccardo Magliocchetti Date: Wed, 20 Nov 2024 16:55:01 +0100 Subject: [PATCH 3/9] Create a span only for the choice --- docs/manual-instrumentation.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/manual-instrumentation.md b/docs/manual-instrumentation.md index 83a17b0..1d81a4b 100644 --- a/docs/manual-instrumentation.md +++ b/docs/manual-instrumentation.md @@ -122,10 +122,11 @@ app = Flask(__name__) @app.route("/hello") def hello(): choices = ["there", "world", "folks", "hello"] + # create a span for the choice of the name, this may be a costly call in your real world application with tracer.start_as_current_span("choice") as span: - choice = random.choice(choices) - span.set_attribute("choice.value", choice) - return f"Hello {choice}!" + choice = random.choice(choices) + span.set_attribute("choice.value", choice) + return f"Hello {choice}!" ``` And then we can run this application with the following command: @@ -156,11 +157,12 @@ app = Flask(__name__) @app.route("/hello") def hello(): choices = ["there", "world", "folks", "hello"] + # create a span for the choice of the name, this may be a costly call in your real world application with tracer.start_as_current_span("choice") as span: - choice = random.choice(choices) - span.set_attribute("choice.value", choice) - hello_counter.add(1, {"choice.value": choice}) - return f"Hello {choice}!" + choice = random.choice(choices) + span.set_attribute("choice.value", choice) + hello_counter.add(1, {"choice.value": choice}) + return f"Hello {choice}!" ``` From c7df23d80e1d7f5e226ffa9267e4bc0d70ca0fc6 Mon Sep 17 00:00:00 2001 From: Riccardo Magliocchetti Date: Fri, 22 Nov 2024 11:42:22 +0100 Subject: [PATCH 4/9] Fix code examples --- docs/manual-instrumentation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/manual-instrumentation.md b/docs/manual-instrumentation.md index 1d81a4b..ccc6b66 100644 --- a/docs/manual-instrumentation.md +++ b/docs/manual-instrumentation.md @@ -125,7 +125,7 @@ def hello(): # create a span for the choice of the name, this may be a costly call in your real world application with tracer.start_as_current_span("choice") as span: choice = random.choice(choices) - span.set_attribute("choice.value", choice) + span.set_attribute("choice.value", choice) return f"Hello {choice}!" ``` @@ -142,7 +142,7 @@ are tracking how many times we are getting one of the possible choices for our s import random from flask import Flask -from opentelemetry import trace +from opentelemetry import metrics, trace tracer = trace.get_tracer(__name__) meter = metrics.get_meter(__name__) @@ -160,7 +160,7 @@ def hello(): # create a span for the choice of the name, this may be a costly call in your real world application with tracer.start_as_current_span("choice") as span: choice = random.choice(choices) - span.set_attribute("choice.value", choice) + span.set_attribute("choice.value", choice) hello_counter.add(1, {"choice.value": choice}) return f"Hello {choice}!" ``` From e29b6d38150a2e7c03f70e921360532d3c3b1ea3 Mon Sep 17 00:00:00 2001 From: Riccardo Magliocchetti Date: Thu, 21 Nov 2024 15:01:45 +0100 Subject: [PATCH 5/9] Update docs/manual-instrumentation.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Edu González de la Herrán <25320357+eedugon@users.noreply.github.com> --- docs/manual-instrumentation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/manual-instrumentation.md b/docs/manual-instrumentation.md index ccc6b66..dc1459a 100644 --- a/docs/manual-instrumentation.md +++ b/docs/manual-instrumentation.md @@ -9,7 +9,7 @@ This guide shows you how to use the Elastic Distribution of OpenTelemetry Python **Already familiar with OpenTelemetry?** It's an explicit goal of this distribution to introduce _no new concepts_ outside those defined by the wider OpenTelemetry community. -**New to OpenTelemetry?** If your are new to OpenTelemetry we encourage you to take a look at our [get started documentation](./get-started.md) instead that will introduce you to autoinstrumentation. +**New to OpenTelemetry?** If your are new to OpenTelemetry we encourage you to take a look at our [get started documentation](./get-started.md) instead, which will introduce you to autoinstrumentation. **Already added autoinstrumentation with OpenTelemetry to your application?** Skip to the [Manually instrument your Python application chapter](#Manually-instrument-your-Python-application). From 4371e11f3e3f161e07f8406f3e872c282d83345d Mon Sep 17 00:00:00 2001 From: Riccardo Magliocchetti Date: Mon, 25 Nov 2024 15:32:36 +0100 Subject: [PATCH 6/9] Consider manual instrumentation as next step for get-started To drop most of duplicated content. Add cross reference between the two documents. --- docs/get-started.md | 1 + docs/manual-instrumentation.md | 86 ++++------------------------------ 2 files changed, 11 insertions(+), 76 deletions(-) diff --git a/docs/get-started.md b/docs/get-started.md index ee387d6..5ba1919 100644 --- a/docs/get-started.md +++ b/docs/get-started.md @@ -129,4 +129,5 @@ To confirm that EDOT Python has successfully connected to Elastic: ## Next steps * Reference all available [configuration options](./configure.md). +* Learn how to add [manual instrumentation](./manual-instrumentation.md). * Learn more about viewing and interpreting data in the [Observability guide](https://www.elastic.co/guide/en/observability/current/apm.html). diff --git a/docs/manual-instrumentation.md b/docs/manual-instrumentation.md index dc1459a..8c4b246 100644 --- a/docs/manual-instrumentation.md +++ b/docs/manual-instrumentation.md @@ -7,97 +7,24 @@ The user is able to manually instrument their Python application This guide shows you how to use the Elastic Distribution of OpenTelemetry Python (EDOT Python) to manually instrument your Python application and send OpenTelemetry data to an Elastic Observability deployment. -**Already familiar with OpenTelemetry?** It's an explicit goal of this distribution to introduce _no new concepts_ outside those defined by the wider OpenTelemetry community. +This guide requires to have already added autoinstrumentation with OpenTelemetry to your application per [Get started](./get-started.md). **New to OpenTelemetry?** If your are new to OpenTelemetry we encourage you to take a look at our [get started documentation](./get-started.md) instead, which will introduce you to autoinstrumentation. -**Already added autoinstrumentation with OpenTelemetry to your application?** Skip to the [Manually instrument your Python application chapter](#Manually-instrument-your-Python-application). - - -## Prerequisites - -Before getting started, you'll need somewhere to send the gathered OpenTelemetry data, so it can be viewed and analyzed. EDOT Python supports sending data to any OpenTelemetry protocol (OTLP) endpoint, but this guide assumes you are sending data to an [Elastic Observability](https://www.elastic.co/observability) cloud deployment. You can use an existing one or set up a new one. - -
-Expand for setup instructions - -To create your first Elastic Observability deployment: - -1. Sign up for a [free Elastic Cloud trial](https://cloud.elastic.co/registration) or sign into an existing account. -1. Go to . -1. Click **Create deployment**. -1. When the deployment is ready, click **Open** to visit your Kibana home page (for example, `https://{DEPLOYMENT_NAME}.kb.{REGION}.cloud.es.io/app/home#/getting_started`). -
- - -## Install - -### Install the distribution - -Install EDOT Python: - -```bash -pip install elastic-opentelemetry -``` - -### Install the available instrumentation - -EDOT Python does not install any instrumentation package by default, instead it relies on the -`opentelemetry-bootstrap` command to scan the installed packages and install the available instrumentation. -The following command will install all the instrumentations available for libraries found installed -in your environment: - -```bash -opentelemetry-bootstrap --action=install -``` - -> [!NOTE] -> Add this command every time you deploy an updated version of your application (in other words, add it to your container image build process). - - -## Send data to Elastic - -After installing EDOT Python, configure and initialize it to start sending data to Elastic. - ### Configure EDOT Python To configure EDOT Python, at a minimum you'll need your Elastic Observability cloud deployment's OTLP endpoint and authorization data to set a few `OTLP_*` environment variables that will be available when running EDOT Python: -* `OTEL_RESOURCE_ATTRIBUTES`: Use this to add a service name that will make it easier to recognize your application when reviewing data sent to Elastic. -* `OTEL_EXPORTER_OTLP_ENDPOINT`: The full URL of the endpoint where data will be sent. -* `OTEL_EXPORTER_OTLP_HEADERS`: A comma-separated list of `key=value` pairs that will -be added to the headers of every request. This is typically used for authentication information. - -You can find the values of the endpoint and header variables in Kibana's APM tutorial. In Kibana: - -1. Go to **Setup guides**. -1. Select **Observability**. -1. Select **Monitor my application performance**. -1. Scroll down and select the **OpenTelemetry** option. -1. The appropriate values for `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_HEADERS` are shown there. - -Here's an example: - ```sh export OTEL_RESOURCE_ATTRIBUTES=service.name= export OTEL_EXPORTER_OTLP_ENDPOINT=https://my-deployment.apm.us-west1.gcp.cloud.es.io export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer P....l" +opentelemetry-instrument ``` -> [!NOTE] -> Alternatively, you can use an [APM agent key](https://www.elastic.co/guide/en/observability/current/apm-api-key.html) to authorize requests to an Elastic Observability endpoint. APM agent keys are revocable, you can have more than one of them, and you can add or remove them without restarting APM Server. -> -> To create and manage APM Agent keys in Kibana: -> -> 1. Go to **APM Settings**. -> 1. Select the **Agent Keys** tab. -> -> When using an APM Agent key, the `OTEL_EXPORTER_OTLP_HEADERS` is set using different auth schema (`ApiKey` rather than `Bearer`). For example: -> ```sh -> export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey TkpXUkx...dVZGQQ==" -> ``` +Refer to our [get started](./get-started.md) page for more details. ### Manually instrument your auto instrumented Python application @@ -129,6 +56,13 @@ def hello(): return f"Hello {choice}!" ``` +We need to make sure to have Flask and the Flask OpenTelemetry instrumentation installed: + +```bash +pip install flask +opentelemetry-bootstrap --action=install +``` + And then we can run this application with the following command: ```bash From 2cb049ca44beb1680546309de3b0631b4d8356e0 Mon Sep 17 00:00:00 2001 From: Riccardo Magliocchetti Date: Mon, 25 Nov 2024 15:34:48 +0100 Subject: [PATCH 7/9] Drop opentelemetry-instrument call from configuration chapter --- docs/manual-instrumentation.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/manual-instrumentation.md b/docs/manual-instrumentation.md index 8c4b246..389a53e 100644 --- a/docs/manual-instrumentation.md +++ b/docs/manual-instrumentation.md @@ -21,7 +21,6 @@ authorization data to set a few `OTLP_*` environment variables that will be avai export OTEL_RESOURCE_ATTRIBUTES=service.name= export OTEL_EXPORTER_OTLP_ENDPOINT=https://my-deployment.apm.us-west1.gcp.cloud.es.io export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer P....l" -opentelemetry-instrument ``` Refer to our [get started](./get-started.md) page for more details. From 915ed3f47e28234a3f5161613a34eea825402109 Mon Sep 17 00:00:00 2001 From: Riccardo Magliocchetti Date: Thu, 28 Nov 2024 09:38:33 +0100 Subject: [PATCH 8/9] Reference manual instrumentation in README --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index c9003b1..5946a53 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,7 @@ With EDOT Python you have access to all the features of the OpenTelemetry Python ## Read the docs * [Get started](./docs/get-started.md) +* [Manual instrumentation](./docs/manual-instrumentation.md) * [Configuration](./docs/configure.md) ## Install From 263d4fca3d9f0ee4e286a37f88633c73430bf19a Mon Sep 17 00:00:00 2001 From: Riccardo Magliocchetti Date: Thu, 28 Nov 2024 15:48:14 +0100 Subject: [PATCH 9/9] Update docs/manual-instrumentation.md Co-authored-by: Emily S --- docs/manual-instrumentation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/manual-instrumentation.md b/docs/manual-instrumentation.md index 389a53e..8af8e4d 100644 --- a/docs/manual-instrumentation.md +++ b/docs/manual-instrumentation.md @@ -68,7 +68,7 @@ And then we can run this application with the following command: opentelemetry-instrument flask run ``` -We may not only need to add custom span to our application we may also want to use a custom metric, like in the example below where we +We may not only need to add a custom span to our application but also want to use a custom metric, like in the example below where we are tracking how many times we are getting one of the possible choices for our salutes. ```