diff --git a/content/nginx/admin-guide/dynamic-modules/opentelemetry.md b/content/nginx/admin-guide/dynamic-modules/opentelemetry.md index c6a4530ca..ef9b4f0d6 100644 --- a/content/nginx/admin-guide/dynamic-modules/opentelemetry.md +++ b/content/nginx/admin-guide/dynamic-modules/opentelemetry.md @@ -9,246 +9,230 @@ type: - how-to --- - -## Module Overview +## Overview -The module provides [OpenTelemetry](https://opentelemetry.io/) distributed tracing support. The module supports [W3C](https://w3c.github.io/trace-context/) context propagation and OTLP/gRPC export protocol. +[OpenTelemetry](https://opentelemetry.io/) (OTel) is an observability framework for monitoring, tracing, troubleshooting, and optimizing applications. OTel enables the collection of telemetry data from a deployed application stack. - {{< note >}} the code of NGINX OpenTelemetry module is open source since [NGINX Open Source](https://nginx.org) 1.25.2 and NGINX Plus [Release 30]({{< ref "nginx/releases.md#r30" >}}). The source code is available on [GitHub](https://github.com/nginxinc/nginx-otel).{{< /note >}} +The `nginx-plus-module-otel` module is an NGINX-authored dynamic module that enables NGINX Plus to send telemetry data to an OTel collector. The module supports [W3C](https://w3c.github.io/trace-context/) trace context propagation, OpenTelemetry Protocol (OTLP)/gRPC trace exports, and offers several advantages over existing OTel modules including: +- Enhanced performance: with the module enabled, request processing overhead is limited to 10-15%, compared to other OpenTelemetry implementations, which can introduce performance degradation of up to 50%. - -## Installation - -1. Check the [Technical Specifications]({{< ref "nginx/technical-specs.md" >}}) page to verify that the module is supported by your operating system. - -2. Install the OpenTelemetry module package `nginx-plus-module-otel`. - - For CentOS, Oracle Linux, and RHEL: - - ```shell - yum install nginx-plus-module-otel - ``` - - For Amazon Linux 2023, AlmaLinux, Rocky Linux: +- Simplified provisioning through NGINX configuration file. - ```shell - dnf install nginx-plus-module-otel - ``` +- Dynamic, variable-based control of trace parameters with cookies, tokens, and variables. See [Ratio-based Tracing](#example) example for details. - For Debian and Ubuntu: +- Dynamic control of sampling parameters via the [NGINX Plus API]({{< ref "/nginx/admin-guide/monitoring/live-activity-monitoring.md#using-the-rest-api" >}}) and [key-value storage]({{< ref "/nginx/admin-guide/security-controls/denylisting-ip-addresses.md" >}}). - ```shell - apt-get install nginx-plus-module-otel - ``` +The source code for the module is available in the official [GitHub repository](https://github.com/nginxinc/nginx-otel). The official documentation, including module reference and usage examples, is available on the [nginx.org](https://nginx.org/en/docs/ngx_otel_module.html) website. - For SLES: +The OpenTelemetry module supersedes the deprecated [OpenTracing]({{< ref "opentracing.md" >}}) module which was available until NGINX Plus [Release 34]({{< ref "nginx/releases.md#r34" >}}). - ```shell - zypper install nginx-plus-module-otel - ``` - For Alpine: +## Installation - ```shell - apk add nginx-plus-module-otel - ``` +The installation process closely follows the [NGINX Plus installation procedure]({{< ref "/nginx/admin-guide/installing-nginx/installing-nginx-plus.md" >}}). Prebuilt packages of the module for various Linux distributions can can be installed directly from the official repository. Prior to installation, you need to add the NGINX Plus package repository for your distribution and update the repository metadata. - For FreeBSD: +1. Check the [Technical Specifications]({{< ref "nginx/technical-specs.md" >}}) page to verify that the module is supported by your operating system. - ```shell - pkg install nginx-plus-module-otel - ``` + {{< note >}} The OpenTelemetry module cannot be installed on Amazon Linux 2 LTS and SLES 15 SP5+. {{< /note >}} - {{< note >}} the OpenTelemetry module cannot be installed on RHEL/Oracle Linux/AlmaLinux/Rocky Linux 7, Ubuntu 18.04, and Amazon Linux 2. {{< /note >}} +2. Make sure you have the latest version of NGINX Plus. In Terminal, run the command: + ```shell + nginx -v + ``` - + Expected output of the command: -## Configuration + ```shell + nginx version: nginx/1.27.4 (nginx-plus-r34) + ``` -After installation you will need to enable and configure the module in NGINX Plus configuration file `nginx.conf`. +3. Ensure you have the **nginx-repo.crt** and **nginx-repo.key** files from [MyF5 Customer Portal](https://account.f5.com/myf5) in the **/etc/ssl/nginx/** directory. These files are required for accessing the NGINX Plus repository. -1. Enable dynamic loading of the module with the [`load_module`](https://nginx.org/en/docs/ngx_core_module.html#load_module) directive specified in the top-level (“`main`”) context: + ```shell + sudo cp .crt /etc/ssl/nginx/nginx-repo.crt && \ + sudo cp .key /etc/ssl/nginx/nginx-repo.key + ``` - ```nginx - load_module modules/ngx_otel_module.so; - ``` + For Alpine, the **nginx-repo.crt** to **/etc/apk/cert.pem** and **nginx-repo.key** files should be added to **/etc/apk/cert.key**. Ensure these files contain only the specific key and certificate as Alpine Linux does not support mixing client certificates for multiple repositories. -2. Test the configuration and reload NGINX Plus to enable the module: + For FreeBSD, the path to these files should also be added to the `/usr/local/etc/pkg.conf` file: - ```shell - nginx -t && nginx -s reload - ``` + ```shell + PKG_ENV: { SSL_NO_VERIFY_PEER: "1", + SSL_CLIENT_CERT_FILE: "/etc/ssl/nginx/nginx-repo.crt", + SSL_CLIENT_KEY_FILE: "/etc/ssl/nginx/nginx-repo.key" } + ``` +4. Ensure that all required dependencies for your operating system are installed. - -## Module directives + For Amazon Linux 2023, AlmaLinux, CentOS, Oracle Linux, RHEL, and Rocky Linux: - -### `otel_exporter` + ```shell + sudo dnf update && \ + sudo dnf install ca-certificates + ``` -**Syntax:** `otel_exporter { ... }`; + For Debian: -**Default:** — + ```shell + sudo apt update && \ + sudo apt install apt-transport-https \ + lsb-release \ + ca-certificates \ + wget \ + gnupg2 \ + debian-archive-keyring + ``` -**Context:** [`http`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) + For Ubuntu: -Specifies OTel data export parameters: + ```shell + sudo apt update && \ + sudo apt install apt-transport-https \ + lsb-release \ + ca-certificates \ + wget \ + gnupg2 \ + ubuntu-keyring + ``` -- `endpoint` — the address of OTLP/gRPC endpoint that will accept telemetry data. -- `interval` — the maximum interval between two exports, by default is 5 seconds. -- `batch_size` — the maximum number of spans to be sent in one batch per worker, by default is `512`. -- `batch_count` — the number of pending batches per worker, spans exceeding the limit are dropped, by default is `4`. + For FreeBSD: -**Example:** + ```shell + sudo pkg update && \ + sudo pkg install ca_root_nss + ``` -```nginx -otel_exporter { - endpoint localhost:4317; - interval 5s; - batch_size 512; - batch_count 4; -} -``` +5. Ensure that the NGINX signing key has been added, if required by your operating system. -
+ For Debian: - -### `otel_service_name` + ```shell + wget -qO - https://cs.nginx.com/static/keys/nginx_signing.key \ + | gpg --dearmor \ + | sudo tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null + ``` -**Syntax:** `otel_service_name` name; + For Ubuntu: -**Default:** `otel_service_name` unknown_service:nginx; + ```shell + printf "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \ + https://pkgs.nginx.com/plus/ubuntu `lsb_release -cs` nginx-plus\n" \ + | sudo tee /etc/apt/sources.list.d/nginx-plus.list + ``` -**Context:** [`http`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http) + For Alpine: -Sets the [“service.name”](https://opentelemetry.io/docs/specs/otel/resource/semantic_conventions/#service) attribute of the OTel resource. -
-
+ ```shell + sudo wget -O /etc/apk/keys/nginx_signing.rsa.pub https://cs.nginx.com/static/keys/nginx_signing.rsa.pub + ``` - -### `otel_trace` +6. Ensure that your package management system is configured to pull packages from the NGINX Plus repository. See [Installing NGINX Plus]({{< ref "/nginx/admin-guide/installing-nginx/installing-nginx-plus.md" >}}) for details. -**Syntax:** `otel_trace` on | off | $variable; +7. Update the repository information and install the package. In a terminal, run the appropriate command for your operating system. -**Default:** `otel_trace` off; + For CentOS, Oracle Linux, and RHEL: -**Context:** [`http`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http), [`server`](https://nginx.org/en/docs/http/ngx_http_core_module.html#server), [`location`](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) + ```shell + sudo yum update && \ + sudo yum install nginx-plus-module-otel + ``` -Enables or disables OpenTelemetry tracing. The directive can also be enabled by specifying a variable. + For Amazon Linux 2023, AlmaLinux, Rocky Linux: -**Example:** + ```shell + sudo dnf update && \ + sudo dnf install nginx-plus-module-otel + ``` -```nginx -split_clients "$otel_trace_id" $ratio_sampler { - 10% on; - * off; -} + For Debian and Ubuntu: -server { - location / { - otel_trace $ratio_sampler; - otel_trace_context inject; - proxy_pass http://backend; - } -} -``` + ```shell + sudo apt update && \ + sudo apt install nginx-plus-module-otel + ``` -
+ For Alpine: - -### `otel_trace_context` + ```shell + sudo apk update && \ + sudo apk add nginx-plus-module-otel + ``` -**Syntax:** `otel_trace_context` extract | inject | propagate | ignore; + For FreeBSD: -**Default:** `otel_trace_context` ignore; + ```shell + sudo pkg update && \ + sudo pkg install nginx-plus-module-otel + ``` -**Context:** [`http`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http), [`server`](https://nginx.org/en/docs/http/ngx_http_core_module.html#server), [`location`](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) + The resulting `ngx_otel_module.so` dynamic module will be written to the following directory, depending on your operating system: -Specifies how to propagate [traceparent/tracestate](https://www.w3.org/TR/trace-context/#design-overview) headers: + - `/usr/local/nginx/modules` for most Linux Distributions + - `/usr/lib/nginx/modules` for Ubuntu + - `/usr/local/etc/nginx/modules` for FreeBSD -- `extract` — uses an existing trace context from the request, so that the identifiers of a [trace](#var_otel_trace_id) and the [parent span](#var_otel_parent_id) are inherited from the incoming request. -- `inject` — adds a new context to the request, overwriting existing headers, if any. -- `propagate` — updates the existing context (combines `extract` and `inject`). -- `ignore` — skips context headers processing. -
+8. Enable dynamic loading of the module. -
+ - In a text editor, open the NGINX Plus configuration file (`/etc/nginx/nginx.conf` for Linux or `/usr/local/etc/nginx/nginx.conf` for FreeBSD). - -### `otel_span_name` + - On the top-level (or “`main`”) context, specify the path to the dynamic module with the [`load_module`](https://nginx.org/en/docs/ngx_core_module.html#load_module) directive: -**Syntax:** `otel_span_name` name; + ```nginx + load_module modules/ngx_otel_module.so; -**Default:** — + http { + #... + } + ``` + - Save the configuration file. -**Context:** [`http`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http), [`server`](https://nginx.org/en/docs/http/ngx_http_core_module.html#server), [`location`](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) +9. Test the NGINX Plus configuration. In a terminal, type-in the command: -Defines the name of the OTel [span](https://opentelemetry.io/docs/concepts/observability-primer/#spans). By default, it is a name of the location for a request. The name can contain variables. -
-
+ ```shell + nginx -t + ``` - -### `otel_span_attr` + Expected output of the command: -**Syntax:** `otel_span_attr` name value; + ```shell + nginx: the configuration file /etc/nginx/nginx.conf syntax is ok + nginx: configuration file /etc/nginx/nginx.conf is successful + ``` -**Default:** — +10. Reload the NGINX Plus configuration to enable the module: -**Context:** [`http`](https://nginx.org/en/docs/http/ngx_http_core_module.html#http), [`server`](https://nginx.org/en/docs/http/ngx_http_core_module.html#server), [`location`](https://nginx.org/en/docs/http/ngx_http_core_module.html#location) + ```shell + nginx -s reload + ``` -Adds a custom OTel span attribute. The value can contain variables. -
+## Configuration - -## Default span attributes + In a text editor, open the NGINX Plus configuration file (`/etc/nginx/nginx.conf` for Linux or `/usr/local/etc/nginx/nginx.conf` for FreeBSD). -The following [span attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/http.md) are added automatically: +For a complete list of directives, embedded variables, default span attributes, refer to the `ngx_otel_module` official documentation. -- `http.method` -- `http.target` -- `http.route` -- `http.scheme` -- `http.flavor` -- `http.user_agent` -- `http.request_content_length` -- `http.response_content_length` -- `http.status_code` -- `net.host.name` -- `net.host.port` -- `net.sock.peer.addr` -- `net.sock.peer.port` +List of directives: +[`https://nginx.org/en/docs/ngx_otel_module.html#directives`](https://nginx.org/en/docs/ngx_otel_module.html#directives) - -## Module variables +List of variables: - -### `$otel_trace_id` -the identifier of the trace the current span belongs to, for example, `56552bc4daa3bf39c08362527e1dd6c4` +[`https://nginx.org/en/docs/ngx_otel_module.html#variables`](https://nginx.org/en/docs/ngx_otel_module.html#variables) - -### `$otel_span_id` -the identifier of the current span, for example, `4c0b8531ec38ca59` +Default span attributes: - -### `$otel_parent_id` -the identifier of the parent span, for example, `dc94d281b0f884ea` +[`https://nginx.org/en/docs/ngx_otel_module.html#span`](https://nginx.org/en/docs/ngx_otel_module.html#span) - -### `$otel_parent_sampled` -the `sampled` flag of the parent span, can be `1` or `0` -
-
- ## Usage examples ### Simple Tracing -Dumping all the requests could be useful even in non-distributed environment. +This configuration enables basic request tracing, capturing tracing information for every incoming request, even in non-distributed environments. ```nginx http { @@ -263,6 +247,8 @@ http { ### Parent-based Tracing +This configuration enables parent-based tracing, where NGINX Plus captures and propagates trace information from incoming requests, allowing tracing contexts to be inherited from the parent request. It is useful in scenarios where NGINX Plus is used as a reverse proxy within a distributed tracing system. + ```nginx http { server { @@ -278,6 +264,8 @@ http { ### Ratio-based Tracing +This configuration enables sampling of a specified percentage of requests or user sessions for tracing, based on configurable ratios. + ```nginx http { # trace 10% of requests @@ -303,11 +291,14 @@ http { } ``` - -## More Info +## More info -- [NGINX OpenTelemetry module on GitHub](https://github.com/nginxinc/nginx-otel) +- [GitHub Repository for the NGINX Native OpenTelemetry Module](https://github.com/nginxinc/nginx-otel) -- [NGINX Dynamic Modules]({{< ref "dynamic-modules.md" >}}) +- [Official Documentation for the NGINX Native OpenTelemetry Module](https://nginx.org/en/docs/ngx_otel_module.html) - [NGINX Plus Technical Specifications]({{< ref "nginx/technical-specs.md" >}}) + +- [NGINX Dynamic Modules]({{< ref "dynamic-modules.md" >}}) + +- [Uninstalling a Dynamic Module]({{< ref "uninstall.md" >}})