Skip to content

Commit 7cae5bb

Browse files
craigshoemakercraigshoemaker
committed
updates
1 parent bf37746 commit 7cae5bb

File tree

1 file changed

+137
-113
lines changed

1 file changed

+137
-113
lines changed

‎articles/container-apps/opentelemetry-agents.md

Lines changed: 137 additions & 113 deletions
Original file line numberDiff line numberDiff line change
@@ -10,57 +10,89 @@ ms.author: cshoe
1010

1111
# Collect and read OpenTelemetry data in Azure Container Apps (preview)
1212

13-
Using an OpenTelemetry (OTel) data agent with your Azure Container Apps environment, you can send observability data in OpenTelemetry format to:
13+
Using an OpenTelemetry data agent with your Azure Container Apps environment, you can choose to send observability data in an OpenTelemetry format by:
1414

15-
- Azure App Insights
16-
- Datadog
17-
- Any OTLP-configured endpoint
15+
- Piping data from an agent into a desired endpoint. Destination options include Azure Monitor Application Insights, Datadog, and any OTLP-configured endpoint.
1816

19-
The following table shows you what type of data you can send to each destination:
17+
- Easily changing destination endpoints without having to reconfigure how they emit data, and without having to manually run an OpenTelemetry agent.
2018

21-
| Destination | Logs | Metrics | Traces |
22-
|---|------|---------|--------|
23-
| [Azure App Insights](/azure/azure-monitor/app/app-insights-overview) | Yes | Yes | Yes |
24-
| [Datadog](https://datadoghq.com/) | No | Yes | Yes |
25-
| [OpenTelemetry](https://opentelemetry.io/) protocol (OTLP) configured endpoint | Yes | Yes | Yes |
19+
This article shows you how to set up and configure an OpenTelemetry agent for your container app.
2620

27-
Through simple configuration settings, the Container Apps OTel agent makes it easy for you to:
21+
## Configure an OpenTelemetry agent
2822

29-
- Send data to one or multiple destinations
30-
- Switch collection destinations
23+
OpenTelemetry agents live within your container app environment. You configure agent settings via an ARM template or Bicep calls to the environment, or through the CLI.
3124

32-
This article shows you how to set up and configure an OTel agent for your container app.
33-
34-
## Set up an agent
25+
Each endpoint type (Azure Application Insights, DataDog, and OTLP) has specific configuration requirements.
3526

3627
Setting up an agent is a two step process. The first step is to create an instance of the destination service to accept data from your container app. For instance, if you want to send data to Azure Application Insights, you first need to create an App Insights instance.
3728

3829
The second step is to configure your container app to send data to the destination.
3930

4031
The following examples show how to configure your container app to send telemetry data to different agents.
4132

42-
## Azure App Insights
33+
## Prerequisites
34+
35+
Enabling the managed OpenTelemetry agent to your environment doesn't automatically mean the agent collects data. Agents only send data based on your configuration settings.
36+
37+
### Configure source code
38+
39+
Prepare your application to collect data by installing the [OpenTelemetry SDK](https://opentelemetry.io/ecosystem/integrations/) and follow the OpenTelemetry guidelines to instrument [metrics](https://opentelemetry.io/docs/concepts/signals/logs/), [logs](https://opentelemetry.io/docs/concepts/signals/metrics), or [traces](https://opentelemetry.io/docs/concepts/signals/traces/).
40+
41+
### Initialize endpoints
42+
43+
Before you can send data to a collection destination, you first need to create an instance of the destination service. For example, if you want to send data to Azure Application Insights, you need to create an Application Insights instance ahead of time.
44+
45+
The managed OpenTelemetry agent accepts the following destinations:
46+
47+
- Azure Application Insights
48+
- Datadog
49+
- Any OTLP endpoint (For example: New Relic or Honeycomb)
50+
51+
The following table shows you what type of data you can send to each destination:
52+
53+
| Destination | Logs | Metrics | Traces |
54+
|---|------|---------|--------|
55+
| [Azure App Insights](/azure/azure-monitor/app/app-insights-overview) | Yes | Yes | Yes |
56+
| [Datadog](https://datadoghq.com/) | No | Yes | Yes |
57+
| [OpenTelemetry](https://opentelemetry.io/) protocol (OTLP) configured endpoint | Yes | Yes | Yes |
58+
59+
## Azure Application Insights
4360

4461
The only configuration detail required from Application Insights is the connection string. Once you have the connection string, you can configure the agent via your container app's ARM template or with Azure CLI commands.
4562

4663
# [ARM template](#tab/arm)
4764

65+
Before you deploy this template, replace placeholders surrounded by `<>` with your values.
66+
4867
```json
4968
{
5069
...
5170
"properties": {
5271
"appInsightsConfiguration ": { 
5372
"connectionString": "<YOUR_APP_INSIGHTS_CONNECTION_STRING>"
5473
}
74+
"openTelemetryConfiguration": {
75+
...
76+
"tracesConfiguration":{
77+
"destinations": ["appInsights"]
78+
},
79+
"logsConfiguration": {
80+
"destinations": ["apInsights"]
81+
}
82+
}
5583
}
5684
}
5785
```
5886

5987
# [Azure CLI](#tab/azure-cli)
6088

89+
Before you run this command, replace placeholders surrounded by `<>` with your values.
90+
6191
```azurecli
62-
az containerapp env telemetry app-insights set \
63-
--connection-string <YOUR_APP_INSIGHTS_CONNECTION_STRING>
92+
az containerapp env telemetry app-insights set \
93+
--connection-string <YOUR_APP_INSIGHTS_CONNECTION_STRING> \
94+
--EnableOpenTelemetryTraces true \
95+
--EnableOpenTelemetryLogs true
6496
```
6597

6698
---
@@ -78,97 +110,107 @@ Once you have these configuration details, you can configure the agent via your
78110

79111
# [ARM template](#tab/arm)
80112

113+
Before you deploy this template, replace placeholders surrounded by `<>` with your values.
114+
81115
```json
82116
{
83117
...
84118
"properties": {
85119
...
86120
"openTelemetryConfiguration": {
87121
...
88-
"destinationsConfiguration": {
89-
"dataDogConfiguration": { 
90-
"site": "<YOUR_DATADOG_SUBDOMAIN>.datadoghq.com",
91-
  "key": "<YOUR_DATADOG_KEY>"
122+
"destinationsConfiguration":{
123+
...
124+
"dataDogConfiguration":{
125+
"site": "<YOUR_DATADOG_SUBDOMAIN>",
126+
"key": "<YOUR_DATADOG_KEY>"
92127
}
128+
},
129+
"tracesConfiguration":{
130+
"destinations": ["dataDog"]
131+
},
132+
"metricsConfiguration": {
133+
"destinations": ["dataDog"]
93134
}
94135
}
95136
}
96137
}
97138
```
98139

140+
Before you run this command, replace placeholders surrounded by `<>` with your values.
141+
99142
# [Azure CLI](#tab/azure-cli)
100143

101144
```azurecli
102-
az containerapp env telemetry data-dog set \
145+
az containerapp env telemetry data-dog set \
103146
--site "<YOUR_DATADOG_SUBDOMAIN>.datadoghq.com" \
104-
--key <YOUR_DATADOG_KEY>
147+
--key <YOUR_DATADOG_KEY> \
148+
--EnableOpenTelemetryTraces true \
149+
--EnableOpenTelemetryMetrics true
105150
```
106151

107152
---
108153

109154
## OTLP endpoint
110155

111-
An OpenTelemetry protocol (OTLP) endpoint is a telemetry data destination that consumes OpenTelemery data. You can use existing solutions that support OTLP, or develop your own according to the OpenTelemetry protocol.
112-
113-
While you can set up as many OTLP-configured endpoints as you like, each endpoint must be distinct.
114-
115-
# [ARM template](#tab/arm)
116-
117-
```json
118-
{
119-
...
120-
"openTelemetryConfiguration": {
121-
...
122-
"destinationsConfiguration": {
123-
"otlpConfigurations": [
124-
{
125-
  "name": "<YOUR_CONFIGURATION_NAME>",
126-
    "endpoint": "<YOUR_ENDPOINT_URL>",
127-
"header": "<YOUR_HEADER_VALUE>",
128-
  "insecure": "true"
129-
}
130-
]
131-
}
132-
}
133-
}
134-
```
156+
An OpenTelemetry protocol (OTLP) endpoint is a telemetry data destination that consumes OpenTelemetry data. In your application configuration, you can add multiple OTLP endpoints. The following example adds two endpoints and sends the following data to these endpoints.
135157

136-
| Name | Description |
158+
| Endpoint name | Data sent to endpoint |
137159
|---|---|
138-
| `name` | A name you select to identify your OTLP-configured endpoint. |
139-
| `endpoint` | The URL of the destination that receives collected data. |
140-
| `header` | List of security headers. |
141-
| `insecure` | True/false. If false, include headers to |
160+
| `oltp1` | Metrics and/or traces |
161+
| `oltp2` | Logs and/or traces |
162+
163+
While you can set up as many OTLP-configured endpoints as you like, each endpoint must have a distinct name.
142164

143165
# [Azure CLI](#tab/azure-cli)
144166

145167
```azurecli
146-
az containerapp env telemetry otlp add \
147-
--name <ENDPOINT_NAME> \
148-
--endpoint <ENDPOINT_URL> \
149-
--insecure <IS_INSECURE>
150-
--headers <HEADERS>
168+
az containerap env telemetry otlp add \
169+
--name "otlp1" \
170+
--endpoint "ENDPOINT_URL_1" \
171+
--insecure false \
172+
--headers "api-key-1=key" \
173+
--EnableOpenTelemetryTraces true \
174+
--EnableOpenTelemetryMetrics true
175+
az containerap env telemetry otlp add \
176+
--name "otlp2" \
177+
--endpoint "ENDPOINT_URL_2" \
178+
--insecure true \
179+
--EnableOpenTelemetryTraces true \
180+
--EnableOpenTelemetryLogs true
151181
```
152182

153183
| Name | Description |
154184
|---|---|
155-
| `<ENDPOINT_NAME>` | A name you select to identify your OTLP-configured endpoint. |
156-
| `<ENDPOINT_URL>` | The URL of the destination that receives collected data. |
157-
| `<IS_INSECURE>` | True/false. If false, include headers to |
158-
| `<HEADERS>` | List of security headers. |
185+
| `--name` | A name you select to identify your OTLP-configured endpoint. |
186+
| `--endpoint` | The URL of the destination that receives collected data. |
187+
| `--insecure` | Defaults to `true`. Defines whether to enable client transport security for the exporter's gRPC connection. If set to `false`, the `headers` parameter is required. |
188+
| `--headers` | Space separated values in `key=value` format that provides required security information for an OTLP endpoint. Example: `"api-key=key other-config-value=value"` |
159189

160190
---
161191

162-
## Restrictions
192+
## Configuration options
163193

164-
Keep in mind the following restrictions as you use an OTel agent in your container app:
194+
You can control how an agent behaves based off data type and endpoint-related options.
165195

166-
- You can only set up one Application Insights and Datadog endpoint at a time.
167-
- While you can define more than one OTLP-configured endpoint, each one must be distinct.
196+
### By data type
197+
198+
| Option | Example |
199+
|---|---|
200+
| Each data type (for instance: logs, metrics, and traces), is individually configured. | |
201+
| Enable or disable any data type. | You can choose to send only traces and no other data. |
202+
| Send one data type to multiple endpoints. | You can send logs to both DataDog and an OTLP-configured endpoint. |
203+
| Send different data types to different locations. | You can send traces to an OTLP endpoint and metrics to DataDog. |
204+
| Disable sending all data types. | You can choose to not send any data through the OpenTelemetry agent. |
205+
206+
### By endpoint
207+
208+
- You can only set up one Application Insights and Datadog endpoint each at a time.
209+
- While you can define more than one OTLP-configured endpoint, each one must have a distinct name.
168210

169211
## Configure what data is collected
170212

171-
The OTel agent divides data up into the following categories:
213+
The OpenTelemetry agent divides data up into the following categories:
172214

173215
- Traces
174216
- Metrics
@@ -210,11 +252,11 @@ The following example shows how to use an OTLP endpoint named `customDashboard`.
210252
}
211253
```
212254

213-
## Example OTel configuration
255+
## Example OpenTelemetry configuration
214256

215-
The following example show how you might configure your container app to collect telemetry data using Azure Application Insights, Datadog, and with a custom OTLP agent named `customDashboard`.
257+
The following example ARM template shows how you might configure your container app to collect telemetry data using Azure Application Insights, Datadog, and with a custom OTLP agent named `customDashboard`.
216258

217-
# [ARM template](#tab/arm)
259+
Before you deploy this template, replace placeholders surrounded by `<>` with your values.
218260

219261
```json
220262
{
@@ -224,16 +266,15 @@ The following example show how you might configure your container app to collect
224266
"connectionString": "<APP_INSIGHTS_CONNECTION_STRING>"
225267
},
226268
"openTelemetryConfiguration": {
227-
"includeSystemTelemetry": true,
228269
"destinationsConfiguration": {
229270
"dataDogConfiguration": {
230271
"site": "datadoghq.com",
231-
"key": "<DATADOG_KEY>"
272+
"key": "<YOUR_DATADOG_KEY>"
232273
},
233274
"otlpConfigurations": [
234275
{
235276
"name": "customDashboard",
236-
"endpoint": "<OTLP_ENDPOINT_URI>",
277+
"endpoint": "<OTLP_ENDPOINT_URL>",
237278
"insecure": true
238279
}
239280
]
@@ -261,59 +302,42 @@ The following example show how you might configure your container app to collect
261302
}
262303
```
263304

264-
# [CLI](#tab/azure-cli)
305+
## Environment variables
265306

266-
Use a combination of commands with `az containerapp env telemetry` that match the type of agent you want to enable. The following table lists the agents you can enable.
307+
The OpenTelemetry agent automatically injects a set of environment variables into your application at runtime.
267308

268-
| Command | Description |
269-
|---|---|
270-
| `app-insights` | Application Insights agent |
271-
| `data-dog` | Datadog agent |
272-
| `otlp` | Custom OTLP agent |
309+
The first two environment variables follow standard OpenTelemetry exporter configuration and are used in OTLP standard software development kits. If you explicitly set the environment variable in the container app specification, your value overwrites the automatically injected value.
273310

274-
Once you select an agent, then you can enable the type of data you want to collect. The following table lists the parameters available to enable or disable different categories of data to collect.
311+
Learn about the OTLP exporter configuration see, [OTLP Exporter Configuration](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/).
275312

276-
| Parameter | Value |
313+
| Name | Description |
277314
|---|---|
278-
| `--enable-open-telemetry-traces` | `true` or `false` |
279-
| `--enable-open-telemetry-logs` | `true` or `false` |
280-
| `--enable-open-telemetry-metrics` | `true` or `false` |
281-
282-
For example, if you wanted to start collecting traces and logs with Application Insights you would use the following command.
283-
284-
```azurecli
285-
az containerapp env telemetry app-insights set
286-
  --enable-open-telemetry-traces true
287-
  --enable-open-telemetry-logs true
288-
```
289-
290-
---
291-
292-
## Send data from your app to an OTel agent
293-
294-
To send data to an agent, install the [OTel SDK](https://opentelemetry.io/ecosystem/integrations/) into your application. The OTel agent automatically injects environment variables when your application runs to pick up logs, metrics, or traces produced while using the SDK.
295-
296-
## OTel agent costs
297-
298-
There's no cost for enabling a data agent or adding a data destination.
315+
| `OTEL_EXPORTER_OTLP_ENDPOINT` | A base endpoint URL for any signal type, with an optionally specified port number. This setting is helpful when you’re sending more than one signal to the same endpoint and want one environment variable to control the endpoint. Example: `http://otel.service.k8se-apps:4317/` |
316+
| `OTEL_EXPORTER_OTLP_PROTOCOL` | Specifies the OTLP transport protocol used for all telemetry data. The managed agent only supports `grpc`. Value: `grpc`. |
299317

300-
Costs are applied to the data processed by a destination agent. See the billing terms for the data destination of your choice for cost related details.
318+
The other three environment variables are specific to Azure Container Apps, and are always injected. These variables hold agent’s endpoint URLs for each specific data type (logs, metrics, traces).
301319

302-
For example, if you send data to both Azure App Insights and Datadog, you're responsible for the charges applied by both services.
320+
These variables are only necessary if you're using both the managed OpenTelemetry agent and another OpenTelemetry agent. Using these variables gives you control over how to route data between the different OpenTelemetry agents.
303321

304-
## Frequently asked questions
322+
| Name | Description | Example |
323+
|---|---|---|
324+
| `CONTAINERAPP_OTEL_TRACING_GRPC_ENDPOINT` | Endpoint URL for trace data only. | `http://otel.service.k8se-apps:43178/v1/traces/` |
325+
| `CONTAINERAPP_OTEL_LOGGING_GRPC_ENDPOINT` | Endpoint URL for log data only. | `http://otel.service.k8se-apps:43178/v1/logs/ ` |
326+
| `CONTAINERAPP_OTEL_METRIC_GRPC_ENDPOINT` | Endpoint URL for metric data only. | `http://otel.service.k8se-apps:43178/v1/metrics/` |
305327

306-
### How can I use an OTLP agent with a Dapr Sidecar?
328+
## OpenTelemetry agent costs
307329

308-
You can configure Dapr to send traces to App Insights without an agent, but you can choose to use an OTel agent as an alternative.
330+
Customers are billed for the underlying compute of the agent. For details about billing, see [billing.md].
309331

310-
By default an OTel agent doesn't include system data, but you can include system level messages (including Dapr telemetry) by setting `includeSystemTelemetry` to `true`.
332+
See the destination service for their billing structure and terms. For example, if you send data to both Azure Application Insights and Datadog, you're responsible for the charges applied by both services.
311333

312-
To prevent you from collecting redundant data, make sure to remove `daprAIConectionString` (or set it to an empty string) if you plan to use an OTel agent to send Dapr data.
334+
## Known Limitations
313335

314-
### How granular is collected data?
336+
This feature is in preview, and has the following known limitations:
315337

316-
Data is collected at the environment level. All logs, metrics, and traces generated in an environment are sent through the OTLP agent and to the configured destinations.
338+
- System data, such as system logs or Container Apps standard metrics, isn't available to be sent to the OpenTelemetry agent.
339+
- The Application Insights endpoint doesn't accept metrics.
340+
- The Datadog endpoint doesn't accept logs.
317341

318342
## Next steps
319343

0 commit comments

Comments
 (0)