|
| 1 | +--- |
| 2 | +title: Send Azure Event Hubs Logs to Observability Pipelines |
| 3 | +disable_toc: false |
| 4 | +--- |
| 5 | + |
| 6 | +## Overview |
| 7 | + |
| 8 | +This document walks through how to send Azure Event Hubs logs to Observability Pipelines using the Kafka source. The setup steps include setting up Azure Event Hubs for the Kafka source: |
| 9 | + |
| 10 | +- [Create an Event Hubs namespace](#create-an-azure-event-hubs-namespace) |
| 11 | +- [Create an Event Hub (Kafka topic)](#create-an-event-hub-kafka-topic) |
| 12 | +- [Configure shared access policy](#configure-shared-access-policy) |
| 13 | +- [Set up diagnostic settings](#set-up-diagnostic-settings) |
| 14 | +- [Configure Kafka-compatible connection for the event hub](#configure-kafka-compatible-connection-for-the-event-hub) |
| 15 | + |
| 16 | +After Azure Event Hubs has been set up, you [set up a pipeline with the Kafka source](#set-up-a-pipeline-with-the-kafka-source) to send Azure Event Hubs logs to Observability Pipelines. |
| 17 | + |
| 18 | +## Set up Azure Event Hubs for the Kafka source |
| 19 | + |
| 20 | +### Create an Azure Event Hubs namespace |
| 21 | + |
| 22 | +1. In the Azure Portal, navigate to [Event Hubs](https://portal.azure.com/#browse/Microsoft.EventHub%2Fnamespaces). |
| 23 | +1. Click **Create**. |
| 24 | +1. Fill in the **Project Details** (subscription, resource group) and **Instance Details** (namespace name, region, select Standard, Premium, or Dedicated tier). |
| 25 | +1. Ensure the region matches your Azure resources (for example, `westus`). |
| 26 | +1. Click **Review + create**. |
| 27 | + |
| 28 | +**Note**: The Kafka endpoint is automatically enabled for standard and higher tiers. |
| 29 | + |
| 30 | +### Create an event hub (Kafka topic) |
| 31 | + |
| 32 | +1. In the namespace you created, select **Event Hubs** and click **+ Event Hub**. |
| 33 | +1. Enter a name (for example, `datadog-topic`) and configure the settings (for example, 4 partitions and a 7-day retention time). |
| 34 | +1. Click **Review + create**. This Event Hub acts as a Kafka topic. |
| 35 | + |
| 36 | +### Configure shared access policy |
| 37 | + |
| 38 | +1. In the Event Hub you created, navigate to **Settings** > **Shared access policies**. |
| 39 | +1. Click **+ Add**. |
| 40 | +1. Enter a policy name (for example, `DatadogKafkaPolicy`). |
| 41 | +1. Select the **Manage** checkbox, which should automatically select the **Send** and **Listen** checkboxes. |
| 42 | +1. Click **Create**. |
| 43 | +1. The **Primary Key** and **Primary connection string** is needed for Kafka authentication when you set up the Observability Pipeline's Kafka source. |
| 44 | + |
| 45 | +### Set up diagnostic settings |
| 46 | + |
| 47 | +1. Configure Azure resources (for example, VMs, App Services) or subscription-level activity logs to stream logs to the Event Hub. |
| 48 | +1. For resources: |
| 49 | + 1. Navigate to the resource and then to **Monitoring** > **Diagnostic settings**. |
| 50 | + 1. Click **+ Add diagnostic setting**. |
| 51 | + 1. Select log categories you want (for example, AuditLogs, SignInLogs for Microsoft Entra ID). |
| 52 | + 1. In **Destination details**: |
| 53 | + 1. Check the **Stream to an event hub** box. |
| 54 | + 1. Select the namespace and Event Hub (`datadog-topic`). |
| 55 | + 1. Click **Save**. |
| 56 | +1. For activity logs: |
| 57 | + 1. Navigate to **Microsoft Entra ID** > **Monitoring** > **Audit logs** > **Export Data Settings**. |
| 58 | + 1. Check the **Stream to the Event Hub** box. |
| 59 | +1. Repeat for each region. Logs must stream to Event Hubs in the same region. |
| 60 | + |
| 61 | +### Configure Kafka-compatible connection for the Event Hub |
| 62 | + |
| 63 | +Azure Event Hubs exposes a Kafka endpoint at `NAMESPACE.servicebus.windows.net:9093`, which Observability Pipelines uses as the Kafka source. |
| 64 | + |
| 65 | +#### Get the Kafka endpoint |
| 66 | + |
| 67 | +1. In the Azure Portal, navigate to your Event Hubs Namespace (for example, `myeventhubns`). |
| 68 | +1. On the **Overview** page, under the **Essentials** section, locate the **Host name** or **Fully Qualified Domain Name (FQDN)**. It is in the format: `<NAMESPACE>.servicebus.windows.net` (for example, `myeventhubns.servicebus.windows.net`). |
| 69 | +1. Append the Kafka port `:9093` to form the Bootstrap Servers value: `<NAMESPACE>.servicebus.windows.net:9093`. |
| 70 | + - For example, if your namespace is `myeventhubns`, the Bootstrap Servers is `myeventhubns.servicebus.windows.net:9093`. |
| 71 | + - You need this information when you set up the Observability Pipelines Kafka source. |
| 72 | + |
| 73 | +#### Set up authentication |
| 74 | + |
| 75 | +1. Azure Event Hubs uses SASL_SSL with the PLAIN mechanism for Kafka authentication. |
| 76 | +1. The connection string is formatted for Observability Pipelines: |
| 77 | + ``` |
| 78 | + Username: $$ConnectionString |
| 79 | + Password: Endpoint=sb://<NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=<PolicyName>;SharedAccessKey=<Key> |
| 80 | + ``` |
| 81 | +
|
| 82 | +## Set up a pipeline with the Kafka source |
| 83 | +
|
| 84 | +Select your platform. |
| 85 | +
|
| 86 | +{{< tabs >}} |
| 87 | +{{% tab "Kubernetes" %}} |
| 88 | +1. Navigate to [Observability Pipelines](https://app.datadoghq.com/observability-pipelines). |
| 89 | +1. Select the Kafka source. |
| 90 | + 1. In the **Group ID** field, specify or create a unique consumer group (for example, `datadog-consumer-group`). |
| 91 | + 1. In the **Topics** field, enter `datadog-topic` or the topic you configured for your Event Hub earlier. |
| 92 | + 1. Toggle the switch to enable SASL authentication. |
| 93 | + 1. In the **Mechanism** dropdown menu, select **PLAIN**. |
| 94 | + 1. Enable TLS. |
| 95 | + 1. Configure your `values.yaml` file to use the certificate that works as part of the container image: |
| 96 | + ``` |
| 97 | + initContainers: |
| 98 | + - name: copy-config |
| 99 | + image: gcr.io/datadoghq/observability-pipelines-worker:latest |
| 100 | + imagePullPolicy: IfNotPresent |
| 101 | + command: ['/bin/sh', '-c', 'mkdir -p /config-volume/observability-pipelines-worker/config/ && cp /etc/ssl/certs/ca-certificates.crt /config-volume/observability-pipelines-worker/config/ca-certificates.crt'] |
| 102 | + volumeMounts: |
| 103 | + - name: config-volume |
| 104 | + mountPath: /config-volume |
| 105 | + extraVolumes: |
| 106 | + - name: config-volume |
| 107 | + emptyDir: {} |
| 108 | + extraVolumeMounts: |
| 109 | + - name: config-volume |
| 110 | + mountPath: /config-volume |
| 111 | + ``` |
| 112 | + **Note**: When install the Worker with the install command you need to add: |
| 113 | + ``` |
| 114 | + --set env[0].name=DD_OP_DATA_DIR,env[0].value='/config-volume/observability-pipelines-worker/' |
| 115 | + ``` |
| 116 | + 1. In the **Certificate path** field, enter `/ca-certificates.crt` if you used the example above. Otherwise, enter the name of your certificate. |
| 117 | + {{< img src="observability_pipelines/sources/kafka_settings.png" alt="The Kafka source settings with example values" style="width:45%;" >}} |
| 118 | +1. Click **Next: Select Destination**. |
| 119 | +1. After you set up your destinations and processors, click **Next: Install**. |
| 120 | +1. Select your platform in the **Choose your installation platform** dropdown menu. |
| 121 | +1. Enter the environment variables for your Kafka source: |
| 122 | + 1. For **Kafka Bootstrap Servers**, enter `<NAMESPACE>.servicebus.windows.net:9093` (for example, `myeventhubns.servicebus.windows.net:9093`). |
| 123 | + 1. For **Kafka SASL Username**, enter `$$$$ConnectionString`. **Note**: You must have `$$$$` in front of `ConnectionString` because `$$$$` ends up being `$$` when transposed into the environment. |
| 124 | + 1. For **Kafka SASL Password**, enter the full connection string. For example, `Endpoint=sb://<NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=<PolicyName>;SharedAccessKey=<Key>`. |
| 125 | + - This is the **Primary Connection String** in your Event Hub instance [shared access policies](#configure-shared-access-policy). |
| 126 | + 1. Enter your Kafka TLS passphrase. |
| 127 | + - This is the **Primary Key** in your Event Hub instance's [shared access policies](#configure-shared-access-policy). |
| 128 | + {{< img src="observability_pipelines/sources/kafka_env_vars.png" alt="The install page with example values for the kafka environment variables" style="width:70%;" >}} |
| 129 | +1. Enter the environment variables for your destinations, if applicable. |
| 130 | +1. Follow the rest of the instructions on the page to install the Worker based on your platform. |
| 131 | +{{% /tab %}} |
| 132 | +{{% tab "Virtual machine (VM)" %}} |
| 133 | +
|
| 134 | +1. Navigate to [Observability Pipelines](https://app.datadoghq.com/observability-pipelines). |
| 135 | +1. Select the Kafka source. |
| 136 | + 1. In the **Group ID** field, specify or create a unique consumer group (for example, `datadog-consumer-group`). |
| 137 | + 1. Enter `datadog-topic` in the **Topics** field. |
| 138 | + 1. Toggle the switch to enable SASL authentication. |
| 139 | + 1. In the **Mechanism** dropdown menu, select **PLAIN**. |
| 140 | + 1. Enable TLS. For the certificate, copy the certificate from its original location to the default Observability Pipelines data configuration directory: |
| 141 | + 1. Since the Observability Pipelines Worker hasn't been installed yet, run this command to create the directory for the certificate: |
| 142 | + ``` |
| 143 | + sudo mkdir -p /var/lib/observability-pipelines-worker/config |
| 144 | + ``` |
| 145 | + 1. Run this command to copy the certificate to the directory you created: |
| 146 | + ``` |
| 147 | + sudo cp /etc/ssl/certs/ca-certificates.crt /var/lib/observability-pipelines-worker/config/ |
| 148 | + ``` |
| 149 | + 1. In the **Certificate path** field, enter `/ca-certificates.crt`. |
| 150 | + {{< img src="observability_pipelines/sources/kafka_settings_vm.png" alt="The Kafka source settings with example values" style="width:45%;" >}} |
| 151 | +1. Click **Next: Select Destination**. |
| 152 | +1. After you set up your destinations and processors, click **Next: Install**. |
| 153 | +1. Select your platform in the **Choose your installation platform** dropdown menu. |
| 154 | +1. Enter the environment variables for your Kafka source: |
| 155 | + 1. For **Kafka Bootstrap Servers**, enter `<NAMESPACE>.servicebus.windows.net:9093` (for example, `myeventhubns.servicebus.windows.net:9093`). |
| 156 | + 1. For **Kafka SASL Username**, enter `\$\$ConnectionString`. **Note**: You must escape the `$` in front of `ConnectionString`, otherwise the environment variable won't be loaded. |
| 157 | + 1. For **Kafka SASL Password**, enter the full connection string wrapped in quotes (`"`). For example, `"Endpoint=sb://<NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=<PolicyName>;SharedAccessKey=<Key>"`. |
| 158 | + - This is the **Primary Connection String** in your Event Hub instance [shared access policies](#configure-shared-access-policy). |
| 159 | + 1. Enter your Kafka TLS passphrase. |
| 160 | + - This is the **Primary Key** in your Event Hub instance's [shared access policies](#configure-shared-access-policy). |
| 161 | + {{< img src="observability_pipelines/sources/kafka_env_vars_vm.png" alt="The install page with example values for the kafka environment variables" style="width:70%;" >}} |
| 162 | +
|
| 163 | +{{% /tab %}} |
| 164 | +{{< /tabs >}} |
| 165 | +
|
| 166 | +## Troubleshooting |
| 167 | +
|
| 168 | +If you run into issues after installing the Worker, check your Observability Pipelines environment file (`/etc/default/observability-pipelines-worker`) to make sure the environment variables are correctly set: |
| 169 | +
|
| 170 | +- `DD_OP_SOURCE_KAFKA_SASL_USERNAME="$$ConnectionString"` |
| 171 | +- `DD_OP_SOURCE_KAFKA_BOOTSTRAP_SERVERS=<NAMESPACE>.servicebus.windows.net:9093` |
| 172 | +- `DD_OP_SOURCE_KAFKA_SASL_PASSWORD=<Endpoint=sb://<NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=<PolicyName>;SharedAccessKey=<Key>>` |
| 173 | +- `DD_OP_SOURCE_KAFKA_KEY_PASS=password` |
| 174 | +
|
| 175 | +### Missing environment variable |
| 176 | +
|
| 177 | +If you see the error `Missing environment variable DD_OP_SOURCE_KAFKA_SASL_PASSWORD` and you are running the Worker in a VM, make sure that the variable is in quotes (`"`) when you run the Worker install script. For example: |
| 178 | +``` |
| 179 | +DD_OP_SOURCE_KAFKA_SASL_PASSWORD=`"Endpoint=sb://<NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=<PolicyName>;SharedAccessKey=<Key>"` |
| 180 | +``` |
0 commit comments