Skip to content

Update instructions with Datadog Agent source and Cloudprem dest #33314

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
estherk15 merged 1 commit into from
Jan 8, 2026
Merged

Update instructions with Datadog Agent source and Cloudprem dest #33314

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 34 additions & 19 deletions content/en/cloudprem/ingest_logs/observability_pipelines.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@
- link: "/observability_pipelines/"
tag: "Documentation"
text: "Observability Pipelines Overview"
- link: "/observability_pipelines/destinations/cloudprem/"
tag: "Documentation"
text: "CloudPrem Destination for Observability Pipelines"
---

{{< callout url="https://www.datadoghq.com/product-preview/cloudprem/" btn_hidden="false" header="CloudPrem is in Preview" >}}
Expand All @@ -19,42 +22,48 @@

## Overview

You can send logs to CloudPrem using Observability Pipelines with the following setup:
- Datadog Agent collects logs and send them to Observability Pipelines HTTP server source.
- Observability Pipelines forwards logs to the HTTP client destination.
Observability Pipelines provides a flexible intermediary layer between your Datadog Agents and CloudPrem, allowing you to process, transform, and route logs before they reach your CloudPrem deployment. Configure Observability Pipelines to receive logs from the Datadog Agent and forward them to CloudPrem:

Check notice on line 25 in content/en/cloudprem/ingest_logs/observability_pipelines.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.sentencelength 

Suggestion: Try to keep your sentence length to 25 words or fewer.
1. [**Create and configure the pipeline**](#create-and-configure-an-observability-pipeline) - Define your pipeline configuration (source, processors, destination) in the Observability Pipelines UI. This creates the pipeline definition that will be used by the Worker.

Check warning on line 26 in content/en/cloudprem/ingest_logs/observability_pipelines.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.tense 

Avoid temporal words like 'will'.
2. [**Deploy the Observability Pipelines Worker**](##deploy-your-observability-pipelines) - Install the Worker with your pipeline configuration. The Worker must be running and listening for logs before the Agent can connect to it.
3. [**Configure the Datadog Agent**](#configure-the-datadog-agent) - Point the Agent to send logs to the deployed Worker. This step must come last because the Agent needs the Worker's address to be available.

## Create and configure an Observability Pipeline

You'll need to create a pipeline that acts as an intermediary for your logs. Navigate to the [Observability Pipelines][1] page and create a pipeline using the **Log Volume Control** template. Configure the pipeline with the following components:
- **HTTP server source**: (No configuration) This is the entry point where the Datadog Agent will send logs. Leave the configuration empty.
- **Empty processor**: Remove all default processors from the pipeline.
- **HTTP client destination**: (No configuration) This forwards the logs to your CloudPrem instance. Leave the configuration empty.

After configuring, your pipeline should look like this and you are ready to deploy:
1. Navigate to [Observability Pipelines][1].
1. Select the [**Log Volume Control template**][2].
1. Set up your pipeline:
1. Choose the [**Datadog Agent** source][3].
1. Remove any default processors from the pipeline.
1. Select the [**Datadog CloudPrem** destination][4] to forward logs to your CloudPrem instance. Leave the configuration empty.

{{< img src="/cloudprem/ingest/observability-pipelines-cloudprem-setup.png" alt="Screenshot of the Logs Explorer interface showing how to filter logs by selecting the cloudprem index in the facets panel" style="width:80%;" >}}
<!-- This image shows an example with dual shipping when the instructions say log volume control -->
Copy link
Contributor Author

@estherk15 estherk15 Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fmassot Could I remove this image?

<!-- {{< img src="/cloudprem/ingest/observability-pipelines-cloudprem-setup.png" alt="Screenshot of the Logs Explorer interface showing how to filter logs by selecting the cloudprem index in the facets panel" style="width:80%;" >}} -->
Copy link
Contributor Author

@estherk15 estherk15 Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<!-- {{< img src="/cloudprem/ingest/observability-pipelines-cloudprem-setup.png" alt="Screenshot of the Logs Explorer interface showing how to filter logs by selecting the cloudprem index in the facets panel" style="width:80%;" >}} -->



## Deploy your Observability Pipelines

While other methods exist, this guide uses the Helm command for deployment. The following command installs or upgrades the pipeline, configuring it to listen for logs and forward them to your CloudPrem indexer.
After creating your pipeline in the UI, deploy the Observability Pipelines Worker. The Worker runs your pipeline configuration and listens for logs from the Datadog Agent.

The following Helm command installs or upgrades the Worker, configuring it to listen for logs and forward them to your CloudPrem indexer.
<br>
**Note**: You need the `pipelineId` from the pipeline you created in the previous step. This ID links the Worker to your pipeline configuration.

```shell
helm upgrade --install opw \
-f values.yaml \
--set datadog.apiKey=XXXXXXX \
--set datadog.pipelineId=XXXXXXX \
--set env[0].name=DD_OP_SOURCE_HTTP_SERVER_ADDRESS,env[0].value='0.0.0.0:8282' \
--set env[1].name=DD_OP_DESTINATION_HTTP_CLIENT_URI,env[1].value='http://<RELEASE_NAME>-indexer.<NAMESPACE_NAME>.svc.cluster.local:7280/api/v2/datadog' \
Copy link
Contributor Author

@estherk15 estherk15 Dec 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't sure if this needs to be removed or not? The CloudPrem destination docs state that

The Worker appends /api/v2/logs and /api/v1/validate to the endpoint URL, so these endpoints must be allowed if you are using forwarding or firewall rules.

--set service.ports[0].name=dd-op-source-http-server-address-port,service.ports[0].protocol=TCP,service.ports[0].port=8282,service.ports[0].targetPort=8282 \
--set env[0].name=DD_OP_SOURCE_DATADOG_AGENT_ADDRESS,env[0].value='0.0.0.0:8282' \
--set env[1].name=DD_OP_DESTINATION_CLOUDPREM_ENDPOINT_URL,env[1].value='http://<RELEASE_NAME>-indexer.<NAMESPACE_NAME>.svc.cluster.local:7280' \
--set service.ports[0].name=dd-op-source-datadog-agent-address-port,service.ports[0].protocol=TCP,service.ports[0].port=8282,service.ports[0].targetPort=8282 \
datadog/observability-pipelines-worker
```

After a minute, you should see logs flowing through the pipelines and successfully reach the CloudPrem destination.
After a minute, verify that logs are flowing through the pipeline and reaching the CloudPrem destination. This indicates that the Worker is running and ready to receive logs, and you can proceed to configure the Agent.

## Configure Datadog Agent
## Configure the Datadog Agent

The final step is to update your Datadog Agent's configuration to point its logs to the newly deployed Observability Pipelines worker. You do this by setting the `DD_LOGS_CONFIG_LOGS_DD_URL` environment variable to the pipeline's address.
After the Observability Pipelines Worker is deployed and running, configure your Datadog Agent to send logs to it. The Agent connects to the Worker using the Worker's service address. For more information, see [Connect the Datadog Agent to the Observability Pipelines Worker][5].

Update your Datadog Agent configuration to send logs to the Observability Pipelines worker:

Expand All @@ -72,8 +81,10 @@
secretName: datadog-secret
keyName: api-key
env:
- name: DD_LOGS_CONFIG_LOGS_DD_URL
value: http://observability-pipelines-worker:8282
- name: DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED
value: "true"
- name: DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL
value: "http://observability-pipelines-worker:8282"

features:
logCollection:
Expand Down Expand Up @@ -101,3 +112,7 @@
{{< partial name="whats-next/whats-next.html" >}}

[1]: https://app.datadoghq.com/observability-pipelines
[2]: /observability_pipelines/configuration/explore_templates/?tab=logs#log-volume-control
[3]: /observability_pipelines/sources/datadog_agent/
[4]: /observability_pipelines/destinations/cloudprem/
[5]: /observability_pipelines/sources/datadog_agent/?tab=agenthelmvaluesfile#connect-the-datadog-agent-to-the-observability-pipelines-worker
Loading