How to correctly aggregate metrics when dropping host tag?

[syedg1]

Question

I am currently building a Datadog Agent -> Vector -> Datadog API pipeline to drop the host tag before it is sent to Datadog. The motivation behind this is to reduce cardinality and thereby costs. I understand that dropping the host tag imposes the risk of data point duplication in Datadog, which would result in duplicate points of the same time series overwriting original points.

In order to prevent this issue, I am using an aggregation transform to consolidate data points within Vector after the host tag has been dropped. This seems like it should work without incurring any data loss, but the filtered/aggregated metrics are consistently lower than the original metrics with the host tag. I have attached a screenshot of the test metrics.

blue: original metrics, purple: removing host with aggregation, yellow: removing host without aggregation

I would appreciate any guidance on how to aggregate these metrics such that they are accurate despite removing the host tag. Thank you!

Vector Config

# Sources - receive data from Datadog agents
    sources:
      datadog_agents:
        type: datadog_agent
        address: 0.0.0.0:8282
        multiple_outputs: true
        store_api_key: false
        
 # Transforms - delete unnecessary host tags (cardinality reduction)
    transforms:
      remove_host_tag:
        type: remap
        inputs:
          - datadog_agents.metrics
        source: |-
          # Add missing cluster tags using the same logic as datadog-agent.yaml lines 72-83
          # These should come from DD_TAGS but don't due to agent bug: https://github.com/DataDog/datadog-agent/issues/15128

          log("RAW DATA RECEIVED: " + encode_json(.))

          cluster_env = "${CLUSTER_APP_ENVIRONMENT}"
          cluster_group = "${CLUSTER_APP_GROUP}"
          cluster_name = "${CLUSTER_NAME}"
          cluster_region = "${CLUSTER_REGION}"

          if cluster_env != "" {
            .tags.env = cluster_env
          }
          if cluster_group != "" {
            .tags.app_group = cluster_group
          }
          if cluster_name != "" {
            .tags.eks_cluster = cluster_name
            .tags.cluster_name = cluster_name
            .tags.cluster_id = cluster_name
          }
          if cluster_region != "" {
            .tags.region = cluster_region
          }

          if (exists(.tags.host) && exists(.tags.remove_host_tag)) {
            host_value = .tags.host
            del(.tags.host)
            # If the metric is not a distribution, we need to set a tag for uniqueness. Use the vector host, which should have lower cardinality than the datadog host.
            if .kind != "distribution" {
              .tags.vector_host = "${HOSTNAME}"
            }
          }

          # log("UPDATED TAGS: " + encode_json(.tags))

      consolidate_metrics:
          type: aggregate
          inputs:
            - remove_host_tag

# Sinks - forward processed data to Datadog
    sinks:
      datadog_metrics:
        type: datadog_metrics
        inputs:
          - consolidate_metrics
        default_api_key: "${DATADOG_API_KEY}"
        site: "us5.datadoghq.com"

[pront]

Hi @syedg1, you are looking at requests count. This came up here before: #23675 (comment)

The datadog_metrics sink is batching events into one request hence less requests overall.