Removed tags still appearing in datadog

[jamhamblin]

When removing tags from within our helm config we are still seeing tags appear within datadog. Is there something obvious I'm missing or are we coming across an error.

Helm config:

      remove_unwanted_tags:
        type: remap
        inputs:
          - datadog_agent
        # https://vector.dev/docs/reference/vrl/expressions/#path-example-quoted-path
        source: |-
          del(.tags.aws_autoscaling_groupname)
          del(.tags.aws_ec2launchtemplate_id)
          del(.tags.aws_ec2launchtemplate_version)

[jamhamblin]

For additional context, we have verified that the VRL language is valid using an example json in the documentation and the command line. Would appreciate any guidance as to what config changes we need to make to ensure that tags are dropped from ingested_tags in datadog metrics.

[jszwedko]

One thing I would check is if the events are being sent with those tags to Datadog or if they are associated once ingested in Datadog via host tag enrichment (i.e. the Datadog back-end adds tags that were fetched via other integrations based on the host). You can use vector tap to examine the events flowing through Vector itself. I'd suggest tapping the events going into the Datadog Metrics sink.

[jamhamblin]

Hi thanks for this, Vector tap was really useful and allowed us to prove that the tags are being removed as expected. It must be being added by different integrations. Thanks for your help!

[jszwedko]

Glad to hear you tracked it down!

[lingzhuimg]

Hi @jszwedko, thanks for your reply. When I run vector tap against a source, is there an argument of e.g. vector API (podIP:port/path) to query from outside a vector instance (k8s pod)? I could not find a clear description in Vector docs. Many thanks.

[jszwedko]

Hi @lingzhuimg ,

You can use the --url flag on vector tap to point it to a remote instance. Something like http://podIP:port/graphql should work.