datadog_archive sink?

[cspargo]

curious if this sink is intended to be used? it doesn't appear to have any documentation.

#8929

[jszwedko]

Hi @cspargo !

It's not quite usable yet because there is no way to rehydrate from it in the Datadog UI, but this is something that we hope to make possible soon at which point we will document and release it.

[cspargo-apptio]

sounds good, thanks!

we were wondering how rehydration could possibly work without the archives being defined in datadog.

[topkoa]

Hey. I found a way to do this.

If you already have archives set up that you can re-hydrate from, its all in how you set up the file name and folder, and the sink particulars. I'll explain below:

In your "transforms", you will need to format the logs destined for cloud storage to look like this, with this structure:

transforms:
  format_datadog_style:
    type: "remap"
    inputs:
    - my_upstream_input
    source: |
      # Generate unique ID (using timestamp + random component)
      ._id = encode_base64(to_string(now()) + "-" + uuid_v4())

      # Store timestamp once
      ts = format_timestamp!(now(), format: "%+")

      # Create nested attributes structure - use ! to abort on failure
      .attributes = {
        ...your json attributes, you can usr VRL to pull these in from the original message...
      }

      # Set top-level fields
      .date = ts
      .message = "" # Blank for example, but you will probably want to pull this from original message object

      # Create tags array - ensure strings
      .tags = ["source:foo", "service:bar", "foo:bar"]

In your "sinks", you will need to write the files with these settings:

sinks:
  logs_to_gcs_rehydratable_in_datadog:
    type: gcp_cloud_storage
    credentials_path: /path/to/credentials.json
    inputs:
    - format_datadog_style
    bucket: "my-bucket-name"
    filename_append_uuid: true
    filename_extension: "json.gz"
    compression: "gzip"
    key_prefix: "dt=%Y%m%d/hour=%H/archive_vector_"
    encoding:
      codec: "json"
      json:
        pretty: false
    framing:
      method: "newline_delimited"

I have this tested and working as expected.
It took a little reverse engineering to get it right....

As for this question: "we were wondering how rehydration could possibly work without the archives being defined in datadog."

if you already have the archive set up and re-hydrations are working (i.e. you are currently sending logs that were already ingested) -- all you need to do is create a different service account with IAM rules to write to the same bucket, and use that.

Cheers.