Why is datadog-configuration in the global stack?

[StealthBadger747]

I am using datadog-lambda forwarders in the ue1 and uw2 regions and since I deployed the datadog api keys and configuration in our global stack (defaulting to uw2) and the key to auto/uw2, I have been experiencing drift and ssm access errors.

For example I have a datadog lambda forwarder for vpc-flow-logs in ue1 and while the datadog client is trying to access the ssm that it thinks exists in ue1 the policy and the actual key are in uw2 and I don't see any way to change that aside from modifying the TF itself.

I was wondering what the motivation for moving to the global stack is? And how should I migrate from the previous regional paradigm?

[GabisCampana]

@Benbentwo

[Benbentwo]

I was wondering what the motivation for moving to the global stack is?

The reason for this is because we want a consistent way to lookup the datadog configuration component via remote state.

When we deploy datadog configuration to global region, it has a consistent environment, but a differing region per implementation. For example, it is deployed to us-east-1 for one client, but us-west-2 for another. but their component is both deployed to the gbl environment.

This is useful for the remote state lookup which can then use gbl as the environment rather than the region of the current component. though this can be changed via the variable var.global_environment_name in the "../datadog-configuration/modules/datadog_keys" module (usually the referencing component has this). The datadog configuration component should store the keys in your default or primary region, and the datadog_keys module should look them up in that region. notice the AWS Provider for the datadog_keys submodule is configured to use the region output by the datadog_configuration component.

The other nice thing about global'ly defined components is they make it known that it should only deployed once. As opposed to having multiple datadog configurations which would pollute the stacks and confuse on "which one should I use"

---

how should I migrate from the previous regional paradigm?

First I'd validate that your datadog_configuration component output contains us-west-2 as it's output region. then I'd add a test output to one of your lambda forwarders to double check that the region being input to the datadog keys is us-west-2. It's hard to tell without much code, so I'd add outputs to validate each step along the way

[StealthBadger747]

Thanks Ben! Just have a followup question

When we deploy datadog configuration to global region, it has a consistent environment, but a differing region per implementation. For example, it is deployed to us-east-1 for one client, but us-west-2 for another. but their component is both deployed to the gbl environment.

Just a clarification on what this means. If I were to add it as a mixin in stacks/mixins/region/(global.yaml & us-east-1.yaml & us-west-2.yaml) what would be correct? Or does it belong only in the global file?

---

Actually as a more complete example I am rolling it out in the audit account:
infra/stacks/orgs/hl/core/audit/*
global.yaml

import:
  - mixins/region/global
  - orgs/hl/core/audit/_defaults
  - catalog/terraform/default-team-roles
  - catalog/terraform/datadog-configuration
  - catalog/terraform/datadog-integration

us-east-1.yaml

import:
  - mixins/region/us-east-1
  - orgs/hl/core/audit/_defaults
  - catalog/terraform/vpc-flow-logs-bucket

components:
  terraform:
    datadog-lambda-forwarder:
      vars:
        s3_buckets:
          - "hl-core-ue1-audit-vpc-flow-logs"

us-west-2.yaml

import:
  - mixins/region/us-west-2
  - orgs/hl/core/audit/_defaults
  - catalog/terraform/vpc-flow-logs-bucket

components:
  terraform:
    cloudtrail-bucket:
      vars:
        enabled: true
        name: "cloudtrail"
        noncurrent_version_expiration_days: 180
        noncurrent_version_transition_days: 30
        standard_transition_days: 60
        glacier_transition_days: 180
        expiration_days: 2192
        create_access_log_bucket: false

    datadog-lambda-forwarder:
      vars:
        cloudtrail_read_enabled: true
        s3_bucket_kms_arns: ["*"]
        s3_buckets:
          - "hl-core-uw2-audit-cloudtrail"
          - "hl-core-uw2-audit-vpc-flow-logs"

And my datadog configuration looks like this:

components:
  terraform:
    datadog-configuration:
      vars:
        enabled: true
        name: datadog-configuration
        datadog_secrets_store_type: SSM
        datadog_secrets_source_store_account_stage: auto
        datadog_secrets_source_store_account_region: "us-west-2"
        datadog_api_secret_key: "default"
        datadog_app_secret_key: "default"

And default lambda forwarder:

components:
  terraform:
    datadog-lambda-forwarder:
      settings:
        github:
          actions_enabled: true
      vars:
        enabled: true
        dd_forwarder_version: 3.127.0
        lambda_runtime: "python3.12"
        name: datadog-lambda-forwarder
        forwarder_log_enabled: true
        s3_bucket_kms_arns: ["*"]

[raybotha]

Practically, it doesn't seem like the lambda forwarder reads parameters cross region? Based on the DD_API_KEY_SSM_NAME environment variable passed into it.
If all our AWS components that use datadog need to read from a regional SSM parameter store deployment in the primary region, isn't that a major issue for fault tolerance to regional failures? Our main reason for having a secondary region is to be able to handle regional outages.

[GabisCampana]

@Benbentwo

[raybotha]

Saw your PR @Benbentwo looks like that would address it!

[raybotha]

This seems like it could be of interest for simplifying configuration in the future: https://aws.amazon.com/about-aws/whats-new/2024/02/aws-systems-manager-parameter-store-cross-account-sharing/