Getting "Federated principals must be valid domain names or SAML metadata ARNs" when running atmos workflow deploy -f baseline

[cmarquezrusso]

We are currently deploying CloudTrail and ECR (https://docs.cloudposse.com/layers/accounts/account-baseline/).

When I run

atmos workflow deploy -f baseline

I get:

Error: creating IAM Role (xxxx-core-usw2-artifacts-ecr-gha): operation error IAM: CreateRole, https response error StatusCode: 400, RequestID: xxxxxxxxxxxxxxxxxx, MalformedPolicyDocument: Federated principals must be valid domain names or SAML metadata ARNs

I can see that the error is on the terraform deploy ecr -s core-usw2-artifacts stage. Doing a manual plan shows the error (missing Federated Principal:

data.aws_iam_policy_document.github_actions_iam_policy: Read complete after 0s [id=xxxxxx]

OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated
with the following symbols:
  + create

OpenTofu will perform the following actions:

  # aws_iam_role.github_actions[0] will be created
  + resource "aws_iam_role" "github_actions" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = [
                          + "sts:TagSession",
                          + "sts:SetSourceIdentity",
                          + "sts:AssumeRoleWithWebIdentity",
                        ]
                      + Condition = {
                          + StringEquals = {
                              + "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
                            }
                          + StringLike   = {
                              + "token.actions.githubusercontent.com:sub" = [
                                  + "repo:xxxx/vbpt-infra:*",
                                  + "repo:xxxx/example-app-on-ecs:*",
                                  + "repo:xxxx/example-app-on-lambda-with-gha:*",
                                ]
                            }
                        }
                      + Effect    = "Allow"
                      + Principal = {
                          + Federated = ""
                        }
                      + Sid       = "OidcProviderAssume"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )

The previous commands ran successfully.

Also, running atmos terraform deploy ecr -s core-use1-artifacts as per https://docs.cloudposse.com/layers/accounts/account-baseline/ fails with

Could not find the component 'ecr' in the stack 'core-use1-artifacts'. Check that all the context   
variables are correctly defined in the stack manifests. Are the component and stack names correct?  
Did you forget an import? 

[milldr]

In the ecr stack catalog, stacks/catalog/ecr.yaml, set github_actions_iam_role_enabled: false

Later on you will deploy the GitHub OIDC Provider, which will create that value. When youre ready to start allowing GitHub access to AWS, then you can reenable github_actions_iam_role_enabled and redeploy the ecr component.

[milldr]

After the community workshop call today, we determined the real underlying issue was order of operations. AWS Teams need to be deployed before the ECR policy, so we should deploy the identity layer before ecr.

I've added a task to update the documentation and workflows

[dukarc]

Thanks @milldr

We went to the identity workflow and ran two commands to try to bypass per suggestion. Unfortunately both are failing

1. workflow deploy/sso -f identity

module.sso_account_assignments_root.data.aws_ssoadmin_instances.this: Read complete after 0s [id=us-west-2]

OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

OpenTofu planned the following actions, but then encountered a problem:

  # module.sso_account_assignments.null_resource.dependency will be created
  + resource "null_resource" "dependency" {
      + id       = (known after apply)
      + triggers = {
          + "dependency_id" = ""
        }
    }

  # module.sso_account_assignments_root.null_resource.dependency will be created
  + resource "null_resource" "dependency" {
      + id       = (known after apply)
      + triggers = {
          + "dependency_id" = ""
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + group_ids = {}
╷
│ Warning: Deprecated Parameters
│ 
│   with module.account_map.data.terraform_remote_state.data_source[0],
│   on .terraform/modules/account_map/modules/remote-state/data-source.tf line 88, in data "terraform_remote_state" "data_source":
│   88: data "terraform_remote_state" "data_source" {
│ 
│ The following parameters have been deprecated. Replace them as follows:
│   * role_arn -> assume_role.role_arn
│ 
│ 
│ (and 9 more similar warnings elsewhere)
╵
╷
│ Error: Invalid index
│ 
│   on .terraform/modules/permission_sets/modules/permission-sets/main.tf line 53, in locals:
│   53:   sso_instance_arn    = tolist(data.aws_ssoadmin_instances.this.arns)[0]
│     ├────────────────
│     │ data.aws_ssoadmin_instances.this.arns is empty list of string
│ 
│ The given key does not identify an element in this collection value: the collection has no elements.
╵
╷
│ Error: Invalid index
│ 
│   on .terraform/modules/sso_account_assignments/modules/account-assignments/main.tf line 63, in locals:
│   63:   identity_store_id = tolist(data.aws_ssoadmin_instances.this.identity_store_ids)[0]
│     ├────────────────
│     │ data.aws_ssoadmin_instances.this.identity_store_ids is empty list of string
│ 
│ The given key does not identify an element in this collection value: the collection has no elements.
╵
╷
│ Error: Invalid index
│ 
│   on .terraform/modules/sso_account_assignments/modules/account-assignments/main.tf line 64, in locals:
│   64:   sso_instance_arn  = tolist(data.aws_ssoadmin_instances.this.arns)[0]
│     ├────────────────
│     │ data.aws_ssoadmin_instances.this.arns is empty list of string
│ 
│ The given key does not identify an element in this collection value: the collection has no elements.
╵
╷
│ Error: Invalid index
│ 
│   on .terraform/modules/sso_account_assignments_root/modules/account-assignments/main.tf line 63, in locals:
│   63:   identity_store_id = tolist(data.aws_ssoadmin_instances.this.identity_store_ids)[0]
│     ├────────────────
│     │ data.aws_ssoadmin_instances.this.identity_store_ids is empty list of string
│ 
│ The given key does not identify an element in this collection value: the collection has no elements.
╵
╷
│ Error: Invalid index
│ 
│   on .terraform/modules/sso_account_assignments_root/modules/account-assignments/main.tf line 64, in locals:
│   64:   sso_instance_arn  = tolist(data.aws_ssoadmin_instances.this.arns)[0]
│     ├────────────────
│     │ data.aws_ssoadmin_instances.this.arns is empty list of string
│ 
│ The given key does not identify an element in this collection value: the collection has no elements.
╵
 Error 

exit status 1

2. atmos workflow deploy/teams -f identity

╷
│ Error: creating IAM Role (vbpt-core-gbl-identity-gitops): operation error IAM: CreateRole, https response error StatusCode: 400, RequestID: 6978c972-595a-4095-858f-6c93b24de750, MalformedPolicyDocument: Federated principals must be valid domain names or SAML metadata ARNs
│ 
│   with aws_iam_role.default["gitops"],
│   on main.tf line 61, in resource "aws_iam_role" "default":
│   61: resource "aws_iam_role" "default" {
│ 
╵
╷
│ Error: creating IAM Role (vbpt-core-gbl-identity-planners): operation error IAM: CreateRole, https response error StatusCode: 400, RequestID: afb5b61e-84de-4185-887e-4b75c02922d7, MalformedPolicyDocument: Federated principals must be valid domain names or SAML metadata ARNs
│ 
│   with aws_iam_role.default["planners"],
│   on main.tf line 61, in resource "aws_iam_role" "default":
│   61: resource "aws_iam_role" "default" {
│ 
╵
 Error 

exit status 1
 Error 

Step 'step1' failed!
To resume the workflow from this step, run:
atmos workflow deploy/teams -f identity --from-step step1

we looked into the request being sent for the policy and noticed that the Federated Principal is still empty

 {
      "Sid": "OidcProviderAssume",
      "Effect": "Allow",
      "Principal": {
        "Federated": ""
      },
      "Action": [
        "sts:TagSession",
        "sts:SetSourceIdentity",
        "sts:AssumeRoleWithWebIdentity"
      ],
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:Vibrant-Planet/vbpt-infra:*"
        }
 }

[milldr]

2. atmos workflow deploy/teams -f identity

We saw this same issue recently and it comes down again to order of operations. I will spend some time tomorrow updating the documentation to better explain this. In the meanwhile ...

Why this is happen

Both the gitops and planners roles are intended to be assumed by GitHub Actions to run Atmos and Terraform workflows. However, the GitHub OIDC provider has not been provisioned yet. The policy is attempting to include the ARN from the provider, but is given an empty string. Therefore the policy is invalid and Terraform fails

What do we want

We want to skip anything that requires the GitHub OIDC provider until it's ready. Since we provision roles following the least privilege principal, we have several roles for each task we need from GitHub Actions - planning Terraform, applying Terraform, accessing ECR, accessing EKS, etc.

However, we don't want that to stop us from provisioning the underlying resource. We should be able to provision all other roles, deploy ECR, and deploy EKS.

How do you fix this now and unblock yourselves?

For now, disable all roles created for GitHub Actions.

1. In stacks/catalog/ecr.yaml, set github_actions_iam_role_enabled: false. This will disable the GitHub access policy for ECR, but we do not need it until you need to push images to ECR with GitHub Actions either with your docker build workflow or application CI.
2. In stacks/catalog/aws-teams.yaml, comment out the trusted_github_repos input (most likely lines 8-12). This will remove the GitHub access policy from the planners and gitops roles. The roles should still be provisioned otherwise, but wont be used until you get to the gitops layer.
3. (optional, if relevant) In stacks/catalog/eks/clusters/plat.yaml, set github_actions_iam_role_enabled: false.

What will Cloud Posse do to fix this?

I'll update the docs to better explain this, change the default for all of these to false, and include steps for when these should be enabled

[milldr]

regarding

╷
│ Error: Invalid index
│
│ on .terraform/modules/permission_sets/modules/permission-sets/main.tf line 53, in locals:
│ 53: sso_instance_arn = tolist(data.aws_ssoadmin_instances.this.arns)[0]
│ ├────────────────
│ │ data.aws_ssoadmin_instances.this.arns is empty list of string
│
│ The given key does not identify an element in this collection value: the collection has no elements.
╵

This sounds like Terraform cant find the existing instance. Did you enable the service in the same region as your default?
https://docs.cloudposse.com/layers/identity/aws-sso/#prerequisites

Also, you had mentioned configuring Identity Center previously. Is this the same account and region?

[milldr]

I've opened a PR to address the issue above. For reference

[milldr]

FYI these changes are published to the docs now