Skip to content

Slow performance when managing dozens of repositories #567

@mwarkentin

Description

@mwarkentin

Terraform Version

0.12.6

Affected Resource(s)

Please list the resources as a list, for example:

  • github_repository
  • github_branch_protection
  • github_team_repository
  • github_actions_secret

Terraform Configuration Files

Here's our repo module (slightly redacted ****):

terraform {
  required_providers {
    github = ">= 3.1.0"
  }
}

locals {
  # Terraform modules must be named `terraform-<provider>-<module name>`
  # so we can extract the provider easily
  provider = element(split("-", var.repository), 1)
}

data "github_team" "****" {
  slug = "****"
}

data "github_team" "****" {
  slug = "****"
}

resource "github_repository" "main" {
  name        = var.repository
  description = var.description

  visibility = var.visibility

  topics = [
    "terraform",
    "terraform-module",
    "terraform-${local.provider}"
  ]

  has_issues   = var.has_issues
  has_projects = var.has_projects
  has_wiki     = var.has_wiki

  vulnerability_alerts   = true
  delete_branch_on_merge = true

  archived = var.archived

  dynamic "template" {
    for_each = var.fork ? [] : [var.fork]

    content {
      owner      = "waveaccounting"
      repository = "****"
    }
  }
}

resource "github_branch_protection" "main" {
  repository_id = github_repository.main.node_id
  pattern       = github_repository.main.default_branch

  required_status_checks {
    strict = true
    contexts = [
      "Terraform",
      "docs",
    ]
  }

  required_pull_request_reviews {
    dismiss_stale_reviews      = true
    require_code_owner_reviews = true
  }
}

resource "github_team_repository" "****" {
  team_id    = data.github_team.****.id
  repository = github_repository.main.name
  permission = "admin"
}

resource "github_team_repository" "****" {
  team_id    = data.github_team.****.id
  repository = github_repository.main.name
  permission = "admin"
}

resource "github_actions_secret" "secrets" {
  for_each = var.secrets

  repository      = github_repository.main.name
  secret_name     = each.key
  plaintext_value = each.value
}

Actual Behavior

We are managing approximately 90 repositories using this module via Terraform Cloud remote operations (which means we can't disable refresh or change parallelization afaik). I timed a refresh + plan: 9m22s (562s) == 6.2s per repository

Are there any optimizations we can make on our side or in the github provider / API to try to improve this? We're discussing breaking up our repos into smaller workspaces, but that feels like a bit of a hack.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. terraform plan on large numbers of repositories / branch protection configs

Important Factoids

  • Running on Terraform Cloud Remote Operation

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions