SSM parameters can exceed 4K limit breaking terraform apply

[thresheek]

Hello!

We're using multi-runner 5.9.0 with SSM parameters store.

With a sufficiently high amount of defined runners - we currently have 15, but plan to scale to ~40 - we get the following error when applying terraform changes:

│ Error: updating SSM Parameter (/github-action-runners/gh-ci/webhook/runner-matcher-config): ValidationException:
Standard tier parameters support a maximum parameter value of 4096 characters.
To create a larger parameter value, upgrade the parameter to use the advanced-parameter tier.
For more information, see https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-advanced-parameters.html
│ 	status code: 400, request id: c3f391e5-8d78-46d1-af02-acaead059fe6
│
│   with module.runners.module.webhook.aws_ssm_parameter.runner_matcher_config,
│   on .terraform/modules/runners/modules/webhook/main.tf line 60, in resource "aws_ssm_parameter" "runner_matcher_config":
│   60: resource "aws_ssm_parameter" "runner_matcher_config" {

Now, manually moving the parameter to Advanced tier works around the current issue, but this highlights a problem we'd hitwhen we will define more runners.
While we probably can save a few bytes by not providing a self-hosted tag, I don't think it'll make much of difference. Here's the typical part of a json that's in that parameter:

  {
    "arn": "arn:aws:sqs:us-east-1:1234567890:gh-ci-debian-11-arm64-queued-builds.fifo",
    "fifo": true,
    "id": "https://sqs.us-east-1.amazonaws.com/1234567890/gh-ci-debian-11-arm64-queued-builds.fifo",
    "key": "debian-11-arm64",
    "matcherConfig": {
      "exactMatch": true,
      "labelMatchers": [
        [
          "self-hosted",
          "debian-11-arm64"
        ]
      ],
      "priority": 1
    }
  },

looks like arn/id are the best candidates to trim down.

[npalm]

We moved from environment configuration 5.8.0 to SSM (by @GuptaNavdeep1983). This all to avoid the max size of parameters. It seems we overlooked the max on SSM. The default max is 4kb as well. You can move to advanced paramater store (8kb limit).

[thresheek]

Hi @npalm!

Yep, that's what I did locally - moved to 8kb. However that limit is also kinda low for our use-case with a higher amount of runner definitions - we currently have 20 and take up around 6k bytes in that SSM param.

We plan to scale to 40 at least, quite possibly more. We aim to support a matrix of runners for most popular distributions - e.g. all current RHEL, Rocky, Alma, Ubuntu, Debian, SLES, Alpine Linux - and both amd64 and arm64 at that, so that makes the number of defined runners pretty high.

[npalm]

Which means we should find another way. Any help / suggestions are welcome here.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs. Thank you for your contributions.

[kgoralski]

@npalm Secret Manager supports 64kb. I believe at the current moment 8kb is upper limit?

[thresheek]

Correct, 8kb is the upper limit currently for SSM.

[dyasny]

Just started hitting the same issue

[nmodi1]

Facing these issues in our multi-runner deployment as well. Interestingly, only one of the runner configs seems to be affected by this issue...

There seem to be 2 PR's to resolve the issue. The first one will resolve the issue on the short-term at least.

• fix: Dynamic set SSM-paramater tier #4613 - has been approved. Hopefully it's deployed soon.
• feat: use dynamodb instead of ssm for JIT config #4446 - which probably needs more work