instance.kustomize.patches on a helm_release for the flux_instance

[MeTimesThree]

I would like to increase the memory-limit of the source-controller in the terraform helm-chart (since i run into OOMMemory restarts with a very large repo).

For this i came up with the following patch:

patchesJSON6902:
- target:
    group: apps
    version: v1
    kind: Deployment
    name: source-controller
  patch: |-
    - op: replace
      path: /spec/template/spec/containers/0/resources/limits/memory
      value: 7Gi

The results looked good. I then transferred the patch into a terraform local.

locals {
  kust_source_memory_yaml = {
    patches= [
      {
        target= {
          group= "apps",
          version= "v1",
          kind= "Deployment",
          name= "source-controller"
        },
        patch= [
          {
            op= "replace",
            path= "/spec/template/spec/containers/0/resources/limits/memory",
            value= "7Gi"
          }
        ]
      }
    ]
  }
}

I then tried some ways to use it in the helm_release for the flux_instance as instance.kustomize.patches:

  set = [
    {
      name  = "instance.kustomize.patches[0]"
      value = yamlencode(local.kust_source_memory_yaml)
    },
  ]

Giving the error:
instance.kustomize.patches.0: Invalid type. Expected: object, given: string

  set = [
    {
      name  = "instance.kustomize.patches[0]"
      value = local.kust_source_memory_yaml
    },
  ]

Giving the error:
Inappropriate value for attribute "set": element 0: attribute "value": string required, but have object.

values = [ join("", [ "instance.kustomize.patches[0]: ", yamlencode(local.kust_source_memory_yaml) ]) ]

Giving the error:
│ Error: Error unmarshaling values
│
│ with helm_release.flux_instance,
│ on fluxCD.tf line 77, in resource "helm_release" "flux_instance":
│ 77: resource "helm_release" "flux_instance" {
│
│ ---> error converting YAML to JSON: yaml: mapping values are not allowed in this context
│ instance.kustomize.patches[0]: "spec":
│ "kustomize":
│ "patches":
│ - "patch":
│ - "op": "replace"
│ "path": "/spec/template/spec/containers/0/resources/limits/memory"
│ "value": "7Gi"
│ "target":
│ "group": "apps"
│ "kind": "Deployment"
│ "name": "source-controller"
│ "version": "v1"
│

values = [ join("", [ "instance.kustomize.patches[0]: ", local.kust_source_memory_yaml ]) ]

Giving the error:
│ Error: Invalid function argument
│
│ on fluxCD.tf line 137, in resource "helm_release" "flux_instance":
│ 137: values = [ join("", [ "instance.kustomize.patches[0]: ", local.kust_source_memory_yaml ]) ]
│ ├────────────────
│ │ while calling join(separator, lists...)
│ │ local.kust_source_memory_yaml is object with 1 attribute "spec"
│
│ Invalid value for "lists" parameter: element 1: string required, but have object.

Does anybody have an idea what i am doing wrong here?

FYI:

I am using the Flux Operator with terraform
flux-v2.7.5

[stefanprodan]

Set the patchs in the yaml directly like this https://github.com/controlplaneio-fluxcd/flux-operator/blob/v0.41.1/config/terraform/values/components.yaml

[MeTimesThree]

The following worked like a charm:

locals {

  kustomize_patches = <<EOT
instance:
  kustomize:
    patches:
      - target:
          group: apps
          version: v1
          kind: Deployment
          name: source-controller
        patch: |
          - op: replace
            path: /spec/template/spec/containers/0/resources/limits/memory
            value: 7Gi
EOT
}

And in the helm_release:
values = [ local.kustomize_patches ]

Thank you very much @stefanprodan!

[stefanprodan]

i run into OOMMemory restarts with a very large repo

You can configure Flux to pull from remote only the dirs with Kubernetes manifests with GitRepository.spec.sparseCheckout.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: podinfo
  namespace: default
spec:
  interval: 5m
  url: https://github.com/stefanprodan/podinfo
  ref:
    branch: master
  sparseCheckout:
  - charts
  - kustomize

Docs here: https://fluxcd.io/flux/components/source/gitrepositories/#sparse-checkout

[MeTimesThree]

Worked amazingly well. I just removed all the "instance.sync"-Parameters and manually added the GitRepository and the Kustomization like so:

locals {
  forgejo_git_url = "https://REDACTED.git"
  forgejo_git_path = "REDACTED/clusters/REDACTED"
  flux_repo_name = "base"
}


resource "kubectl_manifest" "flux_repo" {
  depends_on = [helm_release.flux_operator, helm_release.flux_instance]
  yaml_body  = <<YAML

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: ${local.flux_repo_name}
  namespace: ${one(kubernetes_namespace_v1.fluxcd.metadata).name}
spec:
  interval: 10m0s
  ref:
    name: refs/heads/master
    branch: master
  secretRef:
    name: ${one(kubernetes_secret_v1.forgejo.metadata).name}
  timeout: 60s
  url: http://REDACTED.git
  sparseCheckout:
  - ${local.forgejo_git_path}

YAML
}

resource "kubectl_manifest" "flux_kustomization" {
  depends_on = [helm_release.flux_operator, helm_release.flux_instance, kubectl_manifest.flux_repo]
  yaml_body  = <<YAML

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-system
  namespace: ${one(kubernetes_namespace_v1.fluxcd.metadata).name}
spec:
  force: false
  interval: 10m0s
  path: ${local.forgejo_git_path}
  prune: true
  sourceRef:
    kind: GitRepository
    name: ${local.flux_repo_name}

YAML
}

Thanks so much for your remark @stefanprodan!

[stefanprodan]

You shouldn't be doing that, your Terraform is fully broken, the GitRepo CRD is not present on new cluster so it will fail.

Just patch the FluxInstance with:

    patches:
      - patch: |
          - op: add
            path: /spec/sparseCheckout
            value:
              - your-path
        target:
          kind: GitRepository

Docs here: https://fluxoperator.dev/docs/instance/customization/#cluster-sync-gitrepository-verification