Add user_kustomizations-map for specifying Kustomization-"sets" to be applied sequentially

[vsalomaki]

Description

Adds user_kustomizations-parameter and supporting modules, which can be used to define multiple Kustomization-sets that are run sequentially. This allows to e.g. install CRD in one step, wait for their deployment and then proceed with the next set.

NOTE: Breaking change, user_kustomizations replaces extra_kustomize_deployment_commands, extra_kustomize_parameters and extra_kustomize_folder. See MIGRATION.md

Key point from kube.tf.example:

You can add user kustomizations to be deployed in sequence by setting the user_kustomizations variable.
The Kustomization "sets" are run in sequential order (by numeric key) so that you can for example install a CRD and wait for it to be deployed.

Properties of each value:

• source_folder: Sets the source folder for *.yaml.tpl and Kustomization.yaml.tpl
• kustomize_parameters: Key-value map for passing variables into Kustomization. Applies only to the Kustomization-set in the object, but to all files defined in the source_folder of the "set". Defaults to {}.
• pre_commands: Commands to be executed before applying the Kustomization ("kubectl apply -k"). Defaults to "".
• post_commands: Commands to be executed after applying the Kustomization ("kubectl apply -k"). You can use it to wait for CRD deployment etc. Defaults to "".
  -- An example to wait for deployments in all namespaces: kubectl wait --for=condition=Available deployment --all -A --timeout=120s || true (The || true is necessary to prevent the script from exiting on a timeout if you want the sequence to continue.)
  -- It is recommended to use more specific kubectl wait commands depending on the case, for example filtering for a certain deployment or pod.
  -- You can pass full bash-compatible scripts into the post_commands-variable with EOT

An example:

  user_kustomizations = {
    "1" = {
      source_folder        = "extra-manifests"
      kustomize_parameters = { myvar = "myvalue" }
      pre_commands         = ""
      post_commands        = "kubectl wait --for=condition=Available deployment --all -A --timeout=120s || true"
    }
  }

Test plan

• Test without the setting, validate that nothing changes.
• Add example helm-chart to folder extra-manifest-preinstall and apply. Check that the required helm-charts were applied.

Gemini

Summary of Changes

This pull request significantly enhances the Kubernetes cluster bootstrapping process by introducing a flexible, sequential Kustomization deployment mechanism. It refactors existing pre- and post-install Kustomization logic into a modular, configurable system, allowing users to define multiple Kustomization steps with specific ordering and post-deployment actions. This change improves maintainability, addresses potential conflicts, and provides greater control over complex application deployments, particularly for scenarios requiring CRD installation before dependent applications.

Highlights

• Sequential Kustomization Deployment: Introduced a new user_kustomizations variable that allows defining multiple Kustomization 'sets' to be applied sequentially. This enables complex deployment strategies, such as installing CRDs and waiting for their readiness before deploying dependent applications.
• Modular Refactoring: The previous ad-hoc pre- and post-install Kustomization logic has been refactored into two new, reusable Terraform modules: user_kustomization_set and user_kustomizations. This significantly improves code maintainability, reduces duplication, and centralizes deployment logic.
• Enhanced Flexibility with Pre/Post Commands: Each Kustomization set can now include pre_commands and post_commands to execute arbitrary bash scripts before and after the kubectl apply -k step, offering fine-grained control over the deployment flow and enabling robust wait conditions.
• Improved Deployment Robustness: The remote execution script for applying Kustomizations now includes set -e for error handling, a trap cleanup EXIT for guaranteed cleanup of temporary files, and checks ([ -s "$dest_folder/kustomization.yaml" ]) to prevent kubectl apply -k from failing on empty Kustomization directories.
• Sensitive Variable Handling: Key variables, including kustomize_parameters, pre_commands, and post_commands within the new user_kustomizations structure, are now correctly marked as sensitive = true to prevent their values from being exposed in Terraform plan/apply outputs. The old extra_kustomize_deployment_commands and extra_kustomize_parameters variables were also marked sensitive before removal.
• Comprehensive Documentation and Migration Updates: The README.md and kube.tf.example files have been updated to reflect the new user_kustomizations approach, including detailed explanations and usage examples. A new MIGRATION.md file provides clear instructions for users to transition from the deprecated extra_kustomize_* variables to the new user_kustomizations map.

Changelog

• MIGRATION.md

  • Added a new migration guide detailing the transition from extra_kustomize_* variables to the new user_kustomizations map, including steps and an example configuration.

• README.md

  • Updated the summary to include 'pre- and post-install actions'.
  • Added a new section 'Adding applications with Helm, custom pre- and post-install actions' with detailed instructions and an example for pre-installing Helm Charts (e.g., external-secrets).
  • Renamed 'Custom post-install actions' to 'Post-install actions, ArgoCD-example' and updated its content to use the new user_kustomizations structure.
  • Updated references to extra_kustomize_parameters to user_kustomizations.

• docs/llms.md

  • Updated the Kustomize and Post-Deployment Operations section to document the new user_kustomizations variable and its properties, replacing the old extra_kustomize_deployment_commands and extra_kustomize_parameters documentation.

• examples/kustomization_user_deploy/README.md

  • Added a note about using user_kustomizations for different folders.
  • Updated examples to use user_kustomizations and its kustomize_parameters field instead of extra_kustomize_parameters.

• kube.tf.example

  • Replaced comments for extra_kustomize_deployment_commands and extra_kustomize_parameters with detailed documentation and an example for the new user_kustomizations variable, outlining its properties and usage for sequential deployments.

• kustomization_user.tf

  • Removed previous null_resource blocks (kustomization_user, kustomization_user_deploy) related to the old Kustomization handling.
  • Implemented local.default_user_kustomize to define a default two-step Kustomization process (pre-install then main manifests).
  • Introduced local.user_kustomize_defaulted to allow overriding default Kustomization sets with var.user_kustomizations.
  • Added local.processed_kustomizes to process and mark kustomize_parameters, pre_commands, and post_commands as sensitive.
  • Integrated a new module "user_kustomizations" to orchestrate the deployment of the defined Kustomization sets.

• modules/user_kustomization_set/

  • New Module: Created to handle the copying and templating of a single Kustomization 'set' to the remote control plane.
  • locals.tf: Defines SHAs for source files, parameters, and pre/post commands for change detection.
  • main.tf: Manages directory creation, copies templated .yaml files, and preinstall.sh/postinstall.sh scripts to the remote host, ensuring subdirectories are created.
  • out.tf: Exports destination_folder, various SHAs, files_count (number of template files), and changes_sha for use by parent modules.
  • templates/bash.sh.tpl: A simple bash script template for executing commands, including set -e for error handling.
  • variables.tf: Defines input variables including ssh_connection (sensitive), source_folder, destination_folder, template_parameters (sensitive, map(any)), pre_commands_string (sensitive), and post_commands_string (sensitive).

• modules/user_kustomizations/

  • New Module: Created to manage multiple user_kustomization_set instances and orchestrate their sequential deployment.
  • locals.tf: Filters and sorts destination folders for Kustomization sets that contain actual files or post-commands.
  • main.tf: Instantiates module "user_kustomization_set" for each defined Kustomization, then uses a null_resource to execute a remote bash script that applies each Kustomization sequentially, runs its pre/post-install scripts, and cleans up the remote directories. Includes robust error handling (set -e), guaranteed cleanup (trap cleanup EXIT), and checks for valid kustomization files ([ -s ... ]).
  • variables.tf: Defines ssh_connection (sensitive) and kustomizations_map (a map of Kustomization set configurations with validation for numeric keys, kustomize_parameters as map(any), and pre_commands/post_commands as sensitive).

• variables.tf

  • Removed extra_kustomize_deployment_commands, extra_kustomize_parameters, and extra_kustomize_folder variables.
  • Added the new user_kustomizations variable (type map(object) with kustomize_parameters as map(any), and pre_commands/post_commands as optional strings), allowing for sequential Kustomization deployments. Includes validation to ensure keys are positive numeric strings.

[gemini-code-assist]

Summary of Changes

Hello @vsalomaki, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant enhancement to the cluster bootstrapping process by implementing a "pre-install Kustomization" capability. This allows for the execution of specific Kubernetes manifests, such as Helm Charts or Custom Resource Definitions (CRDs), before the primary Kustomization application. This feature is crucial for scenarios where certain components, like CRDs, must be fully deployed and stable before dependent applications can be installed, providing greater control and flexibility in complex Kubernetes deployments.

Highlights

• Pre-install Kustomization: Introduced a new mechanism to apply Kustomization manifests before the main Kustomization step.
• Helm Chart and CRD Deployment: Enables the deployment of Helm Charts and Custom Resource Definitions (CRDs) in a dedicated pre-installation phase, addressing dependencies for complex applications.
• New Configuration Options: Added extra_kustomize_pre_install_folder to specify the pre-install manifest directory and extra_kustomize_pre_install_post_deploy_commands for custom actions (e.g., waiting for CRDs) after pre-install application.
• Documentation Update: The README.md now includes detailed instructions and an example for using the pre-install feature with external-secrets.
• Terraform Resource Integration: New Terraform null_resource blocks (kustomization_user_pre_install_templates, kustomization_user_pre_install_deploy) were added to manage the rendering and application of pre-install Kustomizations.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a pre-install step for Kustomize, allowing Helm charts and other manifests to be applied before the main extra-manifests. This is a useful feature for deploying resources with dependencies, like CRDs.

The implementation has a critical bug in kustomization_user.tf where the pre-install step is triggered by the wrong condition. Additionally, there's a design issue where both pre-install and post-install steps use the same remote directory, which can lead to conflicts and redundant operations. I've provided suggestions to fix these issues.

I've also included some minor suggestions to improve the clarity of the documentation in README.md.

[vsalomaki]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a pre-install step for Kustomize, which is a great feature for handling dependencies like CRDs. The implementation is functional and the documentation updates are clear. However, the new Terraform code in kustomization_user.tf introduces significant code duplication. My main feedback is to refactor this duplicated logic into a reusable local module to improve long-term maintainability. I've also included a few other suggestions to improve robustness and documentation.

[vsalomaki]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant and valuable feature: the ability to define multiple, ordered kustomization sets for pre- and post-install actions. This is achieved by refactoring the existing kustomization logic into new, more flexible Terraform modules (user_kustomizations and user_kustomization_set). The changes are well-structured and the documentation in README.md and kube.tf.example is clear.

My review focuses on improving the robustness of the new implementation. I've identified a critical issue where the deployment script can fail if an optional kustomization directory (like extra-manifests-preinstall) is not present, and an incorrect cleanup command that would also cause failures. The suggested changes will make the new feature more resilient and prevent errors in common use cases.

[vsalomaki]

/gemini review

[gemini-code-assist]

Code Review

This is a great refactoring that introduces a more flexible and powerful way to manage user kustomizations through ordered sets. The use of modules cleans up the root configuration and provides better encapsulation. The changes are well-documented and include helpful examples. I have a few suggestions to improve it further, mainly regarding a potential breaking change in variable types and a possible efficiency improvement.

[vsalomaki]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant and valuable refactoring to support sequential Kustomization deployments, enabling pre-install actions before the main manifests are applied. This is achieved by introducing new Terraform modules (user_kustomizations and user_kustomization_set) that provide a more flexible and powerful way to manage custom deployments. The changes also improve security by marking sensitive variables appropriately.

My review focuses on the correctness and maintainability of this new implementation. I've found a high-severity issue where the default behavior for pre-install steps doesn't pass extra_kustomize_parameters as documented, which could break backward compatibility expectations. I've also identified a couple of medium-severity issues related to a typo in an example and a minor code redundancy that could be improved for better maintainability. Overall, this is a great enhancement, and with these fixes, it will be even more robust.

[vsalomaki]

/gemini summary

[gemini-code-assist]

Summary of Changes

This pull request significantly enhances the Kubernetes cluster bootstrapping process by introducing a flexible, sequential Kustomization deployment mechanism. It refactors existing pre- and post-install Kustomization logic into a modular, configurable system, allowing users to define multiple Kustomization steps with specific ordering and post-deployment actions. This change improves maintainability, addresses potential conflicts, and provides greater control over complex application deployments, particularly for scenarios requiring CRD installation before dependent applications.

Highlights

• Intent: This pull request introduces a 'pre-install Kustomization' capability, allowing for the application of Kubernetes manifests, such as Helm Charts or Custom Resource Definitions (CRDs), before the main Kustomization step. This addresses scenarios where certain components need to be deployed and stable before dependent applications can be installed, providing more control over the deployment order.
• Key Changes: The core change is a significant refactoring of how user-defined Kustomizations are handled. Instead of separate pre- and post-install mechanisms, a new flexible user_kustomizations variable is introduced. This variable allows users to define a sequence of Kustomization 'sets', each with its own source folder, parameters, and post-deployment commands. This refactoring leverages new Terraform modules (user_kustomization_set and user_kustomizations) to manage the copying, templating, and sequential application of these Kustomization sets on the remote host. The README.md and kube.tf.example have been updated to reflect this new approach, including an example for pre-installing external-secrets via Helm. Additionally, sensitive variables (extra_kustomize_deployment_commands, extra_kustomize_parameters, and kustomize_parameters within the new user_kustomizations structure) are now correctly marked as sensitive.
• Reviewer Activity: Initial feedback from gemini-code-assist[bot] highlighted critical issues such as incorrect triggering for pre-install deployments, shared remote directories leading to collisions, and significant code duplication. This led to the major refactoring into reusable Terraform modules. Subsequent reviews focused on refining the new modular structure, addressing issues like kubectl apply failures on empty kustomization sets, ensuring correct data types (map(any)) for parameters, improving cleanup logic, and marking sensitive variables. The author, @vsalomaki, provided clarifications on design choices, such as the intentional replace_triggered_by behavior and the user's responsibility to override kustomize_parameters for pre-install stages.

Changelog

• README.md
  • Updated summary to include 'pre- and post-install actions'.
  • Added a new section 'Pre-install Actions, Example: external-secrets repo and Helm' with detailed instructions and an example for pre-installing Helm Charts.
  • Renamed 'Custom post-install actions' to 'Post-install actions, ArgoCD-example'.
• kube.tf.example
  • Updated comment for extra_kustomize_parameters to mention extra-manifests-preinstall/kustomization.yaml.tpl.
  • Introduced and documented the new user_kustomizations variable, providing an example of how to define sequential Kustomization sets for pre-install and main manifests.
• kustomization_user.tf
  • Removed previous null_resource blocks for user Kustomizations (kustomization_user, kustomization_user_deploy).
  • Implemented local.default_user_kustomize to define a default two-step Kustomization process (pre-install then main manifests).
  • Introduced local.user_kustomize_defaulted to allow overriding default Kustomization sets with var.user_kustomizations.
  • Added local.processed_kustomizes to process and mark kustomize_parameters as sensitive.
  • Integrated a new module "user_kustomizations" to orchestrate the deployment of the defined Kustomization sets.
• modules/user_kustomization_set/
  • New Module: Created a new module to handle the copying and templating of a single Kustomization 'set' to the remote control plane.
  • locals.tf: Defines SHAs for source files, parameters, and post-commands for change detection.
  • main.tf: Manages directory creation, copies templated .yaml files, and a postinstall.sh script to the remote host.
  • out.tf: Exports destination_folder, various SHAs, and files_count (number of template files) for use by parent modules.
  • templates/bash.sh.tpl: A simple bash script template for executing post-commands.
  • variables.tf: Defines input variables including ssh_connection (sensitive), source_folder, destination_folder, template_parameters (sensitive, map(any)), and post_commands_string.
• modules/user_kustomizations/
  • New Module: Created a new module to manage multiple user_kustomization_set instances and orchestrate their sequential deployment.
  • locals.tf: Filters and sorts destination folders for Kustomization sets that contain actual files.
  • main.tf: Instantiates module "user_kustomization_set" for each defined Kustomization, then uses a null_resource to execute a remote bash script that applies each Kustomization sequentially, runs its post-install script, and cleans up the remote directories.
  • variables.tf: Defines ssh_connection (sensitive) and kustomizations_map (a map of Kustomization set configurations with validation for numeric keys, kustomize_parameters is map(any)).
• variables.tf
  • Marked extra_kustomize_deployment_commands and extra_kustomize_parameters as sensitive = true.
  • Added the new user_kustomizations variable (type map(object) with kustomize_parameters as map(any)), allowing for sequential Kustomization deployments.

Activity

• Initial review by gemini-code-assist[bot] identified critical issues regarding pre-install deployment triggering, shared remote directories, and code duplication, recommending module refactoring.
• Subsequent reviews focused on refining the new modular structure, ensuring correct handling of empty Kustomization sets, proper cleanup, and accurate type definitions (map(any)) for parameters.
• Sensitive variables (extra_kustomize_deployment_commands, extra_kustomize_parameters, and kustomize_parameters within user_kustomizations) were correctly marked as sensitive.
• Discussions occurred regarding the intentional nature of replace_triggered_by and the user's explicit control over kustomize_parameters for pre-install stages.

[mysticaltech]

/gemini review

[mysticaltech]

@vsalomaki Appreciate this. If everything checks out, I will keep it for the next major version as it's a breaking changes for people already using kustomization, unless i'm mistaken.

[gemini-code-assist]

Code Review

This pull request introduces a significant and well-executed refactoring to support sequential Kustomization deployments, including pre-install steps. The introduction of user_kustomizations and the new Terraform modules (user_kustomization_set, user_kustomizations) provides a much more flexible and powerful way to manage complex deployment ordering. The code is well-structured, and security considerations like marking sensitive variables have been addressed. I have a couple of suggestions to further improve the flexibility of the new mechanism by allowing steps that only contain commands, which is useful for inserting wait conditions between manifest applications.

[vsalomaki]

@vsalomaki Appreciate this. If everything checks out, I will keep it for the next major version as it's a breaking changes for people already using kustomization, unless i'm mistaken.

Currently it's using the existing variables of extra_kustomize_deployment_commands and extra_kustomize_parameters by default so it doesn't affect existing configurations. But it does become a bit confusing when the default values for the user_kustomizations are retrieved from those values so that when taking this new model into use, one has to move the existing configuration to the new map.

Maybe the best course of action would be to "deprecate" the old variables and remove them later on?

[vsalomaki]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant and well-executed refactoring to support pre-install kustomizations. By replacing the previous null_resource-based implementation with new, modular Terraform modules (user_kustomization_set and user_kustomizations), the changes greatly improve the flexibility, readability, and maintainability of deploying custom manifests. The new user_kustomizations variable provides users with fine-grained control over the deployment sequence, including pre- and post-deployment actions for each step. The documentation and examples have been updated clearly to reflect these powerful new capabilities. My review includes a couple of suggestions to further improve the robustness of the new implementation.

[vsalomaki]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant and well-designed enhancement for deploying user-defined kustomizations. The refactoring into user_kustomization_set and user_kustomizations modules provides a flexible and robust mechanism for sequential deployments, which is a great improvement over the previous implementation. The code is well-structured, and the use of triggers and cleanup traps demonstrates attention to detail.

My review includes a high-severity suggestion to make the deployment script more robust against empty kustomization.yaml files, and a medium-severity suggestion to improve the clarity of an example in the documentation. Overall, this is an excellent contribution.

[gemini-code-assist]

Code Review

This pull request introduces a significant and valuable feature, allowing users to define multiple Kustomization sets that are applied sequentially. This is a great enhancement for managing complex deployments with dependencies, such as installing CRDs before related applications. The implementation is well-structured, refactoring the logic into new, reusable Terraform modules (user_kustomization_set and user_kustomizations). The main deployment script is particularly robust, with good error handling and cleanup mechanisms. My review includes a couple of suggestions to further improve the implementation: one to optimize the file upload process for better efficiency, and another to enhance the robustness of the pre/post command scripts. Overall, this is an excellent contribution that greatly improves the flexibility of the module.

[vsalomaki]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant and well-designed feature for sequential Kustomization deployments. The refactoring into new modules (user_kustomization_set, user_kustomizations) is clean and improves maintainability. The implementation is robust, incorporating good practices like trap for cleanup, set -e for error handling, and marking sensitive variables. The backward compatibility is also handled gracefully. My review includes a few suggestions to further improve robustness and maintainability, primarily by enhancing the kustomization file detection logic, simplifying a local variable definition, and updating the documentation to reflect the new recommended practices.

[vsalomaki]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant and well-designed feature for sequential Kustomization deployments. The refactoring into new modules (user_kustomization_set and user_kustomizations) greatly improves modularity and maintainability. The implementation is robust, with good error handling (set -e), cleanup (trap), and validation. The changes to mark sensitive variables are also a welcome security improvement. My feedback is focused on minor improvements to the documentation and example files to enhance clarity and prevent potential user confusion from copy-pasting. Overall, this is an excellent contribution.

[vsalomaki]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant and valuable enhancement by allowing sequential Kustomization deployments via the new user_kustomizations variable. The refactoring into user_kustomization_set and user_kustomizations modules is clean, improves maintainability, and the implementation is robust, with good use of triggers, cleanup traps, and validation. The documentation updates are also comprehensive. I've found a functional issue related to file discovery for kustomization templates that prevents using filenames like kustomization.yml.tpl, and a minor typo in the documentation. Once these are addressed, this will be an excellent addition.

[mysticaltech]

@vsalomaki Great stuff man! Let's remove it, yes. But also create a small migration guide please (if not already done). About llms.txt, ideally it would be editted by an coding agent like claude code, that would look at the changes from this PR and edit it accordingly. I can do that no worries.

[vsalomaki]

/gemini review

[gemini-code-assist]

Code Review

This is an excellent pull request that significantly improves the flexibility of custom deployments by introducing sequential Kustomization sets. The refactoring into new, dedicated modules (user_kustomization_set and user_kustomizations) is clean and greatly enhances maintainability. The addition of robust error handling, cleanup traps, and comprehensive validation for the new variables is very well done. The migration guide and updated examples are also much appreciated. I've found a few minor inconsistencies and a small gap in the documentation examples, which I've detailed in the comments. Overall, this is a fantastic enhancement to the module.

[vsalomaki]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant and well-executed refactoring to allow for sequential Kustomization deployments via the new user_kustomizations variable. The modular approach is clean, robust, and greatly enhances flexibility for complex scenarios like CRD installations. My review includes a few minor documentation tweaks for clarity and a high-priority security enhancement to ensure that pre_commands and post_commands are treated as sensitive data, preventing potential secret leakage in Terraform plans.

[vsalomaki]

@mysticaltech I added MIGRATION.md that has information on how to update from the previous version

[mysticaltech]

Thanks @vsalomaki !

[mysticaltech]

@codex review please, and let me know if backward compatible or not.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Removing extra_kustomize_ variables breaks existing module inputs*

This change deletes the extra_kustomize_deployment_commands, extra_kustomize_parameters and extra_kustomize_folder variables and replaces them with the new user_kustomizations map. Any consumer that still sets the old variables—either through kube.tf, tfvars, or module inputs—will now fail to plan with “An argument named … is not expected here”. Unless the next release is explicitly a breaking major version with a deprecation cycle, this is not backward compatible with 2.18.x users and will block upgrades until they rename variables. Consider preserving the old variables with deprecation warnings or providing a compatibility wrapper.

Useful? React with 👍 / 👎.

[vsalomaki]

The actions to be made are described in MIGRATION.md

[vsalomaki]

Should we go with the MIGRATION.md, @mysticaltech ?

[vsalomaki]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant and well-designed refactoring of the user kustomization feature. It replaces the old, limited variables with a flexible user_kustomizations map, allowing for the sequential application of multiple kustomization sets. The implementation is robust, using new modules, proper triggers for updates, and security best practices like marking sensitive variables and ensuring cleanup of temporary files on the remote host. The documentation updates are comprehensive and include a helpful migration guide. My review found a couple of minor issues in the documentation examples that could be improved for clarity and correctness, but overall this is an excellent contribution.

[vsalomaki]

/gemini summary

[gemini-code-assist]

Summary of Changes

This pull request significantly enhances the Kubernetes cluster bootstrapping process by introducing a flexible, sequential Kustomization deployment mechanism. It refactors existing pre- and post-install Kustomization logic into a modular, configurable system, allowing users to define multiple Kustomization steps with specific ordering and post-deployment actions. This change improves maintainability, addresses potential conflicts, and provides greater control over complex application deployments, particularly for scenarios requiring CRD installation before dependent applications.

Highlights

• Sequential Kustomization Deployment: Introduced a new user_kustomizations variable that allows defining multiple Kustomization 'sets' to be applied sequentially. This enables complex deployment strategies, such as installing CRDs and waiting for their readiness before deploying dependent applications.
• Modular Refactoring: The previous ad-hoc pre- and post-install Kustomization logic has been refactored into two new, reusable Terraform modules: user_kustomization_set and user_kustomizations. This significantly improves code maintainability, reduces duplication, and centralizes deployment logic.
• Enhanced Flexibility with Pre/Post Commands: Each Kustomization set can now include pre_commands and post_commands to execute arbitrary bash scripts before and after the kubectl apply -k step, offering fine-grained control over the deployment flow and enabling robust wait conditions.
• Improved Deployment Robustness: The remote execution script for applying Kustomizations now includes set -e for error handling, a trap cleanup EXIT for guaranteed cleanup of temporary files, and checks ([ -s "$dest_folder/kustomization.yaml" ]) to prevent kubectl apply -k from failing on empty Kustomization directories.
• Sensitive Variable Handling: Key variables, including kustomize_parameters, pre_commands, and post_commands within the new user_kustomizations structure, are now correctly marked as sensitive = true to prevent their values from being exposed in Terraform plan/apply outputs. The old extra_kustomize_deployment_commands and extra_kustomize_parameters variables were also marked sensitive before removal.
• Comprehensive Documentation and Migration Updates: The README.md and kube.tf.example files have been updated to reflect the new user_kustomizations approach, including detailed explanations and usage examples. A new MIGRATION.md file provides clear instructions for users to transition from the deprecated extra_kustomize_* variables to the new user_kustomizations map.

Changelog

• MIGRATION.md
  • Added a new migration guide detailing the transition from extra_kustomize_* variables to the new user_kustomizations map, including steps and an example configuration.
• README.md
  • Updated the summary to include 'pre- and post-install actions'.
  • Added a new section 'Adding applications with Helm, custom pre- and post-install actions' with detailed instructions and an example for pre-installing Helm Charts (e.g., external-secrets).
  • Renamed 'Custom post-install actions' to 'Post-install actions, ArgoCD-example' and updated its content to use the new user_kustomizations structure.
  • Updated references to extra_kustomize_parameters to user_kustomizations.
• docs/llms.md
  • Updated the Kustomize and Post-Deployment Operations section to document the new user_kustomizations variable and its properties, replacing the old extra_kustomize_deployment_commands and extra_kustomize_parameters documentation.
• examples/kustomization_user_deploy/README.md
  • Added a note about using user_kustomizations for different folders.
  • Updated examples to use user_kustomizations and its kustomize_parameters field instead of extra_kustomize_parameters.
• kube.tf.example
  • Replaced comments for extra_kustomize_deployment_commands and extra_kustomize_parameters with detailed documentation and an example for the new user_kustomizations variable, outlining its properties and usage for sequential deployments.
• kustomization_user.tf
  • Removed previous null_resource blocks (kustomization_user, kustomization_user_deploy) related to the old Kustomization handling.
  • Implemented local.default_user_kustomize to define a default two-step Kustomization process (pre-install then main manifests).
  • Introduced local.user_kustomize_defaulted to allow overriding default Kustomization sets with var.user_kustomizations.
  • Added local.processed_kustomizes to process and mark kustomize_parameters, pre_commands, and post_commands as sensitive.
  • Integrated a new module "user_kustomizations" to orchestrate the deployment of the defined Kustomization sets.
• modules/user_kustomization_set/
  • New Module: Created to handle the copying and templating of a single Kustomization 'set' to the remote control plane.
  • locals.tf: Defines SHAs for source files, parameters, and pre/post commands for change detection.
  • main.tf: Manages directory creation, copies templated .yaml files, and preinstall.sh/postinstall.sh scripts to the remote host, ensuring subdirectories are created.
  • out.tf: Exports destination_folder, various SHAs, files_count (number of template files), and changes_sha for use by parent modules.
  • templates/bash.sh.tpl: A simple bash script template for executing commands, including set -e for error handling.
  • variables.tf: Defines input variables including ssh_connection (sensitive), source_folder, destination_folder, template_parameters (sensitive, map(any)), pre_commands_string (sensitive), and post_commands_string (sensitive).
• modules/user_kustomizations/
  • New Module: Created to manage multiple user_kustomization_set instances and orchestrate their sequential deployment.
  • locals.tf: Filters and sorts destination folders for Kustomization sets that contain actual files or post-commands.
  • main.tf: Instantiates module "user_kustomization_set" for each defined Kustomization, then uses a null_resource to execute a remote bash script that applies each Kustomization sequentially, runs its pre/post-install scripts, and cleans up the remote directories. Includes robust error handling (set -e), guaranteed cleanup (trap cleanup EXIT), and checks for valid kustomization files ([ -s ... ]).
  • variables.tf: Defines ssh_connection (sensitive) and kustomizations_map (a map of Kustomization set configurations with validation for numeric keys, kustomize_parameters as map(any), and pre_commands/post_commands as sensitive).
• variables.tf
  • Removed extra_kustomize_deployment_commands, extra_kustomize_parameters, and extra_kustomize_folder variables.
  • Added the new user_kustomizations variable (type map(object) with kustomize_parameters as map(any), and pre_commands/post_commands as optional strings), allowing for sequential Kustomization deployments. Includes validation to ensure keys are positive numeric strings.

Activity

• The pull request began with an initial bot summary and review, highlighting critical issues such as incorrect triggering for pre-install deployments, shared remote directories, and significant code duplication.
• The author, @vsalomaki, actively engaged with the feedback, triggering multiple bot reviews and initiating a major refactoring of the Kustomization logic into reusable Terraform modules.
• Subsequent reviews focused on refining the new modular structure, addressing issues like kubectl apply failures on empty kustomization sets, ensuring correct data types (map(any)) for parameters, improving cleanup logic, and marking sensitive variables.
• Discussions occurred regarding the intentional replace_triggered_by behavior and the user's explicit control over kustomize_parameters for pre-install stages.
• Further refinements were made to the remote execution script to enhance robustness, including the addition of set -e for error handling, trap cleanup EXIT for guaranteed cleanup, and a check for non-empty kustomization.yaml files ([ -s ... ]) before applying.
• The validation logic for kustomizations_map keys was improved to prevent Terraform errors on non-numeric keys.
• The explanation for the || exit 0 pattern in post_commands examples was clarified for better user understanding.
• A chatgpt-codex-connector[bot] review identified a critical issue with subdirectory creation for nested Kustomization templates, which was subsequently fixed by the author.
• Discussions with @mysticaltech confirmed the intention to remove the old extra_kustomize_* variables and the need for a migration guide, which was then added by @vsalomaki in MIGRATION.md.