feat: GitHub actions documentation

docs/content/contributing/bicep/ci-environment/_index.md

@@ -0,0 +1,22 @@
+---
+title: Bicep CI environment
+linktitle: CI # breadcrumb
+description: Continuous Integration (CI) pipelines are essential for maintaining the quality and reliability of Azure Verified Modules.
+---
+
+Continuous Integration (CI) pipelines are essential for maintaining the quality and reliability of Azure Verified Modules. These pipelines automate the process of integrating code changes from multiple contributors and running tests. By automating these tasks, CI pipelines help to catch errors early, ensure that code changes do not introduce new issues, maintain a consistent codebase and ensuring idempotency for deployments.
+
+Using CI pipelines for Azure Verified Modules also facilitates collaboration among developers. Automated testing and integration reduce the time and effort required for manual code reviews and testing, allowing developers to focus on writing high-quality code. Additionally, CI pipelines provide immediate feedback on code changes, enabling developers to address issues promptly. This continuous feedback loop helps to improve the overall quality of the modules and ensures that they meet the rigorous standards expected by Azure users.
+
+The Continuous Integration environment (CI environment) is a set of automation components that are used for continuously validating and publishing module artifacts (Bicep registry repositories, template specs, universal packages). Technically, the CI environment consists of a DevOps platform hosting related pipelines and scripts, as well as an Azure environment (Azure AD tenant with management group(s) and subscription(s)) in which the modules are validated by the automation pipelines and scripts.
+
+- [Pipeline design]({{% siteparam base %}}/contributing/bicep/ci-environment/pipeline-design)
+  - [Static validation]({{% siteparam base %}}/contributing/bicep/ci-environment/static-validation)
+  - [Deployment Flow]({{% siteparam base %}}/contributing/bicep/ci-environment/deployment-flow)
+    - [Deployment validation]({{% siteparam base %}}/contributing/bicep/ci-environment/deployment-validation)
+    - [Deployment history cleanup]({{% siteparam base %}}/contributing/bicep/ci-environment/deployment-history-cleanup)
+  - [Publishing]({{% siteparam base %}}/contributing/bicep/ci-environment/publishing)
+  - [Token replacement]({{% siteparam base %}}/contributing/bicep/ci-environment/token-replacement)
+- [Pipeline usage]({{% siteparam base %}}/contributing/bicep/ci-environment/pipeline-usage)
+- [Bicep configuration]({{% siteparam base %}}/contributing/bicep/ci-environment/bicep-configuration)
+- [Troubleshooting]({{% siteparam base %}}/contributing/bicep/ci-environment/troubleshooting)

docs/content/contributing/bicep/ci-environment/bicep-configuration.md

@@ -0,0 +1,28 @@
+---
+title: CI environment - Bicep Configuration
+---
+
+This section provides details on the Bicep configuration used in the CARML CI environment. This configuration happens in two places, the Bicep Configuration File (`bicepconfig.json`) and inside the modules themselves.
+
+---
+
+## Bicep Configuration File
+
+Using this file, you can customize your Bicep development experience. This includes
+
+- Linter Rules (e.g., [max parameters](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/linter-rule-max-parameters))
+- Source locations (e.g., [aliases](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-config-modules))
+
+For a full list of available rules, please refer to the [official documentation](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-config).
+
+> If you remove this file, Bicep uses default values
+
+The configuration applied in the CI environment can be found in the `bicepconfig.json` file in the root folder. Next to the corresponding settings you will also find the rational for its application.
+
+## Module-level Bicep configuration
+
+The Bicep DSL (Domain Specific Language) is continuously improved and extended with additional capabilities. Of of them, the Bicep Linter, provides guidance around template design & best practices - and surfaces any findings as warnings. However, while it is a great feature, there can be cases where rules show false-positives - or are not addressed by us immediately. For these cases, we occasionally apply ignore tags such as `#disable-next-line secure-secrets-in-params` on a module level.
+
+> Note: Each ignore tag should be accompanied by a comment to justify its existence.
+
+> Note: As we want to follow best-practices whenever we can, the ignore tags should only be applied when absolutely necessary, on a case-by-case basis.

docs/content/contributing/bicep/ci-environment/ci-environment-pipeline-design.md

@@ -0,0 +1,131 @@
+---
+title: The CI environment Pipeline design
+---
+
+## Module Pipelines
+
+The repository hosts one pipeline for each module in the AVM library.
+
+The purpose of each module pipeline is twofold:
+
+1. **Validation**: To ensure the modules hosted by the AVM library are valid and can perform the intended deployments.
+2. **Publishing**: To publish versioned and already validated modules to one or multiple target locations, from where they can be referenced by solutions consuming them.
+
+As such, each pipeline can be mapped to Phases 1 and 2 described in the Deployment flow section.
+
+![Pipeline Phases](/Azure-Verified-Modules/images/bicep-ci/pipeline-design.png)
+
+The following paragraphs provide an overview of the different phases and shared logic the module pipelines use.
+
+## Pipeline phases
+
+This paragraph provides an overview of the three phases performed by each module pipeline. Further details about the implementation and design of each phase are provided on the dedicated pages linked below.
+
+1. **Static Validation**: Runs a set of static Pester tests on the module and its templates to ensure they comply with the design principles. Further details for this phase are provided on the corresponding wiki page - see the [Static validation](/Azure-Verified-Modules/contributing/bicep/ci-environment/static-validation) section.
+1. **Deployment Validation**: An actual Azure deployment is run in a sandbox subscription leveraging a predefined set of module test files, each validating a different configuration of the same Azure resource in parallel. The test suite is cleaned up by default, removing all test resources post-deployment. Further details for this phase are provided on the corresponding wiki page - see the [Deployment validation](/Azure-Verified-Modules/contributing/bicep/ci-environment/deployment-validation) section.
+1. **Publishing**: Runs only if the previous steps are successful. A new module version is published to all configured target locations such as template specs, private Bicep registry and Azure DevOps Universal Packages. Published module versions can then be referenced by solutions using them. Further details for this phase are provided on the corresponding wiki page - see the [Publishing](/Azure-Verified-Modules/contributing/bicep/ci-environment/publishing) page.
+
+![Pipeline Design Phases](/Azure-Verified-Modules/images/bicep-ci/pipeline-design-phases.png)
+
+### GitHub-specific design
+
+![Pipeline Phases GitHub](/Azure-Verified-Modules/images/bicep-ci/pipeline-phases-github.png)
+
+GitHub workflows map each pipeline phase to a dedicated composite action, to maximize code reusability.
+The mapping to the specific composite action is provided below:
+
+| Composite Action | Pipeline phase |
+| - | - |
+| **Initialize pipeline** | Prepare the pipeline by (for example) publishing pipeline variables, collecting test files, etc. |
+| **Module / Static validation** | Static validation |
+| **Module / PSRule validation** | PS-Rule validation (with one job per template test file) |
+| **Module / Deployment validation** | Deployment validation (with one job per template test file) |
+| **publishModule** | Publishing |
+
+// TODO the next section.
+// is it this?
+    outputs:
+      workflowInput: ${{ steps.get-workflow-param.outputs.workflowInput }}
+      moduleTestFilePaths: ${{ steps.get-module-test-file-paths.outputs.moduleTestFilePaths }}
+      psRuleModuleTestFilePaths: ${{ steps.get-module-test-file-paths.outputs.psRuleModuleTestFilePaths }}
+      modulePath: "${{ env.modulePath }}"
+// end todo
+In addition, workflows leverage the following composite actions:
+
+| Composite Action | Description |
+| - | - |
+| **getWorkflowInput** | This action allows fetching workflow input values from the module's workflow file, even if the pipeline was not triggered via a `workflow_dispatch` action. Without it, we would not be able to process the contained information and would need to duplicate the configuration as workflow variables. Such input values are for example, the removal switch `removeDeployment`. |
+| **setEnvironment** |  This action sets up GitHub runners with relevant PowerShell modules required for specific jobs. It then parses the settings file ([`settings.yml`](https://github.com/Azure/ResourceModules/blob/main/settings.yml)) and sets the key-value pairs in the `variables` list as environment variables. |
+
+Technical documentation for each composite action, such as required input and output variables, is included in each `action.yml` file located in path `.github/actions/templates`.
+
+## Module pipeline inputs
+
+Each module pipeline comes with the following runtime parameters:
+
+- `'Branch'` dropdown: A dropdown to select the branch to run the pipeline from.
+- `'Execute static validation'` switch: Can be enabled or disabled. It controls whether the static tests jobs are executed during the pipeline run.
+  > **Note:** This switch cannot be used to bypass the publishing requirements, where both static tests & deployment validation jobs must be successful.
+- `'Execute deployment validation'` switch: Can be enabled or disabled. It controls whether the deployment validation jobs are executed during the pipeline run.
+- `'Remove deployed module'` switch: Can be enabled or disabled. It controls whether the test-deployed resources are removed after the deployment validation jobs. It is enabled by default.
+  > **Note:** This switch cannot be used to bypass the publishing requirements, where both the static tests & deployment validation jobs must be successful.
+- `'Default location overwrite'` string: By default, a region is choosen randomly, that the resources are deployed into. This parameter let's you overwrite a random region with one you specify (e.g. eastus).
+
+![Module Pipeline Input](/Azure-Verified-Modules/images/bicep-ci/module-pipeline-input.png)
+
+---
+
+## Platform pipelines
+
+In addition to module pipelines, the repository includes several platform pipelines covering further tasks as described below.
+
+- [PSRule Pre-Flight validation pipeline](#psrule-pre-flight-validation-pipeline)
+- [Deployment history cleanup](/Azure-Verified-Modules/contributing/bicep/ci-environment/deployment-history-cleanup)
+// TODO add missing pipelines
+
+## PSRule Pre-Flight validation pipeline
+
+The purpose of the PSRule Pre-Flight validation pipeline is to validate Azure resources deployed by module validation pipeline tests, by leveraging [PSRule for Azure](https://azure.github.io/PSRule.Rules.Azure/about/).
+PSRule for Azure is aligned to the [Well-Architected Framework (WAF)](https://learn.microsoft.com/en-us/azure/architecture/framework/). Tests, called _Rules_, check the configuration of Azure resources against WAF principles.
+
+The pipeline, currently only available as a [GitHub workflow](https://github.com/Azure/bicep-registry-modules/blob/main/.github/workflows/platform.check.psrule.yml), runs weekly on the whole library, providing as output the list of non-compliant resources and corresponding failing rules, if any.
+
+### Configuration settings
+
+PSRule options set for the AVM repository are configured in the [ps-rule.yaml](https://github.com/Azure/bicep-registry-modules/blob/main/utilities/pipelines/staticValidation/psrule/ps-rule.yaml) file.
+
+Documentation for all configuration options is available at the following links:
+
+- [https://aka.ms/ps-rule/options](https://aka.ms/ps-rule/options)
+- [https://aka.ms/ps-rule-azure/options](https://aka.ms/ps-rule-azure/options)
+
+### Baselines
+
+A [baseline](https://azure.github.io/PSRule.Rules.Azure/working-with-baselines/) is a standard PSRule artifact that combines rules and configuration. The PSRule Pre-Flight validation pipeline uses the default baseline to analyze module test resources.
+
+For the list of all rules included see [Azure.Default baseline](https://azure.github.io/PSRule.Rules.Azure/en/baselines/Azure.Default/).
+To view a list of rules by Azure resources see [Rules by resource](https://azure.github.io/PSRule.Rules.Azure/en/rules/resource/).
+
+### Exclusions and suppression rules
+
+Not all baseline rules may be valid for some of the test Azure resources deployed by the module validation pipelines.
+
+For example, resources deployed by the min tests, aim to validate only the required input parameters for each module.
+Therefore, optional features such as diagnostic settings are not configured in those tests. Since enabling logging is a general recommendation for most of the resources supporting them, missing diagnostic settings usually trigger incopliance of PSRule checks, e.g., [Azure.KeyVault.Logs](https://azure.github.io/PSRule.Rules.Azure/en/rules/Azure.KeyVault.Logs/). For this reason, these checks are excluded from being evaluated for resources deployed by min tests.
+
+PSRule allows skipping rules on two levels:
+
+- **Exclusions**: Can be leveraged to exclude specific baseline rules from being evaluated for any resource.
+  - [ps-rule.yaml](https://github.com/Azure/ResourceModules/blob/main/ps-rule.yaml): Lists the name of specific rules to exclude under the option [Rule.Exclude](https://microsoft.github.io/PSRule/v2/concepts/PSRule/en-US/about_PSRule_Options/#ruleexclude)
+- **Suppression Groups**: PSRule can use [Suppression Groups](https://microsoft.github.io/PSRule/v2/concepts/PSRule/en-US/about_PSRule_SuppressionGroups/) to suppress rules based on a condition. Suppression groups can be leveraged when some of the rules in the baseline are not relevant under specific conditions, e.g., only for specific resources. They are stored in the `.ps-rule` repo folder in `.yaml` format. In particular:
+  - [cb-waf-security.Rule.yaml](https://github.com/Azure/bicep-registry-modules/blob/main/utilities/pipelines/staticValidation/psrule/.ps-rule/cb-waf-security.Rule.yaml): Defines the module-list for the WAF Security recommendations
+  - [dep-suppress.Rule.yaml](https://github.com/Azure/bicep-registry-modules/blob/main/utilities/pipelines/staticValidation/psrule/.ps-rule/dep-suppress.Rule.yaml): Lists rules to be ignored for resources deployed as dependencies
+  - [min-suppress.Rule.yaml](https://github.com/Azure/bicep-registry-modules/blob/main/utilities/pipelines/staticValidation/psrule/.ps-rule/min-suppress.Rule.yaml): Lists rules to be ignored for resources deployed by the min tests
+  - [na-suppress.Rule.yaml](https://github.com/Azure/bicep-registry-modules/blob/main/utilities/pipelines/staticValidation/psrule/.ps-rule/na-suppress.Rule.yaml): Lists rules to be ignored for resources not supporting Tags
+  - [storage-firewall-suppress.Rule.yaml](https://github.com/Azure/bicep-registry-modules/blob/main/utilities/pipelines/staticValidation/psrule/.ps-rule/storage-firewall-suppress.Rule.yaml): Surpress Rules for resources, that can't activate firewalls
+
+### Output
+
+To better outline failed rules and allow fixing incompliant resources quickly, the pipeline leverages the script [utilities\pipelines\PSRulestaticValidation\psrule\Set-PSRuleGitHubOutput.ps1](https://github.com/Azure/bicep-registry-modules/blob/main/utilities/pipelines/staticValidation/psrule/Set-PSRuleGitHubOutput.ps1) to aggregate PSRule output into Custom Markdown content and display it to the Actions run summary page.
+
+![PSRule Summary](/Azure-Verified-Modules/images/bicep-ci/psrule-summary.png)