CLOUDP-341025: Refactor of existing snippets on master (#366)

# Summary

This PR refactors how we generate and manage outputs from snippet tests.

Before, every snippet module has its own output/ directory. But because
snippet modules were reused across different tests we had no way of
storing the outputs from different test runs (the files were
overwritten).

Changes were needed to accommodate upcoming Search snippets, but due to
the large scale of this refactoring it was extracted onto master.

**Only changes to public/architectures has been made performed in this
PR. Existing search snippets are left intact and will be refactored on
the search feature branch**. After this PR is approved, a followup PR in
the docs repo will be created adjusting for the new include paths.

This PR incorporated changes from #150 to fix diagnostic dumps from GKE
clusters.

# What this change will help
* committing back snippet output will be easy as adding changes from one
directory `scripts/code_snippets/tests/outputs/test_*` instead of
traversing the tree and dealing with the conflicts pushed from different
test runs.
* more clarity what snippets and outputs we're using in docs
* ability to share snippet modules and gather different outputs from the
same snippet module for different testing scenarios.

## Glossary
* `Snippet module` - directory containing `test.sh` and scripts in a
`code_snippets` sub directory. It's a reusable set of scripts, e.g.
`public/architectures/setup-multi-cluster/ra-01-setup-gke`.
* Snippet test - automated test running in evg that runs multiple
snippet modules. It's a top level entrypoint that runs all the snippet
modules. Examples:
`scripts/code_snippets/tests/test_gke_multi_cluster_no_mesh_snippets.sh`,
`scripts/code_snippets/test_gke_multi_cluster_snippets.sh`. 
* Previously we named these files a "task" and they were directly in
`scripts/code_snippets` dir (e.g.
`scripts/code_snippets/task_gke_multi_cluster_no_mesh_snippets_test.sh`).
* `Snippet outputs` - output files (.out) gathered from a `Snippet
test`.

## Changes 
* Snippet modules don't store their outputs in the snippet directory
now, but in `scripts/code_snippets/tests/outputs/<test name>` directory,
which is gathering *all* outputs from a given snippet test.
* Every file name in any of the code_snippets directories in the
codebase must have a globally unique name. Existing file names has been
refactored to match the new convention (in public/architecture).
* Validation/linter has been added
(`scripts/code_snippets/validate_snippets.py`) added to `pre-commit`
that enforces this convention.
*  The naming convention is the following: 
* Snippet module directory must be prefixed by some arbitrary prefix.
Snippet modules in reference architectures has been prefixed with
"ra-NN`, e.g. `ra-02-setup-operator`. Every snippet file belonging to
this module should be also prefixed (e.g.
`ra-02_0046_create_image_pull_secrets.sh`)
* This allows for storing all the snippets outputs in a flat directory,
guaranteeing we won't overwrite outputs.
* This also helps quickly verifying the paths that will be used in the
docs to include snippets and outputs. Having those prefixes will help
identifying if any included file is coming from a correct snippet test
and module.
* Improved diagnostic gathering from the test runs
     * GKE clusters are correctly handled now
* Every file name is prefixed with the kube context name and the
namespace from which the files were downloaded. *This will impact all
the e2e test runs*, but it will improve diagnostic archive in cases
where we dump more than one namespace or k8s cluster.
* All evergreen yamls related to snippets have been moved into
.evergreen-snippets.yml (included from the main .evergreen-snippets.yml)

Here is the fragment of the current directory structure: 
```
public/architectures
├── ra-06-ops-manager-multi-cluster
│   ├── code_snippets
│   │   ├── ra-06_0250_generate_certs.sh
│   │   ├── ra-06_0300_ops_manager_create_admin_credentials.sh
            [...]
│   │   ├── ra-06_9100_delete_backup_namespaces.sh
│   │   └── ra-06_9200_delete_om.sh
│   ├── env_variables.sh
│   ├── teardown.sh
│   └── test.sh
├── ra-07-mongodb-replicaset-multi-cluster
│   ├── code_snippets
│   │   ├── ra-07_1050_generate_certs.sh
│   │   ├── ra-07_1100_mongodb_replicaset_multi_cluster.sh
            [...]
│   │   └── ra-07_9000_delete_resources.sh
│   ├── env_variables.sh
│   ├── teardown.sh
│   └── test.sh
```

```
scripts/code_snippets
├── tests
│   ├── outputs
│   │   ├── test_gke_multi_cluster_no_mesh_snippets
│   │   ├── test_gke_multi_cluster_snippets
│   │   │   ├── ra-01_0030_verify_access_to_clusters.out
│   │   │   ├── ra-02_0200_kubectl_mongodb_configure_multi_cluster.out
│   │   │   ├── ra-02_0205_helm_configure_repo.out
            [...]
│   │   │   ├── ra-06_0321_ops_manager_wait_for_pending_state.out
│   │   │   ├── ra-06_0322_ops_manager_wait_for_running_state.out
│   │   │   └── ra-06_0522_ops_manager_wait_for_running_state.out
│   │   └── test_kind_search_community_snippets
│   │       ├── 01_0090_helm_add_mogodb_repo.out
│   │       ├── 01_0100_install_operator.out
            [...]
│   │       ├── 03_0450_execute_search_query.out
│   │       └── 03_0455_execute_vector_search_query.out
│   ├── test_gke_multi_cluster_no_mesh_snippets.sh
│   ├── test_gke_multi_cluster_snippets.sh
│   └── test_kind_search_community_snippets.sh

```

## Proof of Work

[Green EVG
run](https://spruce.mongodb.com/version/68ae250c41ef2800073ac896/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC)
for all snippets.

## Checklist

- [ ] Have you linked a jira ticket and/or is the ticket in the title?
- [ ] Have you checked whether your jira ticket required DOCSP changes?
- [ ] Have you added changelog file?
    - use `skip-changelog` label if not needed
- refer to [Changelog files and Release
Notes](https://github.com/mongodb/mongodb-kubernetes/blob/master/CONTRIBUTING.md#changelog-files-and-release-notes)
section in CONTRIBUTING.md for more details

Author: lsierant

.evergreen-functions.yml

@@ -475,33 +475,6 @@ functions:
       params:
         files: [ "src/github.com/mongodb/mongodb-kubernetes/logs/*.suite" ]
 
-  upload_code_snippets_logs:
-    - command: s3.put
-      params:
-        aws_key: ${enterprise_aws_access_key_id}
-        aws_secret: ${enterprise_aws_secret_access_key}
-        local_files_include_filter:
-          - src/github.com/mongodb/mongodb-kubernetes/public/architectures/**/*.log
-          - src/github.com/mongodb/mongodb-kubernetes/docs/**/*.log
-        remote_file: logs/${task_id}/${execution}/
-        bucket: operator-e2e-artifacts
-        permissions: private
-        visibility: signed
-        content_type: text/plain
-    - command: s3.put
-      params:
-        aws_key: ${enterprise_aws_access_key_id}
-        aws_secret: ${enterprise_aws_secret_access_key}
-        local_files_include_filter:
-          - src/github.com/mongodb/mongodb-kubernetes/public/architectures/**/*.out
-          - src/github.com/mongodb/mongodb-kubernetes/docs/**/*.out
-        preserve_path: true
-        remote_file: logs/${task_id}/${execution}/
-        bucket: operator-e2e-artifacts
-        permissions: private
-        visibility: signed
-        content_type: text/plain
-
   preflight_image:
     - *switch_context
     - command: subprocess.exec
@@ -817,7 +790,7 @@ functions:
         working_dir: src/github.com/mongodb/mongodb-kubernetes
         binary: scripts/code_snippets/sample_commit_output.sh
 
-  # it executes a script by convention: ./scripts/code_snippets/${task_name}_test.sh
+  # it executes a script by convention: ./scripts/code_snippets/tests/${task_name}
   test_code_snippets:
     - *switch_context
     - command: shell.exec
@@ -830,7 +803,7 @@ functions:
           - code_snippets_reset
           - task_name
         script: |
-          ./scripts/code_snippets/${task_name}_test.sh
+          ./scripts/code_snippets/tests/${task_name}
 
   #
   # kubectl mongodb plugin release functions

.evergreen-release.yml

@@ -151,19 +151,6 @@ buildvariants:
     tasks:
       - name: run_conditionally_prepare_and_upload_openshift_bundles
 
-  - name: prerelease_gke_code_snippets
-    display_name: prerelease_gke_code_snippets
-    tags: [ "release" ]
-    allowed_requesters: ["patch", "github_tag"]
-    depends_on:
-      - variant: release_images
-        name: '*'
-        patch_optional: true
-    run_on:
-      - ubuntu2404-small
-    tasks:
-      - name: gke_code_snippets_task_group
-
   - name: init_smoke_tests
     display_name: init_smoke_tests
     tags: [ "e2e_smoke_release_test_suite" ]

.evergreen-snippets.yml

@@ -0,0 +1,185 @@
+variables:
+  - &setup_and_teardown_group_gke_code_snippets
+    setup_task_can_fail_task: true
+    setup_group:
+      - func: clone
+      - func: setup_gcloud_cli
+      - func: setup_mongosh
+      - func: download_kube_tools
+      - func: build_multi_cluster_binary
+    teardown_task:
+      - func: upload_e2e_logs
+      - func: upload_code_snippets_logs
+      - func: upload_code_snippets_outputs
+
+  - &setup_and_teardown_group_kind_code_snippets
+    setup_task_can_fail_task: true
+    setup_group:
+      - func: clone
+      - func: cleanup_exec_environment
+      - func: download_kube_tools
+      - func: configure_docker_auth
+      - func: setup_kubernetes_environment
+    teardown_task:
+      - func: upload_e2e_logs
+      - func: upload_code_snippets_logs
+      - func: upload_code_snippets_outputs
+
+# This variable is copied over from .evergreen.yml because anchors don't work for included files
+  - &base_om8_dependency
+    depends_on:
+      - name: build_om_images
+        variant: build_om80_images
+      - name: build_operator_ubi
+        variant: init_test_run
+      - name: build_init_database_image_ubi
+        variant: init_test_run
+      - name: build_database_image_ubi
+        variant: init_test_run
+      - name: build_test_image
+        variant: init_test_run
+      - name: build_init_appdb_images_ubi
+        variant: init_test_run
+      - name: build_init_om_images_ubi
+        variant: init_test_run
+      - name: build_agent_images_ubi
+        variant: init_test_run
+
+functions:
+  upload_code_snippets_logs:
+    - command: s3.put
+      params:
+        aws_key: ${enterprise_aws_access_key_id}
+        aws_secret: ${enterprise_aws_secret_access_key}
+        local_files_include_filter:
+          - src/github.com/mongodb/mongodb-kubernetes/public/architectures/**/*.log
+          - src/github.com/mongodb/mongodb-kubernetes/docs/**/*.log
+        remote_file: logs/${task_id}/${execution}/
+        bucket: operator-e2e-artifacts
+        permissions: private
+        visibility: signed
+        content_type: text/plain
+    - command: s3.put
+      params:
+        aws_key: ${enterprise_aws_access_key_id}
+        aws_secret: ${enterprise_aws_secret_access_key}
+        local_files_include_filter:
+          - src/github.com/mongodb/mongodb-kubernetes/public/architectures/**/*.out
+          - src/github.com/mongodb/mongodb-kubernetes/docs/**/*.out
+        preserve_path: true
+        remote_file: logs/${task_id}/${execution}/
+        bucket: operator-e2e-artifacts
+        permissions: private
+        visibility: signed
+        content_type: text/plain
+
+  upload_code_snippets_outputs:
+    - command: s3.put
+      params:
+        aws_key: ${enterprise_aws_access_key_id}
+        aws_secret: ${enterprise_aws_secret_access_key}
+        local_files_include_filter:
+          - snippets_outputs.tgz
+        remote_file: logs/${task_id}/${execution}/
+        bucket: operator-e2e-artifacts
+        permissions: private
+        visibility: signed
+        content_type: ${content_type|application/x-gzip}
+        display_name: "Snippets Outputs"
+
+tasks:
+  # Code snippets tasks
+  # Each scripts/code_snippets/tests/test_*.sh should have its task defined here.
+  # TODO: it should be autogenerated
+  # Each task executes test_code_snippets functon executes scripts/code_snippets/tests/${task_name} by task name convention
+  - name: test_gke_multi_cluster_snippets.sh
+    tags: [ "code_snippets" ]
+    commands:
+      - func: test_code_snippets
+      - func: sample_commit_output
+
+  - name: test_gke_multi_cluster_no_mesh_snippets.sh
+    tags: [ "code_snippets" ]
+    commands:
+      - func: test_code_snippets
+      - func: sample_commit_output
+
+  - name: test_kind_search_community_snippets.sh
+    tags: [ "code_snippets", "patch-run" ]
+    commands:
+      - func: test_code_snippets
+      - func: sample_commit_output
+
+task_groups:
+  - name: gke_code_snippets_task_group
+    <<: *setup_and_teardown_group_gke_code_snippets
+    max_hosts: -1
+    tasks:
+      - test_gke_multi_cluster_snippets.sh
+      - test_gke_multi_cluster_no_mesh_snippets.sh
+
+  - name: kind_code_snippets_task_group
+    <<: *setup_and_teardown_group_kind_code_snippets
+    max_hosts: -1
+    tasks:
+      - test_kind_search_community_snippets.sh
+
+buildvariants:
+  # These variants are used to test the code snippets and each one can be used in patches
+  # Prerelease is especially used when the repo is tagged
+  # More details in the TD: https://docs.google.com/document/d/1fuTxfRtP8QPtn7sKYxQM_AGcD6xycTZH8svngGxyKhc/edit?tab=t.0#bookmark=id.e8uva0393mbe
+  - name: public_gke_code_snippets
+    display_name: public_gke_code_snippets
+    allowed_requesters: [ "patch" ]
+    run_on:
+      - ubuntu2204-small
+    tasks:
+      - name: gke_code_snippets_task_group
+
+  - name: prerelease_gke_code_snippets
+    display_name: prerelease_gke_code_snippets
+    tags: [ "release" ]
+    allowed_requesters: [ "patch", "github_tag" ]
+    depends_on:
+      - variant: release_images
+        name: '*'
+        patch_optional: true
+    run_on:
+      - ubuntu2204-small
+    tasks:
+      - name: gke_code_snippets_task_group
+
+  - name: private_gke_code_snippets
+    display_name: private_gke_code_snippets
+    allowed_requesters: [ "patch" ]
+    run_on:
+      - ubuntu2204-small
+    <<: *base_om8_dependency
+    tasks:
+      - name: gke_code_snippets_task_group
+
+  - name: private_kind_code_snippets
+    display_name: private_kind_code_snippets
+    tags: [ "e2e_test_suite" ]
+    allowed_requesters: [ "patch", "github_pr" ]
+    run_on:
+      - ubuntu2204-large
+    <<: *base_om8_dependency
+    tasks:
+      - name: kind_code_snippets_task_group
+
+  - name: prerelease_kind_code_snippets
+    display_name: prerelease_kind_code_snippets
+    allowed_requesters: [ "patch" ]
+    run_on:
+      - ubuntu2204-large
+    tasks:
+      - name: kind_code_snippets_task_group
+
+  - name: public_kind_code_snippets
+    display_name: public_kind_code_snippets
+    allowed_requesters: [ "patch" ]
+    run_on:
+      - ubuntu2204-large
+    tasks:
+      - name: kind_code_snippets_task_group

.evergreen.yml

@@ -5,6 +5,7 @@ include:
   - filename: .evergreen-functions.yml
   - filename: .evergreen-tasks.yml
   - filename: .evergreen-mco.yml
+  - filename: .evergreen-snippets.yml
   - filename: .evergreen-release.yml
 
 variables:
@@ -87,29 +88,6 @@ variables:
       - func: setup_building_host
       - func: build_multi_cluster_binary
 
-  - &setup_and_teardown_group_gke_code_snippets
-    setup_task_can_fail_task: true
-    setup_group:
-      - func: clone
-      - func: setup_gcloud_cli
-      - func: setup_mongosh
-      - func: download_kube_tools
-      - func: build_multi_cluster_binary
-    teardown_group:
-      - func: upload_code_snippets_logs
-
-  - &setup_and_teardown_group_kind_code_snippets
-    setup_task_can_fail_task: true
-    setup_group:
-      - func: clone
-      - func: cleanup_exec_environment
-      - func: download_kube_tools
-      - func: configure_docker_auth
-      - func: setup_kubernetes_environment
-    teardown_task:
-      - func: upload_e2e_logs
-      - func: upload_code_snippets_logs
-
   - &setup_and_teardown_task_cloudqa
     setup_task_can_fail_task: true
     setup_task:
@@ -173,6 +151,7 @@ variables:
       - name: build_init_om_images_ubi
         variant: init_test_run
 
+  # Any change to base_om8_dependency should be reflected to its copy in .evergreen-snippets.yml
   - &base_om8_dependency
     depends_on:
       - name: build_om_images
@@ -580,19 +559,6 @@ task_groups:
       - unit_tests_helm
       - sbom_tests
 
-  - name: gke_code_snippets_task_group
-    <<: *setup_and_teardown_group_gke_code_snippets
-    max_hosts: -1
-    tasks:
-      - task_gke_multi_cluster_snippets
-      - task_gke_multi_cluster_no_mesh_snippets
-
-  - name: kind_code_snippets_task_group
-    <<: *setup_and_teardown_group_kind_code_snippets
-    max_hosts: -1
-    tasks:
-      - task_kind_community_search_snippets
-
   # Task group for deploying mongodbcommunity resources and testing the (former) MCO
   - name: e2e_mdb_community_task_group
     max_hosts: -1
@@ -1827,52 +1793,6 @@ buildvariants:
     tasks:
       - name: release_all_currently_used_agents_on_ecr
 
-    # These variants are used to test the code snippets and each one can be used in patches
-    # Prerelease is especially used when the repo is tagged
-    # More details in the TD: https://docs.google.com/document/d/1fuTxfRtP8QPtn7sKYxQM_AGcD6xycTZH8svngGxyKhc/edit?tab=t.0#bookmark=id.e8uva0393mbe
-  - name: public_gke_code_snippets
-    display_name: public_gke_code_snippets
-    allowed_requesters: [ "patch" ]
-    run_on:
-      - ubuntu2404-small
-    tasks:
-      - name: gke_code_snippets_task_group
-
-  - name: private_gke_code_snippets
-    display_name: private_gke_code_snippets
-    allowed_requesters: [ "patch" ]
-    run_on:
-      - ubuntu2404-small
-    <<: *base_om8_dependency
-    tasks:
-      - name: gke_code_snippets_task_group
-
-  - name: private_kind_code_snippets
-    display_name: private_kind_code_snippets
-    tags: [ "e2e_test_suite" ]
-    allowed_requesters: [ "patch", "github_pr" ]
-    run_on:
-      - ubuntu2404-large
-    <<: *base_om8_dependency
-    tasks:
-      - name: kind_code_snippets_task_group
-
-  - name: prerelease_kind_code_snippets
-    display_name: prerelease_kind_code_snippets
-    allowed_requesters: [ "patch" ]
-    run_on:
-      - ubuntu2404-large
-    tasks:
-      - name: kind_code_snippets_task_group
-
-  - name: public_kind_code_snippets
-    display_name: public_kind_code_snippets
-    allowed_requesters: [ "patch" ]
-    run_on:
-      - ubuntu2404-large
-    tasks:
-      - name: kind_code_snippets_task_group
-
   ### Build variants for manual patch only
 
   - name: backup_csv_images

.githooks/pre-commit

@@ -163,6 +163,10 @@ lint_code() {
   scripts/evergreen/lint_code.sh
 }
 
+function validate_snippets() {
+  scripts/code_snippets/validate_snippets.py
+}
+
 # bg_job_ vars are global; run_job_in_background function is appending to them on each call
 bg_job_pids=()
 bg_job_pids_with_names=()
@@ -217,6 +221,7 @@ pre_commit() {
   run_job_in_background "regenerate_public_rbac_multi_cluster"
   run_job_in_background "python_formatting"
   run_job_in_background "check_erroneous_kubebuilder_annotations"
+  run_job_in_background "validate_snippets"
 
   if wait_for_all_background_jobs; then
     echo -e "${GREEN}pre-commit: All checks passed!${NO_COLOR}"