Adding manual job to run entire test suite

Author: Vishal Patel

doc/development/allure-test-report.md

@@ -4,7 +4,7 @@ The following three pipelines are created while generating the allure-report
 
 - Omnibus pipeline
 - TRIGGERED_CE/EE_PIPELINE child pipeline (Manually Triggered)
-- QA-TEST child pipeline
+- QA-SUBSET-TEST child pipeline
 
 ## Omnibus MR Pipeline
 
@@ -26,27 +26,33 @@ The child pipeline, called `TRIGGERED_CE/EE_PIPELINE` is generated in the Omnibu
 
 ## TRIGGERED_CE/EE_PIPELINE child pipeline
 
-This child pipeline consists of a job called `qa-test` which uses the `package-and-test/main.gitlab-ci.yml` file of the main GitLab project.
+This child pipeline consists of a job called `qa-subset-test` which uses the `package-and-test/main.gitlab-ci.yml` file of the main GitLab project.
 
-### qa-test job
+### qa-subset-test job
 
-The `qa-test` job triggers another child pipeline in the Omnibus-GitLab repository
-To get an allure report snapshot as a comment in the MR, following environment variables need to be passed to `qa-job`
+The `qa-subset-test` job triggers another child pipeline in the Omnibus-GitLab repository
+To get an allure report snapshot as a comment in the MR, following environment variables need to be passed to `qa-subset-test`
 
 | Environment Variable              | Description |
 | ----------------------------------|-------------|
 |   `GITLAB_AUTH_TOKEN`             | This is used to give access to the Danger bot to post comment in `omnibus-gitlab` repository. We are using  `$DANGER_GITLAB_API_TOKEN` which is also being used for other Danger bot related access in `omnibugs-gitlab` as mentioned [ci-variable](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/doc/development/ci-variables.md)        |
 |  `ALLURE_MERGE_REQUEST_IID`       | This denotes the MR ID which will be used by [e2e-test-report-job](#e2e-test-report-job) which inturn used `allure-report-publisher` to post message to MR with provided ID e.g. !6190 |
 
-## QA-TEST child pipeline
+### qa-remaining-test-manual
 
-This pipeline runs all the orchestrated tests using GitLab QA project which in turn uses allure gem to generate report source files for each test that is executed and stores the files in a common folder.
+The `qa-remaining-test-manual` job is a manual trigger pipeline. It triggers the same pipeline as `qa-subset-test` but runs the tests which aren't run as a part of `qa-subset-test` job. 
+
+The environment variables used in `qa-subset-test` are the same that are used in this job to generate the allure report. 
+
+## QA-SUBSET-TEST child pipeline
+
+This pipeline runs a subset of all the orchestrated tests using GitLab QA project which in turn uses allure gem to generate report source files for each test that is executed and stores the files in a common folder. Certain orchestrated jobs like `instance`, `decomposition-single-db`, `decomposition-multiple-db` and `praefect` run only smoke and reliable tests which intially used to run the entire suite. 
 
 ### e2e-test-report job
 
 The `e2e-test-report` job includes [.generate-allure-report-base](https://gitlab.com/gitlab-org/quality/pipeline-common/-/blob/master/ci/allure-report.yml) job which uses the `allure-report-publisher` gem to collate all the report in the mentioned folder into a single report and uploads it to the s3 bucket.
 
-It also posts the allure report as a comment on the MR having the ID passed in `ALLURE_MERGE_REQUEST_IID` variable in the [qa-test-job](#qa-test-job).
+It also posts the allure report as a comment on the MR having the ID passed in `ALLURE_MERGE_REQUEST_IID` variable in the [qa-subset-test](#qa-subset-test-job).
 
 [allure-report-publisher](https://github.com/andrcuns/allure-report-publisher) is a gem which uses allure in the backend. It has been catered for GitLab to upload the report and post the comment to MR.
 
@@ -73,11 +79,11 @@ subgraph QA flow in omnibus pipeline
         end
 
     subgraph Trigger:CE/EE-job Child Pipeline
-        A1["`**_trigger-qa_** stage <br> **_qa-test_** job`"]
+        A1["`**_trigger-qa_** stage <br> **_qa-subset-test_** job`"]
         A3(["`_package-and-test/main.gitlab-ci.yml_ <br> from _gitlab-org/gitlab_`"])
     end
 
-    subgraph qa-test Child Pipeline
+    subgraph qa-subset-test Child Pipeline
         A2["`from <br> **_package-and-test/main.gitlab-ci.yml_** in **_gitlab-org/gitlab_**`"]
         B1["`**_report_** stage <br> **_e2e-test-report_** job`"]
         B2(["`_.generate-allure-report-base_ job from<br> _quality/pipeline-common_`"])

doc/development/ci-variables.md

@@ -66,7 +66,7 @@ These additional variables are available to override or enable different build b
 | `DEPS_GITLAB_TOKEN`                                            | Token used by [dependencies.io](https://www.dependencies.io/gitlab/) to create MRs. |
 | `DEPS_TOKEN`                                                   | Token used by CI to auth to [dependencies.io](https://www.dependencies.io/gitlab/). |
 | `DOCS_API_TOKEN`                                               | Token used by CI to trigger a review-app build of the docs site.                    |
-| `MANUAL_QA_TEST`                                               | Variable used to decide if the `qa-test` job should be played automatically or not. |
+| `MANUAL_QA_TEST`                                               | Variable used to decide if the `qa-subset-test` job should be played automatically or not. |
 
 ## Release variables
 

doc/development/pipelines.md

@@ -287,15 +287,25 @@ This job builds a GitLab Docker image using the package that was built by
 
 This job is run only on [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) on triggered pipelines.
 
-#### `qa-test`
+#### `qa-subset-test`
 
-This job triggers a pipeline in the [GitLab QA Mirror](https://gitlab.com/gitlab-org/gitlab-qa-mirror), passing the GitLab
-Docker image created by `Trigger:gitlab-docker` job and the GitLab QA Docker
-image built by the GitLab Rails pipeline, so that a full QA run will be run
-against up using these images
+This job runs subset of test suite and is automatically triggered when either
+ `Trigger:CE-package` or `Trigger:EE-package` manual jobs are triggered manually.
+
+This job triggers a pipeline in the [GitLab QA Mirror](https://gitlab.com/gitlab-org/gitlab-qa-mirror), passing the GitLab Docker image created by `Trigger:gitlab-docker` job and the GitLab QA Docker image built by the GitLab Rails pipeline, so that a subset of test as mentioned in [this issue](https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/1303#we-should-keep) will run using these images
 
 This job is run only on [QA mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) on triggered pipelines.
 
+### `qa-remaining-test-manual` 
+
+This is a manual trigger job which runs the remaining tests that donot run in the `qa-subset-test` job.
+
+To run this QA jobs in MR pipeline, you need to either trigger `Trigger:CE-package` or `Trigger:EE-package` manual jobs.
+
+This job triggers a pipeline in the [GitLab QA Mirror](https://gitlab.com/gitlab-org/gitlab-qa-mirror), passing the GitLab
+Docker image created by `Trigger:gitlab-docker` job and the GitLab QA Docker
+image built by the GitLab Rails pipeline, so that the entire suite is run using these images. 
+
 #### `RAT`
 
 This manual job triggers a pipeline in the

doc/development/upgrading-software-components.md

@@ -115,7 +115,7 @@ At a minimum, the following tests should be performed when updating a software
 component.
 
 - Perform a successful GitLab Enterprise Edition (EE) build on all supported platforms.
-- Run `qa-test` CI/CD test job for both GitLab Enterprise Edition and GitLab Community Edition.
+- Run `qa-subset-test` as well as manual `qa-remaining-test-manual` CI/CD test job for both GitLab Enterprise Edition and GitLab Community Edition.
 - Install and verify that component version has been upgraded.
 - Verify basic functionality of the software component.
 

gitlab-ci-config/gitlab-com.yml

@@ -223,6 +223,27 @@
     - if: '$PIPELINE_TYPE == "DOCS_PIPELINE"'
       when: manual
 
+.qa-template:
+  variables:
+    QA_RUN_ALL_TESTS: "false"
+    RELEASE: $QA_RELEASE
+    QA_IMAGE: $QA_IMAGE
+    GITLAB_SEMVER_VERSION: $GITLAB_SEMVER_VERSION # latest semver gitlab version used for testing upgrade paths
+    SKIP_OMNIBUS_TRIGGER: "true"
+    GITLAB_AUTH_TOKEN: $DANGER_GITLAB_API_TOKEN
+    ALLURE_MERGE_REQUEST_IID: $CI_MERGE_REQUEST_IID
+  inherit:
+   variables: false
+  trigger:
+    strategy: depend
+    forward:
+      yaml_variables: true
+      pipeline_variables: true
+    include:
+      - project: gitlab-org/gitlab
+        ref: master
+        file: .gitlab/ci/package-and-test/main.gitlab-ci.yml
+
 #####################
 # Cache update jobs #
 #####################
@@ -557,28 +578,14 @@ Trigger:gitlab-docker:
       optional: true
       artifacts: true
 
-qa-test:
+qa-subset-test:
+  extends:
+    - .qa-template
   stage: trigger-qa
   variables:
-    RELEASE: $QA_RELEASE
-    QA_IMAGE: $QA_IMAGE
-    QA_RUN_ALL_TESTS: "false"
-    QA_SUITES: "QA::Scenario::Test::Integration::GitalyCluster,QA::Scenario::Test::Integration::InstanceSAML,QA::Scenario::Test::Integration::LDAPNoServer,QA::Scenario::Test::Integration::LDAPNoTLS,QA::Scenario::Test::Integration::LDAPTLS,QA::Scenario::Test::Integration::Mattermost,QA::Scenario::Test::Integration::Mtls,QA::Scenario::Test::Integration::ObjectStorageGcs,QA::Scenario::Test::Integration::RegistryTLS,QA::Scenario::Test::Integration::RegistryWithCDN,QA::Scenario::Test::Integration::SMTP,QA::Scenario::Test::Integration::Registry,QA::Scenario::Test::Instance::ObjectStorage,QA::Scenario::Test::Instance::RepositoryStorage,QA::Scenario::Test::Instance::GitlabPages,QA::Scenario::Test::Instance::Metrics,QA::EE::Scenario::Test::Integration::GroupSAML,QA::Scenario::Test::Instance::All,QA::Scenario::Test::Instance::Smoke"
-    GITLAB_SEMVER_VERSION: $GITLAB_SEMVER_VERSION # latest semver gitlab version used for testing upgrade paths
-    SKIP_OMNIBUS_TRIGGER: "true"
-    GITLAB_AUTH_TOKEN: $DANGER_GITLAB_API_TOKEN
-    ALLURE_MERGE_REQUEST_IID: $CI_MERGE_REQUEST_IID
-  inherit:
-   variables: false
-  trigger:
-    strategy: depend
-    forward:
-      yaml_variables: true
-      pipeline_variables: true
-    include:
-      - project: gitlab-org/gitlab
-        ref: master
-        file: .gitlab/ci/package-and-test/main.gitlab-ci.yml
+    QA_OMNIBUS_MR_TESTS: "only-smoke-reliable"
+    QA_RUN_TYPE: ${CI_JOB_NAME}
+    QA_SUITES: "QA::Scenario::Test::Integration::GitalyCluster,QA::Scenario::Test::Integration::InstanceSAML,QA::Scenario::Test::Integration::LDAPNoServer,QA::Scenario::Test::Integration::LDAPNoTLS,QA::Scenario::Test::Integration::LDAPTLS,QA::Scenario::Test::Integration::Mattermost,QA::Scenario::Test::Integration::Mtls,QA::Scenario::Test::Integration::ObjectStorageGcs,QA::Scenario::Test::Integration::RegistryTLS,QA::Scenario::Test::Integration::RegistryWithCDN,QA::Scenario::Test::Integration::SMTP,QA::Scenario::Test::Integration::Registry,QA::Scenario::Test::Instance::ObjectStorage,QA::Scenario::Test::Instance::RepositoryStorage,QA::Scenario::Test::Instance::GitlabPages,QA::Scenario::Test::Instance::Metrics,QA::EE::Scenario::Test::Integration::GroupSAML,QA::Scenario::Test::Instance::Smoke"
   rules:
     - if: '$SKIP_QA_TEST == "true"'
       when: never
@@ -602,6 +609,33 @@ qa-test:
     - job: Trigger:gitlab-docker
       artifacts: false
 
+qa-remaining-test-manual:
+  extends:
+    - .qa-template
+  stage: trigger-qa
+  variables:
+    QA_RUN_TYPE: ${CI_JOB_NAME}
+    QA_OMNIBUS_MR_TESTS: "except-smoke-reliable"
+    QA_SUITES: "QA::EE::Scenario::Test::Integration::Elasticsearch,QA::Scenario::Test::Integration::Import,QA::Scenario::Test::Integration::Integrations,QA::Scenario::Test::Integration::OAuth,QA::Scenario::Test::Integration::Jira,QA::Scenario::Test::Integration::ServicePingDisabled,QA::Scenario::Test::Instance::LargeSetup,QA::Scenario::Test::Instance::CloudActivation"
+  rules:
+    - if: '$SKIP_QA_TEST == "true"'
+      when: never
+    - if: '$PIPELINE_TYPE =~ /TRIGGERED_CE_PIPELINE/'
+      when: manual
+      allow_failure: true
+      variables:
+        FOSS_ONLY: "1"
+    - if: '$PIPELINE_TYPE =~ /TRIGGERED_EE_PIPELINE/'
+      when: manual
+      allow_failure: true
+  needs:
+    - job: generate-facts
+      artifacts: true
+    - job: Trigger:package
+      artifacts: false
+    - job: Trigger:gitlab-docker
+      artifacts: false
+
 letsencrypt-test:
   extends: .docker_job
   stage: trigger-qa

test-plans/upgrade-component-testplan-template.md

@@ -9,7 +9,7 @@ Add the component test-plan here. Use "[ ]" for each step/task to be completed.
 At a minimum, the following test should be run:
 
 - [ ] Performed a successful GitLab Enterprise Edition (EE) build on all supported platforms.
-- [ ] Ran `qa-test` CI/CD test job for both GitLab Enterprise Edition and GitLab Community Edition.
+- [ ] Ran `qa-subset-test` CI/CD test job for both GitLab Enterprise Edition and GitLab Community Edition.
 - [ ] Installed and verified that the component version has been upgraded.
 - [ ] Verified basic functionality of the software component.
 -->

test-plans/upgrade-mattermost-testplan.md

@@ -6,7 +6,7 @@
 ### Build tests
 
 - [ ] Built on all supported platforms
-- [ ] Ran `Trigger:ee-package` and then `qa-test` CI jobs on `gitlab.com`.
+- [ ] Ran `Trigger:ee-package` and then `qa-subset-test` as well as manual `qa-remaining-test-manual` CI jobs on `gitlab.com`.
 
 ### Fresh installation tests
 

test-plans/upgrade-redis-testplan.md

@@ -1,10 +1,10 @@
-# redis component upgrade test plan
+# Redis component upgrade test plan
 
 <!-- Copy and paste the following into your MR description. -->
 ## Test plan
 
 - [ ] Performed a successful GitLab Enterprise Edition (EE) build on all supported platforms (include `build-package-on-all-os` job).
-- [ ] Ran `qa-test` CI/CD test job for both GitLab Enterprise Edition and GitLab Community Edition.
+- [ ] Ran `qa-subset-test` as well as manual `qa-remaining-test-manual` CI/CD test job for both GitLab Enterprise Edition and GitLab Community Edition.
 - [ ] Redis indicator test cases were not failing.
   - [realtime components via assignee test](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347941)
   - [project template import](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347932)