[Upgrade tests] Add support to run upgrade tests in system tests

[mrodm]

Part of #1831

Add support to run as part of the system tests, and upgrade test for each test defined in the package.

An upgrade test in the scope of system tests consists of:

• Install a previous version of the package
• Create a package policy fulfilling the variables with the values found in the new package (development files).
• Enroll a new Elastic Agent.
• Setup the service.
• Wait for documents to be ingested.
• Perform validations (same as the current ones performed by system tests).
• Install the version of the package that it is being worked.
• Upgrade the package policy.
• Wait for new documents to be ingested.
• Perform validations (same as the current ones performed by system tests).
• Run tear down process
  • Shutdown service
  • Shutdown and unenroll Elastic Agent.
  • Delete package policy.
  • Uninstall package.

In order to perform the upgrade of a package and the corresponding policy, these are the new steps that must be added to the current process:

1. Install package in the new version

POST kbn:/api/fleet/epm/packages/nginx/2.2.1 
{
"force": true
}

2. Get the required package policy ID
   • Search this ID via API

     GET kbn:/api/fleet/package_policies
     

   • It could be stored in the tests directly the package policy ID created in the previous step
3. Upgrade the given package policy ID, so the changes take effect, via API:

   POST kbn:/api/fleet/package_policies/upgrade
   {
     "packagePolicyIds": ["6d916dff-281c-4a78-85a1-908f902e2d8e"]
   }
   

Considerations

Differences with respect current policy tests

Currently, in system tests the package to be tested is installed once at the beginning of the execution and elastic-package runs all the tests defined. Once finished all tests, the package is uninstalled.

Adding upgrade tests in these system tests causes that this process must change since for each test:

• the old version must be installed and configured in a package policy.
• the new version must be installed and configured in the same package policy.
• this package must be uninstalled.

As this process must be performed for each test, it implies that all these tests must run sequentially.

This will cause that executing all the tests of a package will be slower.

Upgrade tests with the same package version

There could be scenarios where the old and new package used in these upgrade tests are the same.

In these cases, the new version must contain a suffix added on the fly when building/installing it (e.g. <version>+<timestamp>). Example: #1831 (comment)

This allows to avoid issues when package is installed in Fleet (reference #1831 (comment)).

Add mechanism to disable tests

As there could be flakiness in those tests, there should be a new setting in the test configuration files to disable a specific test.

This could be an example of the settings to disable (TBD):

upgrade_tests:
  enable: false         # Disable completely test
  disable_until: 9.3.0  # Or, Disable upgrade test until a specific Elastic stack version
vars: ~
data_stream:
  vars:
    paths:
      - "{{SERVICE_LOGS_DIR}}/access.log*"

Requirements

• These upgrade tests must be disabled by default.
• These upgrade tests will focus on upgrading packages, and they are performed with the same Elastic stack version.
• Add new settings to disable upgrade tests in configuration test files.
• If those upgrade tests are not executed, the system tests should execute the same steps as they do now
  • Install the package just once at the beginning, run all the tests, and finally uninstall the package.
• If executed, results must be added as part of the xUnit results.
• If executed, same validations performed in system tests must be executed before and after upgrading the package.
• Add a new flag and environment variable to select which version to use as the old one --upgrade-from:
  • Values: latest, previous or a specific version (e.g. 2.0.1)
  • Allow to define an environment variable to set the same value (e.g. ELASTIC_PACKAGE_UPGRADE_TEST_FROM_VERSION)