[Upgrade tests] Add support to run upgrade tests in policy tests

[mrodm]

Part of #1831

Add support to run as part of the policy tests, and upgrade test for each test defined in the package.

An upgrade test in the scope of policy tests consists of:

• Install a previous version of the package
• Create a package policy fulfilling the variables with the values found in the new package (development files).
• Install the version of the package that it is being worked.
• Upgrade the package policy.
• Perform validations (compare policy test with the expected one).
• Delete package policy.
• Uninstall package.

In order to perform the upgrade of a package and the corresponding policy, these are the new steps that must be added to the current process:

1. Install package in the new version

POST kbn:/api/fleet/epm/packages/nginx/2.2.1 
{
"force": true
}

2. Get the required package policy ID
   • Search this ID via API

     GET kbn:/api/fleet/package_policies
     

   • It could be stored in the tests directly the package policy ID created in the previous step
3. Upgrade the given package policy ID, so the changes take effect, via API:

   POST kbn:/api/fleet/package_policies/upgrade
   {
     "packagePolicyIds": ["6d916dff-281c-4a78-85a1-908f902e2d8e"]
   }
   

Considerations

Differences with respect current policy tests

Currently, in policy tests the package to be tested is installed once at the beginning of the execution and elastic-package runs all the tests defined. Once finished all tests, the package is uninstalled.

Adding upgrade tests in these policy tests causes that this process must change since for each test:

• the old version must be installed and configured in a package policy.
• the new version must be installed and configured in the same package policy.
• this package must be uninstalled.

As this process must be performed for each test, it implies that all these tests must run sequentially.

This will cause that executing all the tests of a package will be slower.

Upgrade tests with the same package version

There could be scenarios where the old and new package used in these upgrade tests are the same.

In these cases, the new version must contain a suffix added on the fly when building/installing it (e.g. <version>+<timestamp>). Example: #1831 (comment)

This allows to avoid issues when package is installed in Fleet (reference #1831 (comment)).

Add mechanism to disable tests

As there could be flakiness in those tests, there should be a new setting in the test configuration files to disable a specific test.

This could be an example of the settings to disable (TBD):

upgrade_tests:
  enable: false         # Disable completely test
  disable_until: 9.3.0  # Or, Disable upgrade test until a specific Elastic stack version
vars: ~
data_stream:
  vars:
    paths:
      - "{{SERVICE_LOGS_DIR}}/access.log*"

Requirements

• These upgrade tests must be disabled by default.
• These upgrade tests will focus on upgrading packages, and they are performed with the same Elastic stack version.
• Add new settings to disable upgrade tests in configuration test files.
• If those upgrade tests are not executed, the policy tests should execute the same steps as they do now
  • Install the package just once at the beginning, run all the tests, and finally uninstall the package.
• If executed, results must be added as part of the xUnit results.
• Add a new flag and environment variable to select which version to use as the old one --upgrade-from:
  • Values: latest, previous or a specific version (e.g. 2.0.1)
  • Allow to define an environment variable to set the same value (e.g. ELASTIC_PACKAGE_UPGRADE_TEST_FROM_VERSION)

[jsoriano]

Create a package policy fulfilling the variables with the values found in the new package (development files).

We will need to be a bit tolerant with this step, because it could be possible that new variables are not available in old package version.

Or maybe we could have a previous step that checks if all the variables used in the test file are defined in the manifest of the old package.