Document how system tests work (#727)

Author: endorama

docs/howto/system_testing.md

@@ -143,6 +143,32 @@ data "aws_ami" "latest-amzn" {
 
 Notice the use of the `TEST_RUN_ID` variable. It contains a unique ID, which can help differentiate resources created in potential concurrent test runs.
 
+#### Environment variables
+
+To use environment variables within the Terraform service deployer a `env.yml` file is required.
+
+The file should be structured like this:
+
+```yaml
+version: '2.3'
+services:
+  terraform:
+    environment:
+      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
+```
+
+It's purpose is to inject environment variables in the Terraform service deployer environment.
+
+To specify a default use this syntax: `AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-default}`, replacing `default` with the desired default value.
+
+**NOTE**: Terraform requires to prefix variables using the environment variables form with `TF_VAR_`. These variables are not available in test case definitions because they are [not injected](https://github.com/elastic/elastic-package/blob/f5312b6022e3527684e591f99e73992a73baafcf/internal/testrunner/runners/system/servicedeployer/terraform_env.go#L43) in the test environment.
+
+#### Cloud Provider CI support
+
+Terraform is often used to interact with Cloud Providers. This require Cloud Provider credentials.
+
+Injecting credentials can be achieved with functions from the [`apm-pipeline-library`](https://github.com/elastic/apm-pipeline-library/tree/main/vars) Jenkins library. For example look for `withAzureCredentials`, `withAWSEnv` or `withGCPEnv`.
+
 ### Kubernetes service deployer
 
 The Kubernetes service deployer requires the `_dev/deploy/k8s` directory to be present. It can include additional `*.yaml` files to deploy
@@ -217,6 +243,9 @@ The `SERVICE_LOGS_DIR` placeholder is not the only one available for use in a da
 
 Placeholders used in the `test-<test_name>-config.yml` must be enclosed in `{{` and `}}` delimiters, per Handlebars syntax.
 
+
+**NOTE**: Terraform variables in the form of environment variables (prefixed with `TF_VAR_`) are not injected and cannot be used as placeholder (their value will always be empty).
+
 ## Running a system test
 
 Once the two levels of configurations are defined as described in the previous section, you are ready to run system tests for a package's data streams.
@@ -264,3 +293,18 @@ to indexing generated data from the integration's data streams into Elasticsearc
 ```
 elastic-package test system --generate
 ```
+
+## Continuous Integration
+
+`elastic-package` runs a set of system tests on some [dummy packages](https://github.com/elastic/elastic-package/tree/main/test/packages) to ensure it's functionalities work as expected. This allows to test changes affecting package testing within `elastic-package` before merging and releasing the changes.
+
+Tests use set of environment variables that are set at the beginning of the `Jenkinsfile`.
+
+The exposed environment variables are passed to the test runners through service deployer specific configuration (refer to the service deployer section for further details).
+
+### Stack version
+
+The tests use the [default version](https://github.com/elastic/elastic-package/blob/main/internal/install/stack_version.go#L9) `elastic-package` provides.
+
+You can override this value by changing it in your PR if needed. To update the default version always create a dedicated PR.
+