Skip to content

Commit 2cc7d7f

Browse files
mashhursmashhurs
authored
[8.x] Backport of #200, set ES treeish to 8.17 (#205)
* [8.x] Backport of #190, dynamic CI pr/build/e2e jobs for target branch. (#200) * Sync up with ES main on `PipelineConfiguration` class changes. (#190) * Sync up with ES main that PipelineConfiguration class getConfigAsMap() method renamed to getConfig(). * Request modifiable donfig when creating a pipeline. * Fixes PR CI jobs, adds build and E2E CI jobs. * Fix the E2E test jobs. * Generate CI steps based on defined test matrix. * Apply suggestions from code review - print target branch for debugging purpose - add return type `None` to void method. Co-authored-by: Cas Donoghue <[email protected]> * Replace the matrix with origin URL (after landing). --------- Co-authored-by: Cas Donoghue <[email protected]> (cherry picked from commit 8e4c02a) * Set ES treeish to 8.17
1 parent dd9b699 commit 2cc7d7f

File tree

22 files changed

+906
-78
lines changed

22 files changed

+906
-78
lines changed

‎.buildkite/build-pipeline.yml

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,30 @@
1-
# Intentional empty definition because it is in 8.x
1+
- label: "Build pipeline"
2+
command: |
3+
#!/usr/bin/env bash
4+
set -eo pipefail
5+
6+
echo "--- Downloading prerequisites"
7+
python3 -m pip install ruamel.yaml
8+
python3 -m pip install requests
9+
curl -fsSL --retry-max-time 60 --retry 3 --retry-delay 5 -o /usr/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64
10+
chmod a+x /usr/bin/yq
11+
12+
export TARGET_BRANCH="$BUILDKITE_BRANCH"
13+
14+
echo "--- Generating steps dynamically"
15+
set +e
16+
python3 .buildkite/scripts/build-pipeline/generate-steps.py > pipeline_steps.yml
17+
if [[ $$? -ne 0 ]]; then
18+
echo "^^^ +++"
19+
echo "There was a problem with generating pipeline steps."
20+
cat pipeline_steps.yml
21+
echo "Exiting now."
22+
exit 1
23+
else
24+
set -eo pipefail
25+
cat pipeline_steps.yml | yq .
26+
fi
27+
28+
set -eo pipefail
29+
echo "--- Uploading steps to buildkite"
30+
cat pipeline_steps.yml | buildkite-agent pipeline upload

‎.buildkite/e2e-pipeline.yml

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,30 @@
1-
# Intentional empty definition because it is in 8.x
1+
- label: "E2E pipeline"
2+
command: |
3+
#!/usr/bin/env bash
4+
set -eo pipefail
5+
6+
echo "--- Downloading prerequisites"
7+
python3 -m pip install ruamel.yaml
8+
python3 -m pip install requests
9+
curl -fsSL --retry-max-time 60 --retry 3 --retry-delay 5 -o /usr/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64
10+
chmod a+x /usr/bin/yq
11+
12+
export TARGET_BRANCH="$BUILDKITE_BRANCH"
13+
14+
echo "--- Generating steps dynamically"
15+
set +e
16+
python3 .buildkite/scripts/e2e-pipeline/generate-steps.py > pipeline_steps.yml
17+
if [[ $$? -ne 0 ]]; then
18+
echo "^^^ +++"
19+
echo "There was a problem with generating pipeline steps."
20+
cat pipeline_steps.yml
21+
echo "Exiting now."
22+
exit 1
23+
else
24+
set -eo pipefail
25+
cat pipeline_steps.yml | yq .
26+
fi
27+
28+
set -eo pipefail
29+
echo "--- Uploading steps to buildkite"
30+
cat pipeline_steps.yml | buildkite-agent pipeline upload

‎.buildkite/pull-request-pipeline.yml

Lines changed: 29 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -1,53 +1,29 @@
1-
# define a GCP VM agent to support container management (by default agent doesn't support)
2-
agents:
3-
provider: "gcp"
4-
machineType: "n1-standard-4"
5-
image: family/core-ubuntu-2204
6-
7-
steps:
8-
# ------------- Unit tests ---------------------
9-
- label: ":hammer: Unit tests with LS & ES 8.current :docker:"
10-
# Builds the plugin (with current changes) with the version defined in gradle.properties
11-
# and against LS and ES 8.current
12-
# Runs unit tests on LS & ES 8.current docker
13-
command:
14-
- .buildkite/scripts/run_tests.sh
15-
env:
16-
ELASTIC_STACK_VERSION: "8.current"
17-
INTEGRATION: false
18-
19-
- label: ":hammer: Unit tests with LS & ES 8.current-SNAPSHOT :docker:"
20-
# Builds the plugin (with current changes) with the version defined in gradle.properties
21-
# and against LS and ES 8.current-SNAPSHOT
22-
# Runs unit tests on LS & ES 8.current-SNAPSHOT docker
23-
command:
24-
- .buildkite/scripts/run_tests.sh
25-
env:
26-
ELASTIC_STACK_VERSION: "8.current"
27-
SNAPSHOT: true
28-
INTEGRATION: false
29-
30-
# ------------- Integration tests ---------------------
31-
- label: ":hammer: Integration tests with LS & ES 8.current :docker:"
32-
# Builds the plugin (with current changes) with the version defined in gradle.properties
33-
# and against LS and ES 8.current
34-
# Runs integration tests on LS & ES 8.current docker
35-
command:
36-
- .buildkite/scripts/run_tests.sh
37-
env:
38-
ELASTIC_STACK_VERSION: "8.current"
39-
INTEGRATION: true
40-
SECURE_INTEGRATION: true
41-
42-
- label: ":hammer: Integration tests with LS & ES 8.current-SNAPSHOT :docker:"
43-
# Builds the plugin (with current changes) with the version defined in gradle.properties
44-
# and against LS and ES 8.current-SNAPSHOT
45-
# Runs integration tests on LS & ES 8.current-SNAPSHOT docker
46-
command:
47-
- .buildkite/scripts/run_tests.sh
48-
env:
49-
ELASTIC_STACK_VERSION: "8.current"
50-
SNAPSHOT: true
51-
INTEGRATION: true
52-
SECURE_INTEGRATION: true
53-
LOG_LEVEL: "info"
1+
- label: "Pull request pipeline"
2+
command: |
3+
#!/usr/bin/env bash
4+
set -eo pipefail
5+
echo "--- Downloading prerequisites"
6+
python3 -m pip install ruamel.yaml
7+
python3 -m pip install requests
8+
curl -fsSL --retry-max-time 60 --retry 3 --retry-delay 5 -o /usr/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64
9+
chmod a+x /usr/bin/yq
10+
11+
export TARGET_BRANCH="$GITHUB_PR_TARGET_BRANCH"
12+
13+
echo "--- Generating steps dynamically"
14+
set +e
15+
python3 .buildkite/scripts/pull-request-pipeline/generate-steps.py > pipeline_steps.yml
16+
if [[ $$? -ne 0 ]]; then
17+
echo "^^^ +++"
18+
echo "There was a problem with generating pipeline steps."
19+
cat pipeline_steps.yml
20+
echo "Exiting now."
21+
exit 1
22+
else
23+
set -eo pipefail
24+
cat pipeline_steps.yml | yq .
25+
fi
26+
27+
set -eo pipefail
28+
echo "--- Uploading steps to buildkite"
29+
cat pipeline_steps.yml | buildkite-agent pipeline upload

‎.buildkite/scripts/build-pipeline/generate-steps.py

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
import os
2+
import requests
3+
import sys
4+
import typing
5+
from requests.adapters import HTTPAdapter, Retry
6+
7+
from ruamel.yaml import YAML
8+
9+
RELEASES_URL = "https://raw.githubusercontent.com/elastic/logstash/main/ci/logstash_releases.json"
10+
TEST_COMMAND: typing.final = ".buildkite/scripts/run_tests.sh"
11+
12+
13+
def generate_unit_and_integration_test_steps(stack_version, snapshot) -> list[typing.Any]:
14+
test_steps = []
15+
16+
# step-1, unit tests
17+
label_unit_test: typing.final = f"Unit test for {stack_version}, snapshot: {snapshot}"
18+
test_steps.append({
19+
"label": label_unit_test,
20+
"command": TEST_COMMAND,
21+
"env": {
22+
"SNAPSHOT": snapshot,
23+
"ELASTIC_STACK_VERSION": stack_version,
24+
"INTEGRATION": "false"
25+
}
26+
})
27+
28+
# step-2, integration tests
29+
label_integration_test: typing.final = f"Integration test for {stack_version}, snapshot: {snapshot}"
30+
test_steps.append({
31+
"label": label_integration_test,
32+
"command": TEST_COMMAND,
33+
"env": {
34+
"SNAPSHOT": snapshot,
35+
"ELASTIC_STACK_VERSION": stack_version,
36+
"INTEGRATION": "true",
37+
"SECURE_INTEGRATION": "true",
38+
"LOG_LEVEL": "info"
39+
}
40+
})
41+
return test_steps
42+
43+
44+
def call_url_with_retry(url: str, max_retries: int = 5, delay: int = 1) -> requests.Response:
45+
schema = "https://" if "https://" in url else "http://"
46+
session = requests.Session()
47+
# retry on most common failures such as connection timeout(408), etc...
48+
retries = Retry(total=max_retries, backoff_factor=delay, status_forcelist=[408, 502, 503, 504])
49+
session.mount(schema, HTTPAdapter(max_retries=retries))
50+
return session.get(url)
51+
52+
53+
if __name__ == "__main__":
54+
structure = {
55+
"agents": {
56+
"provider": "gcp",
57+
"machineType": "n1-standard-4",
58+
"image": "family/core-ubuntu-2204"
59+
},
60+
"steps": []}
61+
62+
steps = []
63+
response = call_url_with_retry(RELEASES_URL)
64+
versions_json = response.json()
65+
66+
# there are situations to manually run CIs with PR change,
67+
# set MANUAL_TARGET_BRANCH with upstream target branch and run
68+
manually_set_target_branch: typing.final = os.getenv("MANUAL_TARGET_BRANCH")
69+
target_branch: typing.final = manually_set_target_branch if manually_set_target_branch else os.getenv("TARGET_BRANCH")
70+
print(f"Running with target_branch: {target_branch}")
71+
if target_branch == '8.x':
72+
full_stack_version: typing.final = versions_json["snapshots"]["8.future"]
73+
steps += generate_unit_and_integration_test_steps(full_stack_version, "true")
74+
elif target_branch == 'main':
75+
full_stack_version: typing.final = versions_json["snapshots"][target_branch]
76+
steps += generate_unit_and_integration_test_steps(full_stack_version, "true")
77+
else:
78+
# generate steps for the version if released
79+
releases = versions_json["releases"]
80+
for release_version in releases:
81+
if releases[release_version].startswith(target_branch):
82+
steps += generate_unit_and_integration_test_steps(releases[release_version], "false")
83+
break
84+
85+
# steps for snapshot version
86+
snapshots = versions_json["snapshots"]
87+
for snapshot_version in snapshots:
88+
if snapshots[snapshot_version].startswith(target_branch):
89+
steps += generate_unit_and_integration_test_steps(snapshots[snapshot_version], "false")
90+
break
91+
92+
group_desc = f"{target_branch} branch steps"
93+
key_desc = "pr-and-build-steps"
94+
structure["steps"].append({
95+
"group": group_desc,
96+
"key": key_desc,
97+
"steps": steps
98+
})
99+
100+
print(
101+
'# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json')
102+
YAML().dump(structure, sys.stdout)

‎.buildkite/scripts/e2e-pipeline/README.md

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
2+
## Getting started
3+
E2E tests can be also run on your local machine.
4+
5+
### Prerequisites
6+
7+
#### Building the plugin
8+
E2E tries to install the plugin on Logstash container from the gem file. To generate a gem:
9+
1. Building plugin requires Logstash
10+
- If you have `LOGSTASH_PATH` already defined, skip this step
11+
- You can also download Logstash and export its path to `LOGSTASH_PATH`
12+
- OR build from source
13+
```shell
14+
git clone --single-branch https://github.com/elastic/logstash.git
15+
cd logstash && ./gradlew clean bootstrap assemble installDefaultGems && cd ..
16+
LOGSTASH_PATH=$(pwd)/logstash
17+
export LOGSTASH_PATH
18+
```
19+
2. Run the following command:
20+
```shell
21+
./gradlew clean vendor localGem
22+
```
23+
24+
#### Defining a project type
25+
If you want to run tests with serverless, this will be for you.
26+
Defaults to `on_prems` where local stack containers will be spun up and tested.
27+
```bash
28+
export E2E_PROJECT_TYPE="serverless"
29+
```
30+
31+
In order to run tests with serverless, you also need to export `EC_API_KEY` which is an organization API key to create a project.
32+
In the pipelines, this will be automatically retrieved from Vault services.
33+
34+
#### Stack version
35+
E2E also requires `ELASTIC_STACK_VERSION` (ex: "8.12.0") environment variable in order to test against.
36+
Make sure to export it before running. In the Buildkite pipeline, this var will be resolved and exported.
37+
38+
#### Installing dependencies
39+
Make sure you have python installed on you local
40+
```bash
41+
pip install -r .buildkite/scripts/e2e-pipeline/requirements.txt
42+
```
43+
44+
### Run
45+
Run the following command from the repo dir:
46+
```bash
47+
python3 .buildkite/scripts/e2e-pipeline/main.py
48+
```
49+
50+
## Troubleshooting
51+
- The project retries on some operations to overcome timeout issues, uses [`retry` tool](https://formulae.brew.sh/formula/retry). If you get `retry` undefined error, make sure to install it.
52+
```
53+
brew install retry
54+
```
55+
56+
- If you run multiple times, currently cloning `integrations` repo will fail, so make sure to remove the repo folder or comment out the `bootstrap.py#__clone_integrations_repo()` method

‎.buildkite/scripts/e2e-pipeline/__init__.py

Whitespace-only changes.

0 commit comments

Comments
 (0)