Skip to content

Commit 016c4aa

Browse files
mashhursmashhurs
authored
[main branch] Rebuilds pull-request, build and E2E pipelines. (#199)
* Rebuilds pull-request, build and E2E pipelines for main branch. * Generate pull-request CI steps based on the test matrix. * Apply pull-request test matrix, build and E2E CIs considered manual/auto PR-builds and scheduled runs. * Build pipeline steps generation fix.
1 parent 7cd9cb7 commit 016c4aa

20 files changed

+392
-167
lines changed

‎.buildkite/build-pipeline.yml

Lines changed: 27 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1,43 +1,28 @@
1-
# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
1+
- label: "Build pipeline"
2+
command: |
3+
#!/usr/bin/env bash
4+
set -eo pipefail
25
3-
agents:
4-
provider: "gcp"
5-
machineType: "n1-standard-4"
6-
image: family/core-ubuntu-2204
7-
8-
steps:
9-
- label: ":hammer: Build plugin with LS & ES 8.x :elasticsearch:"
10-
# Builds with LS and ES last 8.x released version
11-
# Runs integration tests on 8.x released versions
12-
command:
13-
- .buildkite/scripts/run_tests.sh
14-
env:
15-
ELASTIC_STACK_VERSION: "8.current"
16-
SNAPSHOT: false
17-
INTEGRATION: true
18-
SECURE_INTEGRATION: true
19-
TARGET_BRANCH: "8.x"
20-
21-
- label: ":hammer: Build plugin with LS 8.x-SNAPSHOT & ES `main` branch :elasticsearch:"
22-
# Builds with LS last 8.x released version and ES main
23-
# Runs integration tests on 8.x released versions
24-
command:
25-
- .buildkite/scripts/run_tests.sh
26-
env:
27-
ELASTIC_STACK_VERSION: "8.current"
28-
ELASTICSEARCH_TREEISH: "main"
29-
SNAPSHOT: true
30-
INTEGRATION: true
31-
SECURE_INTEGRATION: true
32-
33-
- label: ":hammer: Build plugin with LS & ES `main` branch :elasticsearch:"
34-
# Builds with LS last 8.x released version and ES main
35-
# Runs integration tests on 8.x released versions
36-
command:
37-
- .buildkite/scripts/run_tests.sh
38-
env:
39-
ELASTIC_STACK_VERSION: "main"
40-
ELASTICSEARCH_TREEISH: "main"
41-
SNAPSHOT: true
42-
INTEGRATION: true
43-
SECURE_INTEGRATION: true
6+
echo "--- Downloading prerequisites"
7+
python3 -m pip install ruamel.yaml
8+
python3 -m pip install requests
9+
curl -fsSL --retry-max-time 60 --retry 3 --retry-delay 5 -o /usr/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64
10+
chmod a+x /usr/bin/yq
11+
12+
echo "--- Generating dynamic steps"
13+
set +e
14+
python3 .buildkite/scripts/build-pipeline/generate-steps.py > pipeline_steps.yml
15+
if [[ $$? -ne 0 ]]; then
16+
echo "^^^ +++"
17+
echo "There was a problem with generating pipeline steps."
18+
cat pipeline_steps.yml
19+
echo "Exiting now."
20+
exit 1
21+
else
22+
set -eo pipefail
23+
cat pipeline_steps.yml | yq .
24+
fi
25+
26+
set -eo pipefail
27+
echo "--- Uploading steps to buildkite"
28+
cat pipeline_steps.yml | buildkite-agent pipeline upload

‎.buildkite/e2e-pipeline.yml

Lines changed: 27 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,39 +1,28 @@
1-
# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
1+
- label: "E2E pipeline"
2+
command: |
3+
#!/usr/bin/env bash
4+
set -eo pipefail
25
3-
agents:
4-
provider: gcp
5-
imageProject: elastic-images-prod
6-
image: family/platform-ingest-logstash-multi-jdk-ubuntu-2204
7-
machineType: "n2-standard-4"
8-
diskSizeGb: 120
9-
10-
steps:
11-
# ------------- Run E2E tests ---------------------
12-
- label: ":test_tube: Run E2E tests with LS 8.x :rocket:"
13-
# uses the LS main & plugin 8.x branch when building the plugin
14-
# Runs integration tests against 8.x release version
15-
command:
16-
- .buildkite/scripts/run_e2e_tests.sh
17-
env:
18-
ELASTIC_STACK_VERSION: "8.current"
19-
TARGET_BRANCH: "8.x"
20-
21-
- label: ":test_tube: Run E2E tests with LS 8.x-SNAPSHOT :rocket:"
22-
# uses the LS & plugin main branch when building the plugin
23-
# Runs integration tests against 8.x-SNAPSHOT version
24-
command:
25-
- .buildkite/scripts/run_e2e_tests.sh
26-
env:
27-
ELASTIC_STACK_VERSION: "8.current"
28-
SNAPSHOT: true
29-
TARGET_BRANCH: "8.x"
30-
31-
- label: ":test_tube: Run E2E tests with LS `main` :rocket:"
32-
# uses the LS & plugin main branch when building the plugin
33-
# Runs integration tests against snapshot.main of https://raw.githubusercontent.com/elastic/logstash/main/ci/logstash_releases.json
34-
command:
35-
- .buildkite/scripts/run_e2e_tests.sh
36-
env:
37-
ELASTIC_STACK_VERSION: main
38-
SNAPSHOT: true
39-
ELASTICSEARCH_TREEISH: main
6+
echo "--- Downloading prerequisites"
7+
python3 -m pip install ruamel.yaml
8+
python3 -m pip install requests
9+
curl -fsSL --retry-max-time 60 --retry 3 --retry-delay 5 -o /usr/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64
10+
chmod a+x /usr/bin/yq
11+
12+
echo "--- Generating steps dynamically"
13+
set +e
14+
python3 .buildkite/scripts/e2e-pipeline/generate-steps.py > pipeline_steps.yml
15+
if [[ $$? -ne 0 ]]; then
16+
echo "^^^ +++"
17+
echo "There was a problem with generating pipeline steps."
18+
cat pipeline_steps.yml
19+
echo "Exiting now."
20+
exit 1
21+
else
22+
set -eo pipefail
23+
cat pipeline_steps.yml | yq .
24+
fi
25+
26+
set -eo pipefail
27+
echo "--- Uploading steps to buildkite"
28+
cat pipeline_steps.yml | buildkite-agent pipeline upload

‎.buildkite/pull-request-pipeline.yml

Lines changed: 27 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,29 @@
1-
# define a GCP VM agent to support container management (by default agent doesn't support)
2-
agents:
3-
provider: "gcp"
4-
machineType: "n1-standard-4"
5-
image: family/core-ubuntu-2204
1+
- label: "Pull request pipeline"
2+
command: |
3+
#!/usr/bin/env bash
4+
set -eo pipefail
5+
echo "--- Downloading prerequisites"
6+
python3 -m pip install ruamel.yaml
7+
python3 -m pip install requests
8+
curl -fsSL --retry-max-time 60 --retry 3 --retry-delay 5 -o /usr/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64
9+
chmod a+x /usr/bin/yq
610
7-
steps:
8-
# ------------- Unit tests ---------------------
9-
- label: ":hammer: Unit tests with LS & ES main :docker:"
10-
# Builds the plugin (with current changes) against LS and ES main
11-
# Runs unit tests on LS & ES main docker
12-
command:
13-
- .buildkite/scripts/run_tests.sh
14-
env:
15-
ELASTIC_STACK_VERSION: main
16-
ELASTICSEARCH_TREEISH: main
17-
INTEGRATION: false
18-
SNAPSHOT: true
11+
export TARGET_BRANCH="$GITHUB_PR_TARGET_BRANCH"
1912
20-
# ------------- Integration tests ---------------------
21-
- label: ":hammer: Integration tests with LS & ES main :docker:"
22-
# Builds the plugin (with current changes) against LS and ES main
23-
# Runs integration tests on snapshot.main of https://raw.githubusercontent.com/elastic/logstash/main/ci/logstash_releases.json
24-
command:
25-
- .buildkite/scripts/run_tests.sh
26-
env:
27-
ELASTIC_STACK_VERSION: main
28-
ELASTICSEARCH_TREEISH: main
29-
SNAPSHOT: true
30-
INTEGRATION: true
31-
SECURE_INTEGRATION: true
32-
LOG_LEVEL: "info"
13+
echo "--- Generating steps dynamically"
14+
set +e
15+
python3 .buildkite/scripts/pull-request-pipeline/generate-steps.py > pipeline_steps.yml
16+
if [[ $$? -ne 0 ]]; then
17+
echo "^^^ +++"
18+
echo "There was a problem with generating pipeline steps."
19+
cat pipeline_steps.yml
20+
echo "Exiting now."
21+
exit 1
22+
else
23+
set -eo pipefail
24+
cat pipeline_steps.yml | yq .
25+
fi
26+
27+
set -eo pipefail
28+
echo "--- Uploading steps to buildkite"
29+
cat pipeline_steps.yml | buildkite-agent pipeline upload

‎.buildkite/scripts/build-pipeline/generate-steps.py

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
import os
2+
import requests
3+
import sys
4+
import typing
5+
from requests.adapters import HTTPAdapter, Retry
6+
7+
from ruamel.yaml import YAML
8+
9+
RELEASES_URL = "https://raw.githubusercontent.com/elastic/logstash/main/ci/logstash_releases.json"
10+
TEST_COMMAND: typing.final = ".buildkite/scripts/run_tests.sh"
11+
12+
13+
def call_url_with_retry(url: str, max_retries: int = 5, delay: int = 1) -> requests.Response:
14+
schema = "https://" if "https://" in url else "http://"
15+
session = requests.Session()
16+
# retry on most common failures such as connection timeout(408), etc...
17+
retries = Retry(total=max_retries, backoff_factor=delay, status_forcelist=[408, 502, 503, 504])
18+
session.mount(schema, HTTPAdapter(max_retries=retries))
19+
return session.get(url)
20+
21+
22+
def generate_test_step(stack_version, branch, snapshot) -> dict:
23+
label_integration_test: typing.final = f"Integration test for {stack_version}, snapshot: {snapshot}"
24+
step: dict = {
25+
"label": label_integration_test,
26+
"command": TEST_COMMAND,
27+
"env": {
28+
"SNAPSHOT": snapshot,
29+
"ELASTIC_STACK_VERSION": stack_version,
30+
"INTEGRATION": "true",
31+
"SECURE_INTEGRATION": "true",
32+
"LOG_LEVEL": "info"
33+
}
34+
}
35+
# we are not going to set branch if job kicked of through webhook (PR merge or manual PR run)
36+
if branch is not None:
37+
step["env"]["TARGET_BRANCH"] = branch
38+
return step
39+
40+
41+
def generate_steps_for_scheduler(versions) -> list:
42+
steps: list = []
43+
snapshots = versions["snapshots"]
44+
for snapshot_version in snapshots:
45+
if snapshots[snapshot_version].startswith("7."):
46+
continue
47+
full_stack_version = snapshots[snapshot_version]
48+
version_parts = snapshots[snapshot_version].split(".")
49+
major_minor_versions = snapshot_version if snapshot_version == "main" else f"{version_parts[0]}.{version_parts[1]}"
50+
branch = f"{version_parts[0]}.x" if snapshot_version.find("future") > -1 else major_minor_versions
51+
steps.append(generate_test_step(full_stack_version, branch, "true"))
52+
return steps
53+
54+
55+
def generate_steps_for_main_branch(versions) -> list:
56+
steps: list = []
57+
full_stack_version: typing.final = versions["snapshots"]["main"]
58+
steps.append(generate_test_step(full_stack_version, None, "true"))
59+
return steps
60+
61+
62+
if __name__ == "__main__":
63+
structure = {
64+
"agents": {
65+
"provider": "gcp",
66+
"machineType": "n1-standard-4",
67+
"image": "family/core-ubuntu-2204"
68+
},
69+
"steps": []}
70+
71+
response = call_url_with_retry(RELEASES_URL)
72+
versions_json: typing.final = response.json()
73+
74+
# Use BUILDKITE_SOURCE to figure out PR merge or schedule.
75+
# If PR merge, no need to run builds on all branches, target branch will be good
76+
# - webhook when PR gets merged
77+
# - schedule when daily schedule starts
78+
# - ui when manually kicking job from BK UI
79+
# - manual kick off will be on PR or entire main branch, can be decided with BUILDKITE_BRANCH
80+
bk_source = os.getenv("BUILDKITE_SOURCE")
81+
bk_branch = os.getenv("BUILDKITE_BRANCH")
82+
steps = generate_steps_for_scheduler(versions_json) if (bk_source == "schedule" or bk_branch == "main") \
83+
else generate_steps_for_main_branch(versions_json)
84+
85+
group_desc = f"Build steps"
86+
key_desc = "build-steps"
87+
structure["steps"].append({
88+
"group": group_desc,
89+
"key": key_desc,
90+
"steps": steps,
91+
})
92+
93+
print(
94+
'# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json')
95+
YAML().dump(structure, sys.stdout)

‎.buildkite/scripts/e2e/README.md renamed to ‎.buildkite/scripts/e2e-pipeline/README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,19 +32,19 @@ In order to run tests with serverless, you also need to export `EC_API_KEY` whic
3232
In the pipelines, this will be automatically retrieved from Vault services.
3333

3434
#### Stack version
35-
E2E also requires `STACK_VERSION` (ex: "8.12.0") environment variable in order to test against.
35+
E2E also requires `ELASTIC_STACK_VERSION` (ex: "8.12.0") environment variable in order to test against.
3636
Make sure to export it before running. In the Buildkite pipeline, this var will be resolved and exported.
3737

3838
#### Installing dependencies
3939
Make sure you have python installed on you local
4040
```bash
41-
pip install -r .buildkite/scripts/e2e/requirements.txt
41+
pip install -r .buildkite/scripts/e2e-pipeline/requirements.txt
4242
```
4343

4444
### Run
4545
Run the following command from the repo dir:
4646
```bash
47-
python3 .buildkite/scripts/e2e/main.py
47+
python3 .buildkite/scripts/e2e-pipeline/main.py
4848
```
4949

5050
## Troubleshooting

‎.buildkite/scripts/e2e/__init__.py renamed to ‎.buildkite/scripts/e2e-pipeline/__init__.py

File renamed without changes.

‎.buildkite/scripts/e2e/bootstrap.py renamed to ‎.buildkite/scripts/e2e-pipeline/bootstrap.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -122,7 +122,7 @@ def __reload_container(self) -> None:
122122
time.sleep(20) # give a time Logstash pipeline to fully start
123123

124124
def __update_pipeline_config(self) -> None:
125-
local_config_file_path = ".buildkite/scripts/e2e/config/"
125+
local_config_file_path = ".buildkite/scripts/e2e-pipeline/config/"
126126
config_file = "serverless_pipeline.conf" if self.project_type == "serverless" else "pipeline.conf"
127127
local_config_file = local_config_file_path + config_file
128128
container_config_file_path = "/usr/share/logstash/pipeline/logstash.conf"

‎.buildkite/scripts/e2e/config/pipeline.conf renamed to ‎.buildkite/scripts/e2e-pipeline/config/pipeline.conf

File renamed without changes.

‎.buildkite/scripts/e2e/config/serverless_pipeline.conf renamed to ‎.buildkite/scripts/e2e-pipeline/config/serverless_pipeline.conf

File renamed without changes.

0 commit comments

Comments
 (0)