Skip to content

Commit 92e139c

Browse files
pkoutsovasilispkoutsovasilis
authored
[tests] split up serverless and resource leaks integration tests (#8396)
* fix: expose PreinstalledPackages * fix: make groups.go importable * feat: introduce new leak package for integration tests * feat: introduce new serverless package for integration tests * feat: incorporate serverless and extended runtime leak tests in mage * fix: normalise powershell (windows) and bash (unix) integration tests invocation flow * fix: use -test prefixed args for test-wise argument in buildkite-integration-tests.ps1 * fix: add comment about test timeout 2h0m0s * fix: correct wording * doc: add integration:TestServerless and update integration:TestForResourceLeaks in test-framework-dev-guide.md * fix: return directly the test exit code for ps1 scripts * fix: rename file that host TestMain to main_test.go * fix: correct build tags for serverless integration tests package * fix: leverage log.Fatalf in serverless/main_test.go * feat: expand the BK log group on failure * tmp: introduce artificial test failure to test buildkite logging group expansion * Revert "tmp: introduce artificial test failure to test buildkite logging group expansion" This reverts commit ee6bd63. * fix: typo * fix: remove redundant BK log group expansion
1 parent 83a46db commit 92e139c

28 files changed

+882
-659
lines changed

‎.buildkite/bk.integration.pipeline.yml‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -39,10 +39,10 @@ steps:
3939
depends_on:
4040
- packaging-windows
4141
env:
42-
TEST_LONG_RUNNING: "true"
42+
TEST_PACKAGE: "github.com/elastic/elastic-agent/testing/integration/leak"
4343
command: |
4444
buildkite-agent artifact download build/distributions/** . --step 'packaging-windows'
45-
.buildkite/scripts/integration-tests.ps1 fleet true TestLongRunningAgentForLeaks
45+
.buildkite/scripts/steps/integration_tests_tf.ps1 fleet true
4646
artifact_paths:
4747
- build/**
4848
- build/diagnostics/**
@@ -57,10 +57,10 @@ steps:
5757
depends_on:
5858
- packaging-windows
5959
env:
60-
TEST_LONG_RUNNING: "true"
60+
TEST_PACKAGE: "github.com/elastic/elastic-agent/testing/integration/leak"
6161
command: |
6262
buildkite-agent artifact download build/distributions/** . --step 'packaging-windows'
63-
.buildkite/scripts/integration-tests.ps1 fleet true TestLongRunningAgentForLeaks
63+
.buildkite/scripts/steps/integration_tests_tf.ps1 fleet true
6464
artifact_paths:
6565
- build/**
6666
- build/diagnostics/**
@@ -74,10 +74,10 @@ steps:
7474
- label: "Ubuntu:2404:amd64:sudo"
7575
depends_on: packaging-ubuntu-x86-64
7676
env:
77-
TEST_LONG_RUNNING: "true"
77+
TEST_PACKAGE: "github.com/elastic/elastic-agent/testing/integration/leak"
7878
command: |
7979
buildkite-agent artifact download build/distributions/** . --step 'packaging-ubuntu-x86-64'
80-
.buildkite/scripts/steps/integration_tests_tf.sh fleet true TestLongRunningAgentForLeaks
80+
.buildkite/scripts/steps/integration_tests_tf.sh fleet true
8181
artifact_paths:
8282
- build/**
8383
- build/diagnostics/**
@@ -99,7 +99,7 @@ steps:
9999
- packaging-windows
100100
command: |
101101
buildkite-agent artifact download build/distributions/** . --step 'packaging-windows'
102-
.buildkite/scripts/integration-tests.ps1 {{matrix}} true
102+
.buildkite/scripts/steps/integration_tests_tf.ps1 {{matrix}} true
103103
artifact_paths:
104104
- build/**
105105
- build/diagnostics/**
@@ -125,7 +125,7 @@ steps:
125125
- packaging-windows
126126
command: |
127127
buildkite-agent artifact download build/distributions/** . --step 'packaging-windows'
128-
.buildkite/scripts/integration-tests.ps1 {{matrix}} false
128+
.buildkite/scripts/steps/integration_tests_tf.ps1 {{matrix}} false
129129
artifact_paths:
130130
- build/**
131131
- build/diagnostics/**
@@ -144,7 +144,7 @@ steps:
144144
- packaging-windows
145145
command: |
146146
buildkite-agent artifact download build/distributions/** . --step 'packaging-windows'
147-
.buildkite/scripts/integration-tests.ps1 {{matrix}} true
147+
.buildkite/scripts/steps/integration_tests_tf.ps1 {{matrix}} true
148148
artifact_paths:
149149
- build/**
150150
- build/diagnostics/**
@@ -170,7 +170,7 @@ steps:
170170
- packaging-windows
171171
command: |
172172
buildkite-agent artifact download build/distributions/** . --step 'packaging-windows'
173-
.buildkite/scripts/integration-tests.ps1 {{matrix}} false
173+
.buildkite/scripts/steps/integration_tests_tf.ps1 {{matrix}} false
174174
artifact_paths:
175175
- build/**
176176
- build/diagnostics/**

‎.buildkite/integration.pipeline.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -159,7 +159,7 @@ steps:
159159
TEST_INTEG_AUTH_GCP_DATACENTER: "us-central1-a"
160160
command: |
161161
buildkite-agent artifact download "build/distributions/**" . $BUILDKITE_BUILD_ID
162-
.buildkite/scripts/steps/integration_tests.sh serverless integration:single TestLogIngestionFleetManaged #right now, run a single test in serverless mode as a sort of smoke test, instead of re-running the entire suite
162+
.buildkite/scripts/steps/integration_tests.sh serverless integration:testServerless
163163
artifact_paths:
164164
- "build/TEST-**"
165165
- "build/diagnostics/*"

‎.buildkite/scripts/buildkite-integration-tests.ps1‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
param(
2+
[string]$GROUP_NAME,
3+
[string]$TEST_SUDO
4+
)
5+
6+
if (-not $GROUP_NAME)
7+
{
8+
Write-Error "Error: Specify the group name: buildkite-integration-tests.ps1 <group_name> <true|false>"
9+
exit 1
10+
}
11+
12+
if (-not $TEST_SUDO)
13+
{
14+
Write-Error "Error: Specify the test sudo: buildkite-integration-tests.ps1 <group_name> <true|false>"
15+
exit 1
16+
}
17+
18+
if (-not $env:TEST_PACKAGE)
19+
{
20+
$env:TEST_PACKAGE = "github.com/elastic/elastic-agent/testing/integration"
21+
}
22+
23+
# TODO: make is not available on Windows yet hence we cannot use make install-gotestsum
24+
go install gotest.tools/gotestsum
25+
gotestsum --version
26+
27+
$env:TEST_BINARY_NAME = "elastic-agent"
28+
# Parsing version.go. Will be simplified here: https://github.com/elastic/ingest-dev/issues/4925
29+
$AGENT_VERSION = (Get-Content version/version.go | Select-String -Pattern 'const defaultBeatVersion =' | ForEach-Object { $_ -replace '.*?"(.*?)".*', '$1' })
30+
$env:AGENT_VERSION = $AGENT_VERSION + "-SNAPSHOT"
31+
32+
Write-Output "~~~ Agent version: $env:AGENT_VERSION"
33+
$env:SNAPSHOT = $true
34+
35+
Write-Host "~~~ Running integration tests as $env:USERNAME"
36+
37+
$osInfo = (Get-CimInstance Win32_OperatingSystem).Caption + " " + (Get-CimInstance Win32_OperatingSystem).OSArchitecture -replace " ", "_"
38+
$root_suffix = ""
39+
if ($TEST_SUDO -eq "true")
40+
{
41+
$root_suffix = "_sudo"
42+
}
43+
$fully_qualified_group_name = "${GROUP_NAME}${root_suffix}_${osInfo}"
44+
$outputXML = "build/${fully_qualified_group_name}.integration.xml"
45+
$outputJSON = "build/${fully_qualified_group_name}.integration.out.json"
46+
47+
$TestsExitCode = 0
48+
49+
try
50+
{
51+
Write-Output "~~~ Integration tests: $GROUP_NAME as user: $env:USERNAME"
52+
# -test.timeout=2h0m0s is set because some tests normally take up to 45 minutes.
53+
# This 2-hour timeout provides enough room for future, potentially longer tests,
54+
# while still enforcing a reasonable upper limit on total execution time.
55+
# See: https://pkg.go.dev/cmd/go#hdr-Testing_flags
56+
$gotestArgs = @("-tags=integration", "-test.shuffle=on", "-test.timeout=2h0m0s", "$env:TEST_PACKAGE", "-v", "-args", "-integration.groups=$GROUP_NAME", "-integration.sudo=$TEST_SUDO")
57+
& gotestsum --no-color -f standard-quiet --junitfile-hide-skipped-tests --junitfile "${outputXML}" --jsonfile "${outputJSON}" -- @gotestArgs
58+
$TestsExitCode = $LASTEXITCODE
59+
60+
if ($TestsExitCode -ne 0)
61+
{
62+
Write-Output "^^^ +++"
63+
Write-Output "Integration tests failed"
64+
}
65+
}
66+
finally
67+
{
68+
if (Test-Path $outputXML)
69+
{
70+
# Install junit2html if not installed
71+
go install github.com/alexec/junit2html@latest
72+
Get-Content $outputXML | junit2html > "build/TEST-report.html"
73+
}
74+
else
75+
{
76+
Write-Output "Cannot generate HTML test report: $outputXML not found"
77+
}
78+
}
79+
80+
exit $TestsExitCode

‎.buildkite/scripts/buildkite-integration-tests.sh‎

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,9 @@
33

44
GROUP_NAME=$1
55
TEST_SUDO=$2
6-
TEST_NAME_PATTERN=${3:-""}
6+
7+
# Set default TEST_PACKAGE if not already defined in env or argument
8+
: "${TEST_PACKAGE:="github.com/elastic/elastic-agent/testing/integration"}"
79

810
if [ -z "$GROUP_NAME" ]; then
911
echo "Error: Specify the group name: sudo-integration-tests.sh [group_name]" >&2
@@ -52,18 +54,22 @@ outputXML="build/${fully_qualified_group_name}.integration.xml"
5254
outputJSON="build/${fully_qualified_group_name}.integration.out.json"
5355

5456
echo "~~~ Integration tests: ${GROUP_NAME}"
55-
GOTEST_ARGS=(-tags integration -test.shuffle on -test.timeout 2h0m0s)
56-
if [ -n "$TEST_NAME_PATTERN" ]; then
57-
GOTEST_ARGS+=(-run="${TEST_NAME_PATTERN}")
58-
fi
59-
GOTEST_ARGS+=("github.com/elastic/elastic-agent/testing/integration" -v -args "-integration.groups=${GROUP_NAME}" "-integration.sudo=${TEST_SUDO}" "-integration.fips=${FIPS:-false}")
60-
57+
# -test.timeout=2h0m0s is set because some tests normally take up to 45 minutes.
58+
# This 2-hour timeout provides enough room for future, potentially longer tests,
59+
# while still enforcing a reasonable upper limit on total execution time.
60+
# See: https://pkg.go.dev/cmd/go#hdr-Testing_flags
61+
GOTEST_ARGS=(-tags integration -test.shuffle on -test.timeout 2h0m0s "${TEST_PACKAGE}" -v -args "-integration.groups=${GROUP_NAME}" "-integration.sudo=${TEST_SUDO}" "-integration.fips=${FIPS:-false}")
6162
set +e
6263
TEST_BINARY_NAME="elastic-agent" AGENT_VERSION="${AGENT_VERSION}" SNAPSHOT=true \
6364
gotestsum --no-color -f standard-quiet --junitfile-hide-skipped-tests --junitfile "${outputXML}" --jsonfile "${outputJSON}" -- "${GOTEST_ARGS[@]}"
6465
TESTS_EXIT_STATUS=$?
6566
set -e
6667

68+
if [[ $TESTS_EXIT_STATUS -ne 0 ]]; then
69+
echo "^^^ +++"
70+
echo "Integration tests failed"
71+
fi
72+
6773
if [ -f "$outputXML" ]; then
6874
go install github.com/alexec/junit2html@latest
6975
junit2html < "$outputXML" > build/TEST-report.html

‎.buildkite/scripts/integration-tests.ps1‎

Lines changed: 0 additions & 70 deletions
This file was deleted.

‎.buildkite/scripts/steps/integration_tests_tf.ps1‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
param (
2+
[string]$GROUP_NAME,
3+
[string]$TEST_SUDO
4+
)
5+
6+
Write-Output "~~~ Preparing environment"
7+
8+
$PSVersionTable.PSVersion
9+
10+
. "$PWD\.buildkite\scripts\steps\ess.ps1"
11+
12+
# Read package version from .package-version file
13+
$PACKAGE_VERSION = Get-Content .package-version -ErrorAction SilentlyContinue
14+
if ($PACKAGE_VERSION) {
15+
$PACKAGE_VERSION = "${PACKAGE_VERSION}-SNAPSHOT"
16+
}
17+
18+
Write-Output "~~~ Building test binaries"
19+
& mage build:testBinaries
20+
if ($LASTEXITCODE -ne 0) {
21+
Write-Output "^^^ +++"
22+
Write-Output "Failed to build test binaries"
23+
exit 1
24+
}
25+
26+
$TestsExitCode = 0
27+
try {
28+
Write-Output "~~~ Running integration tests"
29+
# Get-Ess-Stack will start the ESS stack if it is a BK retry, otherwise it will retrieve ESS stack metadata
30+
Get-Ess-Stack -StackVersion $PACKAGE_VERSION
31+
& "$PWD\.buildkite\scripts\buildkite-integration-tests.ps1" $GROUP_NAME $TEST_SUDO
32+
$TestsExitCode = $LASTEXITCODE
33+
if ($TestsExitCode -ne 0)
34+
{
35+
Write-Output "^^^ +++"
36+
Write-Output "Integration tests failed"
37+
}
38+
} finally {
39+
# ess_down will destroy the ESS stack if tf state file is found, aka if this is a BK retry
40+
ess_down
41+
}
42+
43+
exit $TestsExitCode

‎.buildkite/scripts/steps/integration_tests_tf.sh‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,7 @@ asdf install
99

1010
GROUP_NAME=$1
1111
TEST_SUDO=$2
12-
# NOTE: This argument is not used in this script, but is declared to show that it can be set
13-
# and passed down to downstream scripts where it may be used.
14-
TEST_NAME_PATTERN=${3:-""}
12+
1513
if [ -z "$GROUP_NAME" ]; then
1614
echo "Error: Specify the group name: integration_tests_tf.sh [group_name]" >&2
1715
exit 1
@@ -37,7 +35,7 @@ mage build:testBinaries
3735
if [[ "${BUILDKITE_RETRY_COUNT}" -gt 0 ]]; then
3836
echo "~~~ The steps is retried, starting the ESS stack again"
3937
trap 'ess_down' EXIT
40-
ess_up $OVERRIDE_STACK_VERSION || echo "Failed to start ESS stack" >&2
38+
ess_up $OVERRIDE_STACK_VERSION || (echo -e "^^^ +++\nFailed to start ESS stack")
4139
preinstall_fleet_packages
4240
else
4341
# For the first run, we start the stack in the start_ess.sh step and it sets the meta-data

‎docs/test-framework-dev-guide.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,10 +131,13 @@ We pass a `-test.count` flag along with the name match
131131
We pass a `-test.run` flag along with the names of the tests we want to run in OR
132132
`GOTEST_FLAGS="-test.run ^(TestStandaloneUpgrade|TestFleetManagedUpgrade)$" mage integration:test`
133133

134+
##### Run Serverless tests
135+
The test framework includes a smoke test suite to check elastic-agent in a serverless environment. The suite can be run via the `integration:TestServerless` mage target.
136+
134137
##### Run Extended Runtime Leak Test
135138
The test framework includes a "long running" test to check for resource leaks and stability.
136139
The runtime of the test can be set via the `LONG_TEST_RUNTIME` environment variable.
137-
The test itself can be run via the `integration:TestLongRunningAgentForLeaks` mage target.
140+
The test itself can be run via the `integration:TestForResourceLeaks` mage target.
138141

139142
##### Limitations
140143
Due to the way the parameters are passed to `devtools.GoTest` the value of the environment variable

0 commit comments

Comments
 (0)