Split out integration tests into different packages

[ycombinator]

In #8083 (comment), @pchila asked:

Is there a specific reason for the FIPS integration test not to be defined in their own package and run independently from the melting pot that is the integration package right now ?

It's a fair question and can probably be generalized to wondering if we should split out our ever-growing set of tests under the integration package into separate packages.

@pchila since this is your idea, I'll let you expand on any pros/cons so we can then have a discussion about it with the rest of the team. Thanks.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[pchila]

cc @pkoutsovasilis

[pchila]

Splitting the integration package would allow for fewer filters needed to select the e2e tests we need to run, lowering the number of env vars / cli args and limiting the complexity of test selection logic

Furthermore there are multiple type of tests all in the same package with different purposes:

• Serverless Integration tests: runs an e2e test as a smoke test against a serverless project
• Extended runtime leak tests: long-running test to detect aggressive memory leaks while parsing sample logs controlled by TEST_LONG_RUNNING env var
• Beats serverless tests: currently disabled but tests, mage targets and scripts are still in the codebase
• K8s tests: subset of tests run using elastic-agent container images under a kind cluster for all the relevant k8s versions
• Integration tests: e2e elastic-agent tests run against an ECH deployment which tests using various agent setups and different distributions:
  • sudo/non-sudo
  • standalone/managed
  • fips/non-fips

Among the Integration tests we have multiple areas being tested defined as groups (mostly for parallel run and partial isolation):

• container
• deb
• default
• fleet
• fleet-airgapped
• fleet-airgapped-privileged
• fleet-endpoint-security
• fleet-privileged
• fleet-upgrade-to-pr-build
• fqdn
• install-uninstall
• rpm
• standalone-upgrade
• upgrade
• upgrade-flavor

An easy way to split all the tests contained in github.com/elastic/elastic-agent/testing/integration package would be to separate by test type first, for example creating subpackages depending on test setup/type:

• github.com/elastic/elastic-agent/testing/integration/k8s
• github.com/elastic/elastic-agent/testing/integration/runtime
• github.com/elastic/elastic-agent/testing/integration/serverless
• github.com/elastic/elastic-agent/testing/integration/ech
  and have a package for common tests that can be referenced by all the other packages to reuse testcases or utility functions
• github.com/elastic/elastic-agent/testing/integration/common/tests
• github.com/elastic/elastic-agent/testing/integration/common/utils

This would allow to customize and control setup/teardown and cli options using a different TestMain per package and reduce the usage of filters and env variable.
For including (mostly functional, with minimal setup) tests into multiple subpackages with different setups a simple t.Run() call would suffice in order to make explicit that a given common testcase is being run under another package test setup.

Such a setup should bring clarity (it should be obvious which tests belong to which setup), place specific tests into specific packages while still allowing for test reuse under multiple setups and would fit the logical separation between types we already have in buildkite scripts but implemented in a more idiomatic way grouping similar tests under a dedicated package.

Of course this is just one possible split, happy to get feedback about it. I would not split by group/function/feature at first since it may be unnecessary depending on the outcome of the split by type.

Any thoughts @ycombinator @pkoutsovasilis ?

[ycombinator]

@pchila your proposal for the split sounds good to me. Thanks for laying it out in detail. I agree that splitting by type of test at the top level seems to be the right approach as a) all tests of a type likely have the same configuration/setup requirements and b) that such requirements likely differ considerably between types.

I'm going to take a stab at separating out the ECH tests into their own package first, as I'm working on FIPS tests at the moment (and these fall under the ECH type). Let's see how that goes and then we can iterate from there and tease out other types into their own packages or take an altogether different approach to the split (though, again, I tend to agree that splitting by type at the top-level is the way to go).

[pkoutsovasilis]

@pchila @ycombinator I really like the proposal of the issue and the discussion above, so thank you both 🙏 I agree let's take a manageable, in terms of blast radius, first stab and if the outcome seems beneficial I can also join forces with you two and "migrate" the k8s integration tests 🙂