[9.0] (backport #8484) [test] split up k8s integration tests

[mergify]

What does this PR do?

• Split Kubernetes integration tests into a dedicated testing/integration/k8s package.
• Removed the need for the kubernetes build tag; Kubernetes tests are now always compiled as part of the normal integration tag.
• Introduced a TestMain entrypoint in the new package, enabling centralized setup logic.
• Renamed mage targets to be consistent with other test oriented mage targets:
  • integration:TestKubernetes (formerly integration:Kubernetes)
  • integration:TestKubernetesMatrix (formerly integration:KubernetesMatrix)
  • integration:TestKubernetesSingle (formerly integration:KubernetesSingle)
• Updated all references in Buildkite pipeline and documentation accordingly.
• Performed mechanical moves of existing Kubernetes test files into the new package (journald_test.go, kubernetes_agent_service_test.go, kubernetes_agent_standalone_test.go, otel_helm_test.go, and their respective testdata).

Why is it important?

This PR continues the ongoing effort of modularizing the ever-growing integration package as part of #8087. Specifically, it addresses the Kubernetes test split described in #8412.

Having Kubernetes tests in a dedicated package offers multiple benefits:

• Better test ownership and isolation of Kubernetes-specific test logic.
• With this PR and the introduction of the dedicated TestMain, we unlock further future improvements for developer velocity:
  • We can load container images into the kind cluster only once per test run.
  • We can render Kustomize YAML manifests once upfront, avoiding per-test Kustomize render calls which are prone to network-transient failures.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

No user-facing disruption; internal test structure change only.

How to test this PR locally

EXTERNAL=true SNAPSHOT=true PACKAGES=docker DOCKER_VARIANTS=basic PLATFORMS=linux/arm64 mage package

INSTANCE_PROVISIONER="kind" TEST_PLATFORMS="kubernetes/arm64/1.33.0/basic" mage integration:TestKubernetes

Related issues

• Relates Split out integration tests into different packages #8087
• Closes Split up Kubernetes integration tests into dedicated package #8412

  ---

  This is an automatic backport of pull request [test] split up k8s integration tests #8484 done by Mergify.

[mergify]

Cherry-pick of dd52e90 has failed:

On branch mergify/bp/9.0/pr-8484
Your branch is up to date with 'origin/9.0'.

You are currently cherry-picking commit dd52e906c.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .buildkite/scripts/buildkite-k8s-integration-tests.sh
	modified:   .buildkite/scripts/steps/k8s-extended-tests.sh
	modified:   docs/test-framework-dev-guide.md
	modified:   pkg/testing/define/define.go
	modified:   pkg/testing/define/define_all.go
	deleted:    pkg/testing/define/define_kubernetes.go
	modified:   pkg/testing/define/define_local.go
	new file:   testing/integration/common.go
	new file:   testing/integration/k8s/common.go
	renamed:    testing/integration/kubernetes_agent_service_test.go -> testing/integration/k8s/kubernetes_agent_service_test.go
	new file:   testing/integration/k8s/main_test.go
	renamed:    testing/integration/otel_helm_test.go -> testing/integration/k8s/otel_helm_test.go
	renamed:    testing/integration/testdata/connectors.agent.yml -> testing/integration/k8s/testdata/connectors.agent.yml
	renamed:    testing/integration/testdata/java_app.yaml -> testing/integration/k8s/testdata/java_app.yaml
	renamed:    testing/integration/testdata/k8s.hints.redis.yaml -> testing/integration/k8s/testdata/k8s.hints.redis.yaml
	deleted:    testing/integration/kubernetes_agent_standalone_test.go
	modified:   testing/integration/logs_ingestion.go
	modified:   testing/integration/otel_test.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   go.mod
	both modified:   magefile.go
	both modified:   testing/integration/beat_receivers_test.go
	deleted by us:   testing/integration/k8s/journald_test.go
	both modified:   testing/integration/k8s/kubernetes_agent_standalone_test.go
	deleted by us:   testing/integration/k8s/testdata/journald-input.yml
	deleted by us:   testing/integration/k8s/testdata/journald-otel.yml

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: a8498b4

Failed CI Steps

• arm:sudo: upgrade

cc @pkoutsovasilis