Merge pull request #7824 from pohly/e2e-tests-labels

e2e-tests.md: update description of labels

Author: k8s-ci-robot

contributors/devel/sig-testing/e2e-tests.md

@@ -94,21 +94,21 @@ kubetest --up
 # Run all tests
 kubetest --test
 
-# Run tests matching the regex "\[Feature:Performance\]" against a local cluster
+# Run tests which have been labeled with "Feature:Performance" against a local cluster
 # Specify "--provider=local" flag when running the tests locally
-kubetest --test --test_args="--ginkgo.focus=\[Feature:Performance\]" --provider=local
+kubetest --test --test_args='--ginkgo.label-filter=Feature:Performance' --provider=local
 
 # Conversely, exclude tests that match the regex "Pods.*env"
-kubetest --test --test_args="--ginkgo.skip=Pods.*env"
+kubetest --test --test_args='--ginkgo.skip=Pods.*env'
 
-# Exclude tests tha require a certain minimum version of the kubelet
-kubetest --test --test_args="--ginkgo.skip=\[MinimumKubeletVersion:1.20\]"
+# Exclude tests that require a certain minimum version of the kubelet
+kubetest --test --test_args='--ginkgo.label-filter=!MinimumKubeletVersion:1.20'
 
 # Run tests in parallel, skip any that must be run serially
-GINKGO_PARALLEL=y kubetest --test --test_args="--ginkgo.skip=\[Serial\]"
+GINKGO_PARALLEL=y kubetest --test --test_args='--ginkgo.label-filter=!Serial'
 
 # Run tests in parallel, skip any that must be run serially and keep the test namespace if test failed
-GINKGO_PARALLEL=y kubetest --test --test_args="--ginkgo.skip=\[Serial\] --delete-namespace-on-failure=false"
+GINKGO_PARALLEL=y kubetest --test --test_args='--ginkgo.label-filter=!Serial --delete-namespace-on-failure=false'
 
 # Flags can be combined, and their actions will take place in this order:
 # --build, --up, --test, --down
@@ -189,11 +189,15 @@ if any specs are pending.
 --ginkgo.focus="": If set, ginkgo will only run specs that match this regular
 expression.
 
---ginkgo.noColor="n": If set to "y", ginkgo will not use color in the output
-
 --ginkgo.skip="": If set, ginkgo will only run specs that do not match this
 regular expression.
 
+--ginkgo.label-filter="": If set, select tests based on their labels as described under
+"Spec Labels" in https://onsi.github.io/ginkgo/#filtering-specs. This can focus
+on tests and exclude others in a single parameter without using regular expressions.
+
+--ginkgo.noColor="n": If set to "y", ginkgo will not use color in the output
+
 --ginkgo.trace=false: If set, default reporter prints out the full stack trace
 when a failure occurs
 
@@ -283,7 +287,7 @@ First, compile the E2E test suite with additional compiler flags
 make WHAT=test/e2e/e2e.test GOGCFLAGS="all=-N -l" GOLDFLAGS=""
 ```
 
-Then set the env var `E2E_TEST_DEBUG_TOOL=delve` and then run the test with `./hack/gingko.sh` instead of `kubetest`, you should see the delve command line prompt
+Then set the env var `E2E_TEST_DEBUG_TOOL=delve` and then run the test with `./hack/ginkgo.sh` instead of `kubetest`, you should see the delve command line prompt
 
 ```sh
 E2E_TEST_DEBUG_TOOL=delve ./hack/ginkgo-e2e.sh --ginkgo.focus="sig-storage.*csi-hostpath.*Dynamic.PV.*default.fs.*provisioning.should.provision.storage.with.snapshot.data.source" --allowed-not-ready-nodes=10
@@ -462,7 +466,7 @@ kubetest --up
 #
 # You can target Feature:MasterUpgrade or Feature:ClusterUpgrade
 cd ../kubernetes
-kubetest --provider=gke --test --check-version-skew=false --test_args="--ginkgo.focus=\[Feature:MasterUpgrade\]"
+kubetest --provider=gke --test --check-version-skew=false --test_args="--ginkgo.label-filter=Feature:MasterUpgrade"
 
 # Run old tests with new kubectl
 cd ../kubernetes_old
@@ -523,8 +527,10 @@ where `->` means upgrading; container_vm (cvm) and gci are image names.
 
 ## Kinds of tests
 
-Tests can be labeled with any of the following labels, in order of increasing
-precedence (that is, each label listed below supersedes the previous ones):
+Tests can be labeled. Labels appear with square brackets inside the test names
+(the traditional approach) *and* are Ginkgo v2 labels (since Kubernetes v1.29).
+Available labels in order of increasing precedence (that is, each label listed
+below supersedes the previous ones):
 
   - If a test has no labels, it is expected to run fast (under five minutes), be
 able to be run in parallel, and be consistent.
@@ -555,8 +561,12 @@ monitored closely in CI. `[Flaky]` tests are by default not run, unless a
 
   - `[Feature:.+]`: If a test has non-default requirements to run or targets
 some non-core functionality, and thus should not be run as part of the standard
-suite, it receives a `[Feature:.+]` label, e.g. `[Feature:Performance]` or
-`[Feature:Ingress]`. `[Feature:.+]` tests are not run in our core suites,
+suite, it receives a `[Feature:.+]` label. This non-default requirement could
+be some special cluster setup (e.g. `Feature:IPv6DualStack` indicates that the
+cluster must support dual-stack pod and service networks) or that the test has
+special behavior that makes it unsuitable for a normal test run (e.g.
+`Feature:PerformanceDNS` marks a test that stresses cluster DNS performance
+with many services). `[Feature:.+]` tests are not run in our core suites,
 instead running in custom suites. If a feature is experimental or alpha and is
 not enabled by default due to being incomplete or potentially subject to
 breaking changes, it does *not* block PR merges, and thus should run in
@@ -577,21 +587,86 @@ to be eligible for this tag. This tag does not supersed any other labels.
   - `[LinuxOnly]`: If a test is known to be using Linux-specific features
 (e.g.: seLinuxOptions) or is unable to run on Windows nodes, it is labeled
 `[LinuxOnly]`. When using Windows nodes, this tag should be added to the
-`skip` argument.
+`skip` argument. This is not using `[Feature:LinuxOnly]` because that
+would have implied changing all CI jobs which skip tests with unknown
+requirements.
 
   - The following tags are not considered to be exhaustively applied, but are
 intended to further categorize existing `[Conformance]` tests, or tests that are
 being considered as candidate for promotion to `[Conformance]` as we work to
 refine requirements:
     - `[Privileged]`: This is a test that requires privileged access
-    - `[Internet]`: This is a test that assumes access to the public internet
     - `[Deprecated]`: This is a test that exercises a deprecated feature
+
+  - For tests that depend on feature gates, the following are set automatically:
     - `[Alpha]`: This is a test that exercises an alpha feature
     - `[Beta]`: This is a test that exercises a beta feature
 
+    Conceptually, these are non-default requirements as defined above under
+    `[Feature:.+]`, but for historic reasons and the sake of brevity they don't
+    have that prefix when embedded in test names. They *do* have that prefix in the
+    Ginkgo v2 label, so use e.g. `--filter-label=Feature: containsAny Alpha` to
+    run them. The normal `--filter-label=Feature: isEmpty` excludes them.
+
+    Note that at the moment, not all jobs filter out tests with `Alpha` or `Beta`
+    requirements like that. Therefore all tests with such a requirement also
+    have to be annotated with a `[Feature]` tag. This restriction will be lifted
+    once migration of jobs to `--filter-label` is completed.
+
 Every test should be owned by a [SIG](/sig-list.md),
 and have a corresponding `[sig-<name>]` label.
 
+## Selecting tests to run
+
+See https://onsi.github.io/ginkgo/#filtering-specs for a general introduction.
+
+Focusing on a specific test by its name is useful when interactively running
+just one or a few related tests. The test name is a concatenation of multiple
+strings. To get a list of all full test names, run:
+
+```console
+$ e2e.test -list-tests
+The following spec names can be used with 'ginkgo run --focus/skip':
+    test/e2e/apimachinery/watchlist.go:41: [sig-api-machinery] API Streaming (aka. WatchList) [Serial] [Feature:WatchList] should be requested when ENABLE_CLIENT_GO_WATCH_LIST_ALPHA is set
+    test/e2e/apimachinery/flowcontrol.go:65: [sig-api-machinery] API priority and fairness should ensure that requests can be classified by adding FlowSchema and PriorityLevelConfiguration
+    test/e2e/apimachinery/flowcontrol.go:190: [sig-api-machinery] API priority and fairness should ensure that requests can't be drowned out (fairness)
+...
+```
+
+Or within the Kubernetes repo:
+
+```console
+$ go test -v ./test/e2e -args -list-tests
+The following spec names can be used with 'ginkgo run --focus/skip':
+    test/e2e/apimachinery/watchlist.go:41: [sig-api-machinery] API Streaming (aka. WatchList) [Serial] [Feature:WatchList] should be requested when ENABLE_CLIENT_GO_WATCH_LIST_ALPHA is set
+...
+```
+
+The same works for other Kubernetes E2E suites, like `e2e_node`.
+
+In Prow jobs, selection by labels is often simpler. See
+[below]((#kinds-of-tests) for documentation of the different labels that are in
+use. A full list of labels used by a specific E2E suite can be obtained with
+`--list-labels`.
+
+A common pattern is to run only tests which have no special cluster setup
+requirements and are not flaky:
+
+    --filter-label='Feature: isEmpty && !Flaky'
+
+Feature owners have to ensure that tests excluded that way from shared CI
+jobs are executed in dedicated jobs (more on CI below):
+
+    --filter-label='Feature: containsAny MyAwesomeFeature'
+
+In jobs that support certain well-known features it is possible to run tests
+which have no special requirements or at least only depend on the supported
+features:
+
+    # Alpha APIs and features enabled, allow tests depending on that as
+    # long as they have no other special requirements.
+    --filter-label='Feature: isSubsetOf Alpha'
+
 ### Viper configuration and hierarchichal test parameters.
 
 The future of e2e test configuration idioms will be increasingly defined using viper, and decreasingly via flags.
@@ -665,6 +740,14 @@ non-`[Disruptive]`, non-`[Flaky]`, non-`[Feature:.+]` tests in parallel.
   - `kubernetes-e2e-<provider>-serial` runs all `[Serial]` and `[Disruptive]`,
 non-`[Flaky]`, non-`[Feature:.+]` tests in serial.
 
+  - `ci-kubernetes-e2e-kind-alpha-features` runs all tests without any special
+    requirements and tests that only have alpha feature gates and API groups
+    as requirement.
+
+  - `ci-kubernetes-e2e-kind-beta-features` runs all tests without any special
+    requirements and tests that only have beta feature gates and API groups
+    as requirement.
+
 We also run non-default tests if the tests exercise general-availability ("GA")
 features that require a special environment to run in, e.g.
 `kubernetes-e2e-gce-scalability` and `kubernetes-kubemark-gce`, which test for