Re-run flaky E2E tests up to 3 times

Also, report slowest tests and use standard verbose logging for maximum
insight into progress.

This feels a little like giving up, but realistically right now ~80% of
our E2E test runs fail. When they do, we mostly just re-run the entire
failed suites. Each suite run takes ~20-25 mins to complete.

This argument will only re-run the specific test that failed,
automatically. This should result in less time spent running and waiting
for E2Es overall.

Signed-off-by: Nic Cope <[email protected]>

Author: negz

Earthfile

@@ -71,11 +71,18 @@ e2e:
     WITH DOCKER --load crossplane-e2e/crossplane:latest=(+image --CROSSPLANE_VERSION=v0.0.0-e2e)
       # TODO(negz:) Set GITHUB_ACTIONS=true and use RUN --raw-output when
       # https://github.com/earthly/earthly/issues/4143 is fixed.
-      RUN gotestsum --no-color=false --format testname --junitfile e2e-tests.xml --raw-command go tool test2json -t -p E2E ./e2e -test.v ${FLAGS}
+      RUN gotestsum \
+        --rerun-fails=3 \
+      	--no-color=false \
+	--format standard-verbose \
+	--jsonfile e2e-tests.json \
+	--junitfile e2e-tests.xml \
+	--raw-command go tool test2json -t -p E2E ./e2e -test.v ${FLAGS}
     END
   FINALLY
     SAVE ARTIFACT --if-exists e2e-tests.xml AS LOCAL _output/tests/e2e-tests.xml
   END
+  RUN gotestsum tool slowest --jsonfile e2e-tests.json
 
 # hack builds Crossplane, and deploys it to a kind cluster. It runs in your
 # local environment, not a container. The kind cluster will keep running until

test/e2e/README.md

@@ -89,21 +89,6 @@ $ apk add kubectl
 # Now you can use kubectl to access the cluster
 ```
 
-### Test Logging
-
-The E2E tests are run using [`gotestsum`] that helps output the test results in
-a format that integrates easily with CI systems. The downside though is that
-test logging statements, like `t.Logf("foo")`, are not streamed in real time and
-they are only included in the output if there is a failure.
-
-To change this behavior when running the E2E tests locally, you can update the
-`Earthfile` to call `gotestsum` with a format like `--format standard-verbose`
-instead, for example:
-
-```Dockerfile
-RUN gotestsum --no-color=false --format standard-verbose ...
-```
-
 ### Running Tests Directly
 
 The E2E tests are typically run via `earthly` for convenience and consistency.