Update to use gotestfmt in complement CI (#281)

* Update to use gotestfmt in complement CI

* Add failing test

* set pipefail

* Use a bash shell

* Fix test; update README

Author: kegsay

.github/workflows/ci.yaml

@@ -82,6 +82,7 @@ jobs:
         working-directory: homeserver
 
       - run: docker build -t homeserver -f dockerfiles/${{ matrix.homeserver }}.Dockerfile dockerfiles/
-      - run: go test -p 2 -v -tags "${{ matrix.tags }}" ./tests/...
+      - run: set -o pipefail && go test -p 2 -v -json -tags "${{ matrix.tags }}" ./tests/... 2>&1 | gotestfmt
+        shell: bash # required for pipefail to be A Thing. pipefail is required to stop gotestfmt swallowing non-zero exit codes
         env:
           COMPLEMENT_BASE_IMAGE: homeserver

README.md

@@ -76,6 +76,15 @@ If you're looking to run against a custom Dockerfile, it must meet the following
 - The homeserver needs to assume dockerfile `CMD` or `ENTRYPOINT` instructions will be run multiple times.
 - The homeserver can use the CA certificate mounted at /ca to create its own TLS cert (see [Complement PKI](README.md#complement-pki)).
 
+### Getting prettier output
+
+The default output isn't particularly nice to read. You can use [gotestfmt](https://github.com/haveyoudebuggedit/gotestfmt)
+to make this very pretty. To do so, ask for JSON output via `go test -json` then pipe the output to `gotestfmt`.
+If you are doing this in CI, make sure to `set -o pipefail` or else test failures will NOT result in a non-zero exit code
+as `gotestfmt`'s exit code (0 as it successfully printed) will replace the previous commands exit code.
+See Complement's [Github Actions](https://github.com/matrix-org/complement/blob/master/.github/workflows/ci.yaml) file
+for an example of how to do this correctly.
+
 ## Writing tests
 
 To get started developing Complement tests, see [the onboarding documentation](ONBOARDING.md).

dockerfiles/ComplementCIBuildkite.Dockerfile

@@ -4,10 +4,11 @@
 # This Dockerfile prepares an image with Complement/Docker pre-installed.
 # This allows users of this image to issue `docker build` commands to build their HS
 # and then run Complement against it.
-FROM golang:1.15-buster
+FROM golang:1.17-buster
 RUN echo "deb http://deb.debian.org/debian buster-backports main" > /etc/apt/sources.list.d/complement.list && apt-get update && apt-get install -y libolm3 libolm-dev/buster-backports
 RUN curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh
 ADD https://github.com/matrix-org/complement/archive/master.tar.gz .
 RUN tar -xzf master.tar.gz && cd complement-master && go mod download 
+RUN cd complement-master && go get github.com/haveyoudebuggedit/gotestfmt/v2/cmd/gotestfmt@latest
 
 VOLUME [ "/ca" ]