doc/dev,hack/tests: update docs and fix go test configuration (#1629)

* doc/dev,hack/tests: update docs and fix go test configuration

* doc/dev: Include metrics e2e steps in docs

* .travis.yml: remove unused -v make arg

* Apply suggestions from code review

Co-Authored-By: Haseeb Tariq <[email protected]>


Co-authored-by: Lili Cosic <[email protected]>

Author: AlexNPavel

.travis.yml

@@ -71,7 +71,7 @@ jobs:
     # Build and test go
     - <<: *test
       name: Go on OpenShift
-      script: make test/ci-go ARGS="-v"
+      script: make test/ci-go
 
     # Build and test ansible
     - <<: *test

doc/dev/testing/running-the-tests.md

@@ -100,22 +100,15 @@ and the operator images will be built and stored in you local docker registry.
 
 The `make test/e2e/go` command accepts an `ARGS` variable containing flags that will be passed to `go test`:
 
-- `-image` string - Sets the operator test image tag to be built and used in testing. Defaults to "quay.io/example/memcached-operator:v0.0.1"
+- `-image-name` string - Sets the operator test image tag to be built and used in testing. Defaults to "quay.io/example/memcached-operator:v0.0.1"
 - `-local-repo` string - Sets the path to the local SDK repo being tested. Defaults to the path of the SDK repo containing e2e tests. This is useful for testing customized e2e code.
 
 An example of using `ARGS` is in the note below.
 
 **NOTE**: Some of these tests, specifically the ansible (`test/e2e/ansible` and `test/ci-ansible`), helm
-(`test/e2e/helm` and `test/ci-helm`), and CI Go (`test/e2e/ci-go`) tests, only work when the cluster shares the local docker
+(`test/e2e/helm` and `test/ci-helm`), and Go (`test/e2e/go` and `test/e2e/ci-go`) tests, only work when the cluster shares the local docker
 registry, as is the case with `oc cluster up` and `minikube` after running `eval $(minikube docker-env)`.
 
-The E2E go test (`test/e2e/go`) can be run on a remote cluster by specifying an image name using a repo that you are logged into and
-have permission to push to as such:
-
-```sh
-$ make test/e2e/go ARGS="-image=quay.io/example/memcached:e2e-test"
-```
-
 All other tests will run correctly on a remote cluster if `$HOME/.kube/config` points to the remote cluster and your
 `KUBECONFIG` env var is either empty or is set to the path of a kubeconfig for the remote cluster.
 
@@ -127,9 +120,8 @@ during the go tests can cause these cleanups to fail (the ansible and helm E2E t
 always clean up correctly). For example, if a segfault occurs or a user kills the
 testing process, the cleanup functions for the go tests will not run. To manually clean up a test:
 
-1. Delete the CRD (`kubectl delete -f $HOME/projects/example.com/memcached-operator/deploy/crds/cache_v1alpha1_memcached_crd.yaml`).
-2. Delete the created project in `$HOME/projects/example.com/memcached-operator`
-3. Delete the namespaces that the tests run in, which also deletes any resources created within the namespaces. The namespaces start with `memcached-memcached-group` or `main` and are appended with a unix timestamp (seconds since Jan 1 1970). The kubectl command can be used to delete namespaces: `kubectl delete namespace $NAMESPACE`.
+1. Delete the CRD (`kubectl delete crd memcacheds.cache.example.com`).
+2. Delete the namespaces that the tests run in, which also deletes any resources created within the namespaces. The namespaces start with `memcached-memcached-group` or `main` and are appended with a unix timestamp (seconds since Jan 1 1970). The kubectl command can be used to delete namespaces: `kubectl delete namespace $NAMESPACE`.
 
 [travis]: ./travis-build.md
 [quay]: https://quay.io

doc/dev/testing/travis-build.md

@@ -44,24 +44,32 @@ The Go, Ansible, and Helm tests then differ in what tests they run.
     7. Run `scorecard` subcommand and check that expected score matches actual score.
     8. Run `scorecard` subcommand with json output enabled and verify the output.
 4. Run [go e2e tests][go-e2e].
-    1. Use `operator-sdk` to create and configure a new `memcached-operator` project and install the memcached CRD in the cluster.
-    2. Run cluster test (namespace is auto-generated and deleted by test framework).
-        1. Build `memcached-operator` image with `--enable-tests` flag enabled (used in the in-cluster test later).
-        2. Deploy operator and resources to the cluster.
-        3. Run the leader election test.
-            1. Verify that operator deployment is ready.
-            2. Verify that leader configmap specifies 1 leader and that the memcached operator has 2 pods (configuration for this is done in step 4.1).
-            3. Delete current leader and wait for memcached-operator deployment to become ready again.
-            4. Verify that leader configmap specifies 1 leader and that the memcached-operator has 2 pods.
-            5. Verify that the name of the new leader is different from the name of the old leader.
-        4. Run the memcached scale test.
-            1. Create memcached CR specifying a desired cluster size of 3 and wait until memcached cluster is of size 3.
-            2. Increase desired cluster size to 4 and wait until memcached cluster is of size 4.
-    3. Run local test.
-        1. Create new namespace for the test.
-        2. Start operator using `up local` subcommand.
-        3. Run memcached scale test (described in step 4.2.4)
-        4. Delete the test namespace.
+    1. Scaffold a project using `hack/tests/scaffolding/e2e-go-scaffold.sh`
+    2. Build `memcached-operator` image to be used in tests
+    3. Run scaffolded project e2e tests using `operator-sdk up local`
+        1. Run cluster test (namespace is auto-generated and deleted by test framework).
+            1. Deploy operator and required resources to the cluster.
+            2. Run the leader election test.
+                1. Verify that operator deployment is ready.
+                2. Verify that leader configmap specifies 1 leader and that the memcached operator has 2 pods (configuration for this is done in step 4.1).
+                3. Delete current leader and wait for memcached-operator deployment to become ready again.
+                4. Verify that leader configmap specifies 1 leader and that the memcached-operator has 2 pods.
+                5. Verify that the name of the new leader is different from the name of the old leader.
+            3. Run the memcached scale test.
+                1. Create memcached CR specifying a desired cluster size of 3 and wait until memcached cluster is of size 3.
+                2. Increase desired cluster size to 4 and wait until memcached cluster is of size 4.
+            4. Run the memcached metrics test.
+                1. Make sure the metrics Service was created.
+                2. Get metrics via proxy pod and make sure they are present.
+                3. Perform linting of the existing metrics.
+            5. Run the memcached custom resource metrics test.
+                1. Make sure the metrics Service was created.
+                2. Get metrics via proxy pod and make sure they are present.
+                3. Perform linting of the existing metrics.
+                4. Perform checks on each custom resource generated metric and makes sure the name, type, value, labels and metric are correct. 
+        2. Run local test (namespace is auto-generated and deleted by test framework).
+            1. Start operator using `up local` subcommand.
+            2. Run memcached scale test (described in step 4.3.1.3)
     4. Run [TLS library tests][tls-tests].
         1. This test runs multiple simple tests of the operator-sdk's TLS library. The tests run in parallel and each tests runs in its own namespace.
 

hack/tests/e2e-go.sh

@@ -9,4 +9,4 @@ operator-sdk build $IMAGE_NAME
 operator-sdk test local ./test/e2e
 popd
 
-go test ./test/e2e/... -root=. -globalMan=testdata/empty.yaml $1
+go test ./test/e2e/... -root=. -globalMan=testdata/empty.yaml

hack/tests/scaffolding/e2e-go-scaffold.sh

@@ -10,8 +10,31 @@ trap_add 'rm -f $ROOTDIR/go.mod' EXIT
 BASEPROJECTDIR="$(mktemp -d)"
 IMAGE_NAME="quay.io/example/memcached-operator:v0.0.1"
 
+# change IMAGE_NAME in case it is redefined by a flag
+# parse "--image-name value" format
+ORIG_ARGS=("$@")
+while [[ $# -gt 0 ]]
+do
+key="$1"
+case $key in
+    -image-name|--image-name) # "--image-name value" format
+    IMAGE_NAME="$2"
+    shift
+    ;;
+    -image-name=*|--image-name=*) # "--image-name=value" format
+    IMAGE_NAME="${key#*=}"
+    shift
+    ;;
+    *)    # different arg/flag
+    shift
+    ;;
+esac
+done
+
+set -- "${ORIG_ARGS[@]}"
+
 go build -o $BASEPROJECTDIR/scaffold-memcached $ROOTDIR/hack/tests/scaffolding/scaffold-memcached.go
 
 pushd "$BASEPROJECTDIR"
-./scaffold-memcached --local-repo $ROOTDIR --image-name=$IMAGE_NAME --local-image
+./scaffold-memcached --local-repo $ROOTDIR --image-name=$IMAGE_NAME --local-image $@
 popd