GIT-62: Enable --dry-run mode

GIT-62: fix linter issues

Author: harshanarayana

docs/design/dry-run-mode.md

@@ -0,0 +1,83 @@
+# Dry Run Mode
+
+As your test suits get bigger and complicated over the period of time, it is essential that the toolings used for creating tests provide an easy way to identify and list the tests being
+processed as part of your framework when invoked with certain arguments. And this listing needs
+to be quick and clean in order to enable quick turn around time of test development. This requirement bring in the need to introduce a `dry-run` behavior into the `e2e-framework`.
+
+## Unit Of Test
+
+Go treats each function starting with `Testxxx` as the Test unit. However, the same is not entirely true in case of the `e2e-framework`. This introduces dynamic tests that are generated during the runtime programmatically for each assessment of each feature.
+
+From the perspective of the `e2e-framework`, the Unit of test is an `Assessment` that actually performs the assertion of an
+expected behavior or state of the system. These assessments are run as a sub-test of the main test identified by the function
+`Testxxx`. All framework specific behaviors built around this fundamental test unit of `Assessment`.
+
+## Why not use `test.list` from `go test` ?
+
+The `test.list` is a great way to run the dry-run equivalent behavior. However, it is not easily extendable into the core of `e2e-framework` as
+there are framework specific behavior such as `setup` and `teardown` workflows. 
+
+That, in conjunction with how the `test.list` works, it is not possible to extract information such as the `assessments` in the feature using the `test.list` mode brings the need to introduce a framework specific `dry-run` mode that can work well with `test.list` while providing all
+the framework specific benefits of how the Tests to be processed can be listed
+
+## `--dry-run` mode
+`e2e-framework` adds a new CLI flag that can be used while invoking the test called `--dry-run`. This works in conjunction with `test.list` to provide the following behavior. 
+
+1. When the `--dry-run` mode is invoked No Setup/Teardown workflows are processed
+2. Will display the Assessments as individual tests like they would be processed if not invoked with `--dry-run` mode
+3. Skip all pre-post actions around the Before/After Features or Before/After Tests
+
+When tests are invoked with `-test.list` argument, the `--dry-run` mode is automatically switched to enabled to make sure setup/teardown as well as the pre-post actions can be skipped.
+
+## Example Output with `--dry-run`
+```bash
+❯ go test . -test.v -args --dry-run
+=== RUN   TestPodBringUp
+=== RUN   TestPodBringUp/Feature_One
+=== RUN   TestPodBringUp/Feature_One/Create_Nginx_Deployment_1
+=== RUN   TestPodBringUp/Feature_One/Wait_for_Nginx_Deployment_1_to_be_scaled_up
+=== RUN   TestPodBringUp/Feature_Two
+=== RUN   TestPodBringUp/Feature_Two/Create_Nginx_Deployment_2
+=== RUN   TestPodBringUp/Feature_Two/Wait_for_Nginx_Deployment_2_to_be_scaled_up
+--- PASS: TestPodBringUp (0.00s)
+    --- PASS: TestPodBringUp/Feature_One (0.00s)
+        --- PASS: TestPodBringUp/Feature_One/Create_Nginx_Deployment_1 (0.00s)
+        --- PASS: TestPodBringUp/Feature_One/Wait_for_Nginx_Deployment_1_to_be_scaled_up (0.00s)
+    --- PASS: TestPodBringUp/Feature_Two (0.00s)
+        --- PASS: TestPodBringUp/Feature_Two/Create_Nginx_Deployment_2 (0.00s)
+        --- PASS: TestPodBringUp/Feature_Two/Wait_for_Nginx_Deployment_2_to_be_scaled_up (0.00s)
+PASS
+ok  	sigs.k8s.io/e2e-framework/examples/parallel_features	0.353s
+```
+
+```bash
+❯ go test . -test.v -args --dry-run --assess "Deployment 1"
+=== RUN   TestPodBringUp
+=== RUN   TestPodBringUp/Feature_One
+=== RUN   TestPodBringUp/Feature_One/Create_Nginx_Deployment_1
+=== RUN   TestPodBringUp/Feature_One/Wait_for_Nginx_Deployment_1_to_be_scaled_up
+=== RUN   TestPodBringUp/Feature_Two
+=== RUN   TestPodBringUp/Feature_Two/Create_Nginx_Deployment_2
+    env.go:425: Skipping assessment "Create Nginx Deployment 2": name not matched
+=== RUN   TestPodBringUp/Feature_Two/Wait_for_Nginx_Deployment_2_to_be_scaled_up
+    env.go:425: Skipping assessment "Wait for Nginx Deployment 2 to be scaled up": name not matched
+--- PASS: TestPodBringUp (0.00s)
+    --- PASS: TestPodBringUp/Feature_One (0.00s)
+        --- PASS: TestPodBringUp/Feature_One/Create_Nginx_Deployment_1 (0.00s)
+        --- PASS: TestPodBringUp/Feature_One/Wait_for_Nginx_Deployment_1_to_be_scaled_up (0.00s)
+    --- PASS: TestPodBringUp/Feature_Two (0.00s)
+        --- SKIP: TestPodBringUp/Feature_Two/Create_Nginx_Deployment_2 (0.00s)
+        --- SKIP: TestPodBringUp/Feature_Two/Wait_for_Nginx_Deployment_2_to_be_scaled_up (0.00s)
+PASS
+ok  	sigs.k8s.io/e2e-framework/examples/parallel_features	0.945s
+```
+
+## Example with `-test.list`
+```bash
+❯ go test . -test.v -test.list ".*" -args
+TestPodBringUp
+ok  	sigs.k8s.io/e2e-framework/examples/parallel_features	0.645s
+```
+
+As you can see from the above two examples, the output of the two commands are not really the same. Using `--dry-run` gives you a more framework specific behavior of how the tests are going to be processed in comparison to `-test.list`
+

examples/dry_run/README.md

@@ -0,0 +1,54 @@
+# Dry Run of Test Features
+
+This directory contains the example of how to run the test features in `dry-run` mode using framework specific flags.
+
+# Run Tests with flags
+
+These test cases can be executed using the normal `go test` command by passing the right arguments
+
+```bash
+go test -v . -args --dry-run
+```
+
+With the output generated as following.
+
+```bash
+=== RUN   TestDryRunOne
+=== RUN   TestDryRunOne/F1
+=== RUN   TestDryRunOne/F1/Assessment_One
+=== RUN   TestDryRunOne/F2
+=== RUN   TestDryRunOne/F2/Assessment_One
+=== RUN   TestDryRunOne/F2/Assessment_Two
+--- PASS: TestDryRunOne (0.00s)
+    --- PASS: TestDryRunOne/F1 (0.00s)
+        --- PASS: TestDryRunOne/F1/Assessment_One (0.00s)
+    --- PASS: TestDryRunOne/F2 (0.00s)
+        --- PASS: TestDryRunOne/F2/Assessment_One (0.00s)
+        --- PASS: TestDryRunOne/F2/Assessment_Two (0.00s)
+=== RUN   TestDryRunTwo
+=== RUN   TestDryRunTwo/F1
+=== RUN   TestDryRunTwo/F1/Assessment_One
+--- PASS: TestDryRunTwo (0.00s)
+    --- PASS: TestDryRunTwo/F1 (0.00s)
+        --- PASS: TestDryRunTwo/F1/Assessment_One (0.00s)
+PASS
+ok  	sigs.k8s.io/e2e-framework/examples/dry_run	0.618s
+```
+
+Without the `--dry-run` mode you will see the additional log `Do not run this when in dry-run mode` getting printed onto your terminal.
+
+In order to integrate this into the `test.list`, please run the following
+
+```bash
+go test -v -list .
+```
+
+Which generates the output as following
+
+```bash
+TestDryRunOne
+TestDryRunTwo
+ok  	sigs.k8s.io/e2e-framework/examples/dry_run	0.375s
+```
+
+To understand the difference in Output, please refer to the [Design Document](../../docs/design/dry-run-mode.md)

examples/dry_run/dry_run_test.go

@@ -0,0 +1,60 @@
+/*
+Copyright 2021 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package dry_run
+
+import (
+	"context"
+	"testing"
+
+	"k8s.io/klog/v2"
+	"sigs.k8s.io/e2e-framework/pkg/envconf"
+	"sigs.k8s.io/e2e-framework/pkg/features"
+)
+
+func TestDryRunOne(t *testing.T) {
+	f1 := features.New("F1").
+		Assess("Assessment One", func(ctx context.Context, t *testing.T, c *envconf.Config) context.Context {
+			// Perform Some assessment
+			return ctx
+		}).Feature()
+
+	f2 := features.New("F2").
+		Setup(func(ctx context.Context, t *testing.T, c *envconf.Config) context.Context {
+			klog.Info("Do not run this when in dry-run mode")
+			return ctx
+		}).
+		Assess("Assessment One", func(ctx context.Context, t *testing.T, c *envconf.Config) context.Context {
+			// Perform Some assessment
+			return ctx
+		}).
+		Assess("Assessment Two", func(ctx context.Context, t *testing.T, c *envconf.Config) context.Context {
+			// Perform Some assessment
+			return ctx
+		}).Feature()
+
+	testEnv.TestInParallel(t, f1, f2)
+}
+
+func TestDryRunTwo(t *testing.T) {
+	f1 := features.New("F1").
+		Assess("Assessment One", func(ctx context.Context, t *testing.T, c *envconf.Config) context.Context {
+			// Perform Some assessment
+			return ctx
+		}).Feature()
+
+	testEnv.Test(t, f1)
+}

examples/dry_run/main_test.go

@@ -0,0 +1,36 @@
+/*
+Copyright 2021 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package dry_run
+
+import (
+	"os"
+	"testing"
+
+	"sigs.k8s.io/e2e-framework/pkg/env"
+	"sigs.k8s.io/e2e-framework/pkg/envconf"
+)
+
+var (
+	testEnv env.Environment
+)
+
+func TestMain(m *testing.M) {
+	cfg, _ := envconf.NewFromFlags()
+	testEnv = env.NewWithConfig(cfg)
+
+	os.Exit(testEnv.Run(m))
+}

examples/parallel_features/parallel_features_test.go

@@ -24,7 +24,6 @@ import (
 	appsv1 "k8s.io/api/apps/v1"
 	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	log "k8s.io/klog/v2"
 
 	"sigs.k8s.io/e2e-framework/klient/k8s"
 	"sigs.k8s.io/e2e-framework/klient/wait"
@@ -43,7 +42,6 @@ var (
 
 func TestMain(m *testing.M) {
 	cfg, _ := envconf.NewFromFlags()
-	log.InfoS("Args", "flag", cfg)
 	testEnv = env.NewWithConfig(cfg)
 
 	clusterName = envconf.RandomName("kind-parallel", 16)

pkg/env/action.go

@@ -21,6 +21,7 @@ import (
 	"fmt"
 	"testing"
 
+	"k8s.io/klog/v2"
 	"sigs.k8s.io/e2e-framework/pkg/envconf"
 	"sigs.k8s.io/e2e-framework/pkg/internal/types"
 )
@@ -52,6 +53,10 @@ type action struct {
 func (a *action) runWithT(ctx context.Context, cfg *envconf.Config, t *testing.T) (context.Context, error) {
 	switch a.role {
 	case roleBeforeTest, roleAfterTest:
+		if cfg.DryRunMode() {
+			klog.V(2).Info("Skipping execution of roleBeforeTest and roleAfterTest due to framework being in dry-run mode")
+			return ctx, nil
+		}
 		for _, f := range a.testFuncs {
 			if f == nil {
 				continue
@@ -74,6 +79,10 @@ func (a *action) runWithT(ctx context.Context, cfg *envconf.Config, t *testing.T
 func (a *action) runWithFeature(ctx context.Context, cfg *envconf.Config, t *testing.T, fi types.Feature) (context.Context, error) {
 	switch a.role {
 	case roleBeforeFeature, roleAfterFeature:
+		if cfg.DryRunMode() {
+			klog.V(2).Info("Skipping execution of roleBeforeFeature and roleAfterFeature due to framework being in dry-run mode")
+			return ctx, nil
+		}
 		for _, f := range a.featureFuncs {
 			if f == nil {
 				continue
@@ -92,6 +101,10 @@ func (a *action) runWithFeature(ctx context.Context, cfg *envconf.Config, t *tes
 }
 
 func (a *action) run(ctx context.Context, cfg *envconf.Config) (context.Context, error) {
+	if cfg.DryRunMode() {
+		klog.V(2).InfoS("Skipping processing of action due to framework being in dry-run mode")
+		return ctx, nil
+	}
 	for _, f := range a.funcs {
 		if f == nil {
 			continue

pkg/env/action_test.go

@@ -35,6 +35,7 @@ func TestAction_Run(t *testing.T) {
 	}{
 		{
 			name: "single-step action",
+			cfg:  &envconf.Config{},
 			ctx:  context.WithValue(context.TODO(), &ctxTestKeyString{}, 1),
 			setup: func(ctx context.Context, cfg *envconf.Config) (val int, err error) {
 				funcs := []types.EnvFunc{
@@ -50,6 +51,7 @@ func TestAction_Run(t *testing.T) {
 		},
 		{
 			name: "multi-step action",
+			cfg:  &envconf.Config{},
 			ctx:  context.WithValue(context.TODO(), &ctxTestKeyString{}, 1),
 			setup: func(ctx context.Context, cfg *envconf.Config) (val int, err error) {
 				funcs := []types.EnvFunc{
@@ -69,6 +71,7 @@ func TestAction_Run(t *testing.T) {
 		},
 		{
 			name: "read from context",
+			cfg:  &envconf.Config{},
 			ctx:  context.WithValue(context.TODO(), &ctxTestKeyString{}, 1),
 			setup: func(ctx context.Context, cfg *envconf.Config) (val int, err error) {
 				funcs := []types.EnvFunc{