iter8 gen exp subcommand (iter8-tools#1107)

* iter8 gen exp

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* your first experiment and install

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* fixing mkdocs.yml

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* removing images not needed atm

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* readded illustration and fixed gen exp

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* fixing readme

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* fixing assert flags

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* updated hub folder

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* required url value

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* fixed mock qs

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* improving go doc

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* fixing exp docs

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* fixig mkdocs

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* fixing k8s.go doc

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* fixing k8s.go

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* fixing go doc

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* fixing app and id related documentation

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

* fixing docs for k8s.go

Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>

Author: sriumcp

.github/workflows/tests.yaml

@@ -23,10 +23,10 @@ jobs:
       run: |
         export COVERAGE=$(go tool cover -func coverage.out | grep total | awk '{print substr($3, 1, length($3)-1)}')
         echo "code coverage is at ${COVERAGE}"
-        if [ 1 -eq "$(echo "${COVERAGE} > 66.0" | bc)" ]; then \
-          echo "all good... coverage is above 66.0%"; 
+        if [ 1 -eq "$(echo "${COVERAGE} > 67.0" | bc)" ]; then \
+          echo "all good... coverage is above 67.0%"; 
         else \
-          echo "not good... coverage is not above 66.0%";
+          echo "not good... coverage is not above 67.0%";
           exit 1
         fi
     - name: Upload coverage to Codecov

README.md

@@ -11,43 +11,45 @@
 
 ### Kubernetes-friendly metrics-driven <strong>experiments</strong> and <strong>safe rollouts</strong>. 
 
-#### Built for DevOps/SRE/MLOps/data science teams.
-
 ## Use Cases
 
 1.  Load testing with SLOs
-2.  A/B(/n) testing with business reward metrics
-3.  SLOs with metrics from any backend
-4.  Traffic mirroring
-5.  User segmentation
-6.  Session affinity
-7.  Gradual rollout
+2.  A/B(/n) testing for improving business value with each release of app/ML model
+3.  Safe rollout for multi-cluster and edge
+4.  Traffic mirroring experiments
 
-The traffic engineering use-cases (4 - 7 above) are achieved by using Iter8 along with a Kubernetes service mesh or ingress.
+The traffic mirroring use-case is achieved by using Iter8 along with a Kubernetes service mesh or ingress that supports mirroring.
 
 ## Quick Start
 
 ### 1. Install Iter8
 Install Iter8 using [Go 1.16+](https://golang.org/) as follows.
 ```shell
 go install github.com/iter8-tools/iter8@latest
-# you can now run iter8 (from your gopath bin/ directory)
 ```
+You can now run `iter8` (from your gopath bin/ directory)
 
-## 2. Download experiment
-Download the `load-test` experiment folder from Iter8 hub as follows.
+## 2. Download experiment chart
+Download the `load-test` experiment chart from Iter8 hub as follows.
 
 ```shell
 iter8 hub -e load-test
 ```
 
-## 3. Run experiment
-Iter8 experiments are specified using the `experiment.yaml` file. The `iter8 run` command reads this file, runs the specified experiment, and writes the results of the experiment into the `result.yaml` file.
+This creates a local folder called `load-test` containing the chart.
 
-Run the experiment you downloaded above as follows.
+## 3. Generate `experiment.yaml`
+Generate the `experiment.yaml` file which specifies your load test experiment.
 
 ```shell
 cd load-test
+iter8 gen exp --set url=https://example.com
+```
+
+## 4. Run experiment
+The iter8 run command reads the `experiment.yaml` file, runs the specified experiment, and writes the results of the experiment into the `result.yaml` file. Run the experiment as follows.
+
+```shell
 iter8 run
 ```
 
@@ -81,6 +83,20 @@ iter8 report -o text
 
 Congratulations! :tada: You completed your first Iter8 experiment.
 
-## [Documentation](https://iter8.tools)
+## Documentation
+Iter8 documentation is available at https://iter8.tools.
+
+## Contributing
+We are delighted that you want to contribute to Iter8! 💖
+
+As you get started, you are in the best position to give us feedback on areas of
+our project that we need help with including:
+
+* Problems found during setup of Iter8
+* Gaps in our quick start tutorial and other documentation
+* Bugs in our test and automation scripts
+
+If anything doesn't make sense, or doesn't work when you run it, please open a
+bug report and let us know!
 
-## [Contributing](https://iter8.tools/latest/contributing/overview/)
+See [here](https://iter8.tools/latest/contributing/overview/) for information about ways to contribute, Iter8 community meetings, finding an issue, asking for help, pull-request lifecycle, and more.

base/assess.go

@@ -1,3 +1,4 @@
+// Package base provides the core definitions and primitives for Iter8 experiment and experimeent tasks.
 package base
 
 import (
@@ -17,6 +18,7 @@ type assessInputs struct {
 // assessTask enables assessment of versions
 type assessTask struct {
 	taskMeta
+	// With contains the inputs for the assessTask
 	With assessInputs `json:"with" yaml:"with"`
 }
 

base/assess_test.go

@@ -9,7 +9,7 @@ import (
 func TestMakeWrongTask(t *testing.T) {
 	ts := &TaskSpec{
 		taskMeta: taskMeta{
-			Task: stringPointer(CollectTaskName),
+			Task: StringPointer(CollectTaskName),
 		},
 		With: map[string]interface{}{
 			"hello": "world",
@@ -25,7 +25,7 @@ func TestMakeAssess(t *testing.T) {
 	// should succeed
 	ts := &TaskSpec{
 		taskMeta: taskMeta{
-			Task: stringPointer(AssessTaskName),
+			Task: StringPointer(AssessTaskName),
 		},
 	}
 	task, err := MakeAssess(ts)
@@ -35,7 +35,7 @@ func TestMakeAssess(t *testing.T) {
 	// incorrect with clause
 	// should fail
 	ts = &TaskSpec{
-		taskMeta: taskMeta{Task: stringPointer(AssessTaskName)},
+		taskMeta: taskMeta{Task: StringPointer(AssessTaskName)},
 		With: map[string]interface{}{
 			"SLOs": "hello world",
 		},
@@ -51,7 +51,7 @@ func TestRunAssess(t *testing.T) {
 	// should succeed
 	ts := &TaskSpec{
 		taskMeta: taskMeta{
-			Task: stringPointer(AssessTaskName),
+			Task: StringPointer(AssessTaskName),
 		},
 	}
 	task, _ := MakeAssess(ts)
@@ -64,7 +64,7 @@ func TestRunAssess(t *testing.T) {
 	// assess with an SLO
 	// should succeed
 	ts = &TaskSpec{
-		taskMeta: taskMeta{Task: stringPointer(AssessTaskName)},
+		taskMeta: taskMeta{Task: StringPointer(AssessTaskName)},
 		With: map[string]interface{}{
 			"SLOs": []SLO{{
 				Metric:     "m",

base/collect.go

@@ -27,9 +27,11 @@ type version struct {
 	URL string `json:"url" yaml:"url" validate:"required,url"`
 }
 
-// HTTP status code within this range is considered an error
+// errorRange has lower and upper limits for HTTP status codes. HTTP status code within this range is considered an error
 type errorRange struct {
+	// Lower end of the range
 	Lower *int `json:"lower" yaml:"lower" validate:"required_without=Upper"`
+	// Upper end of the range
 	Upper *int `json:"upper" yaml:"upper" validate:"required_without=Lower"`
 }
 
@@ -53,11 +55,12 @@ type collectInputs struct {
 	ErrorRanges []errorRange `json:"errorRanges" yaml:"errorRanges"`
 	// Percentiles are the latency percentiles computed by this task. Percentile values have a single digit precision (i.e., rounded to one decimal place). Default value is {50.0, 75.0, 90.0, 95.0, 99.0, 99.9,}.
 	Percentiles []float64 `json:"percentiles" yaml:"percentiles" validate:"unique,dive,gte=0.0,lte=100.0"`
-	// A non-empty list of version values.
+	// VersionInfo is a non-empty list of version values.
 	VersionInfo []*version `json:"versionInfo" yaml:"versionInfo" validate:"required,notallnil"`
 }
 
 const (
+	// CollectTaskName is the name of this task which performs load generation and metrics collection.
 	CollectTaskName            = "gen-load-and-collect-metrics"
 	defaultQPS                 = float32(8)
 	defaultNumRequests         = int64(100)
@@ -247,7 +250,7 @@ func (t *collectTask) resultForVersion(j int) (*fhttp.HTTPRunnerResults, error)
 	return ifr, err
 }
 
-// GetName returns the name of the assess task
+// GetName returns the name of the collect task
 func (t *collectTask) GetName() string {
 	return CollectTaskName
 }
@@ -348,7 +351,7 @@ func (t *collectTask) Run(exp *Experiment) error {
 			in.MetricsInfo[m] = MetricMeta{
 				Description: "mean of observed latency values",
 				Type:        GaugeMetricType,
-				Units:       stringPointer("msec"),
+				Units:       StringPointer("msec"),
 			}
 			in.MetricValues[i][m] = append(in.MetricValues[i][m], 1000.0*fm[i].DurationHistogram.Avg)
 
@@ -357,7 +360,7 @@ func (t *collectTask) Run(exp *Experiment) error {
 			in.MetricsInfo[m] = MetricMeta{
 				Description: "standard deviation of observed latency values",
 				Type:        GaugeMetricType,
-				Units:       stringPointer("msec"),
+				Units:       StringPointer("msec"),
 			}
 			in.MetricValues[i][m] = append(in.MetricValues[i][m], 1000.0*fm[i].DurationHistogram.StdDev)
 
@@ -366,7 +369,7 @@ func (t *collectTask) Run(exp *Experiment) error {
 			in.MetricsInfo[m] = MetricMeta{
 				Description: "minimum of observed latency values",
 				Type:        GaugeMetricType,
-				Units:       stringPointer("msec"),
+				Units:       StringPointer("msec"),
 			}
 			in.MetricValues[i][m] = append(in.MetricValues[i][m], 1000.0*fm[i].DurationHistogram.Min)
 
@@ -375,7 +378,7 @@ func (t *collectTask) Run(exp *Experiment) error {
 			in.MetricsInfo[m] = MetricMeta{
 				Description: "maximum of observed latency values",
 				Type:        GaugeMetricType,
-				Units:       stringPointer("msec"),
+				Units:       StringPointer("msec"),
 			}
 			in.MetricValues[i][m] = append(in.MetricValues[i][m], 1000.0*fm[i].DurationHistogram.Max)
 
@@ -385,7 +388,7 @@ func (t *collectTask) Run(exp *Experiment) error {
 				in.MetricsInfo[m] = MetricMeta{
 					Description: fmt.Sprintf("%0.1f-th percentile of observed latency values", p.Percentile),
 					Type:        GaugeMetricType,
-					Units:       stringPointer("msec"),
+					Units:       StringPointer("msec"),
 				}
 				in.MetricValues[i][m] = append(in.MetricValues[i][m], 1000.0*p.Value)
 			}
@@ -395,7 +398,7 @@ func (t *collectTask) Run(exp *Experiment) error {
 			in.MetricsInfo[m] = MetricMeta{
 				Description: "Latency Histogram",
 				Type:        HistogramMetricType,
-				Units:       stringPointer("msec"),
+				Units:       StringPointer("msec"),
 				XMin:        float64Pointer(xMin),
 				XMax:        float64Pointer(xMax),
 				NumBuckets:  intPointer(20),
@@ -414,7 +417,7 @@ func (t *collectTask) Run(exp *Experiment) error {
 	return nil
 }
 
-// sum up two vectors
+// vectorSum produces the sum of two equi-length vectors
 func vectorSum(a []float64, b []float64) ([]float64, error) {
 	if len(a) != len(b) {
 		log.Logger.Error("vector lengths do not match: ", len(a), " != ", len(b))

base/collect_test.go

@@ -11,7 +11,7 @@ import (
 func TestMakeWrongCollectTask(t *testing.T) {
 	ts := &TaskSpec{
 		taskMeta: taskMeta{
-			Task: stringPointer(AssessTaskName),
+			Task: StringPointer(AssessTaskName),
 		},
 		With: map[string]interface{}{
 			"hello": "world",
@@ -26,7 +26,7 @@ func TestMakeCollect(t *testing.T) {
 	// collect without version info ... should fail
 	ts := &TaskSpec{
 		taskMeta: taskMeta{
-			Task: stringPointer(CollectTaskName),
+			Task: StringPointer(CollectTaskName),
 		},
 	}
 	task, err := MakeCollect(ts)
@@ -36,7 +36,7 @@ func TestMakeCollect(t *testing.T) {
 	// collect task with only nil versions... should fail
 	ct := &collectTask{
 		taskMeta: taskMeta{
-			Task: stringPointer(CollectTaskName),
+			Task: StringPointer(CollectTaskName),
 		},
 		With: collectInputs{
 			VersionInfo: []*version{nil, nil},
@@ -52,7 +52,7 @@ func TestMakeCollect(t *testing.T) {
 	// valid collect task... should succeed
 	ct = &collectTask{
 		taskMeta: taskMeta{
-			Task: stringPointer(CollectTaskName),
+			Task: StringPointer(CollectTaskName),
 		},
 		With: collectInputs{
 			VersionInfo: []*version{nil, {
@@ -73,7 +73,7 @@ func TestRunCollect(t *testing.T) {
 	// valid collect task... should succeed
 	ct := &collectTask{
 		taskMeta: taskMeta{
-			Task: stringPointer(CollectTaskName),
+			Task: StringPointer(CollectTaskName),
 		},
 		With: collectInputs{
 			VersionInfo: []*version{{

base/experiment.go

@@ -13,16 +13,21 @@ import (
 // Experiment specification and result
 type Experiment struct {
 	// Tasks is the sequence of tasks that constitute this experiment
-	Tasks  []TaskSpec        `json:"tasks" yaml:"tasks"`
+	Tasks []TaskSpec `json:"tasks" yaml:"tasks"`
+	// Result is the current results from this experiment.
+	// The experiment may not have completed in which case results may be partial.
 	Result *ExperimentResult `json:"result" yaml:"result"`
 }
 
 // Task is an object that can be run
 type Task interface {
+	// Run this task
 	Run(exp *Experiment) error
+	// Get the name of this task
 	GetName() string
 }
 
+// GetIf returns the condition if any which determines whether of not if this task needs to run
 func GetIf(t Task) *string {
 	var jsonBytes []byte
 	var tm taskMeta
@@ -116,14 +121,19 @@ type SLO struct {
 }
 
 type taskMeta struct {
+	// Task is the name of the task
 	Task *string `json:"task,omitempty" yaml:"task,omitempty" validate:"required_without=Run,excluded_with=Run"`
-	Run  *string `json:"run,omitempty" yaml:"run,omitempty" validate:"required_without=Task,excluded_with=Task"`
-	If   *string `json:"if,omitempty" yaml:"if,omitempty"`
+	// Run is the script used in a run task
+	// Specify either Task or Run but not both
+	Run *string `json:"run,omitempty" yaml:"run,omitempty" validate:"required_without=Task,excluded_with=Task"`
+	// If is the condition used to determine if this task needs to run.
+	If *string `json:"if,omitempty" yaml:"if,omitempty"`
 }
 
 // TaskSpec has information needed to construct a Task
 type TaskSpec struct {
 	taskMeta
+	// With contains the inputs for this task.
 	With map[string]interface{} `json:"with,omitempty" yaml:"with,omitempty"`
 }
 

base/log/log.go

@@ -1,3 +1,4 @@
+// Package log provides primitives for logging.
 package log
 
 import (
@@ -8,20 +9,21 @@ import (
 	"github.com/spf13/viper"
 )
 
-// Iter8Logger inherits all methods from logrus logger
+// Iter8Logger inherits all methods from logrus logger.
 type Iter8Logger struct {
 	*logrus.Logger
 }
 
-// StackTrace is the trace from external components like a shell scripts run by an Iter8 task
+// StackTrace is the trace from external components like a shell scripts run by an Iter8 task.
 type StackTrace struct {
+	// Trace is the raw trace
 	Trace string
 }
 
-// Logger to be used in all of Iter8
+// Logger to be used in all of Iter8.
 var Logger *Iter8Logger
 
-// Initialize logger
+// init initializes the logger.
 func init() {
 	Logger = &Iter8Logger{logrus.New()}
 	Logger.SetFormatter(&logrus.TextFormatter{
@@ -39,17 +41,20 @@ func init() {
 	SetLogLevel(ll)
 }
 
+// SetLogLevel to a given logrus log level.
 func SetLogLevel(ll logrus.Level) {
 	Logger.SetLevel(ll)
 }
 
-// WithStackTrace yields a log entry with a formatted stack trace field embedded in it
+// WithStackTrace yields a log entry with a formatted stack trace field embedded in it.
 func (l *Iter8Logger) WithStackTrace(t string) *logrus.Entry {
 	return l.WithField("stack-trace", &StackTrace{
 		Trace: t,
 	})
 }
 
+// String processes the stack trace by prefixing each line of the trace with ::Trace::.
+// This enables other tools like grep to easily filter out these traces if needed.
 func (st *StackTrace) String() string {
 	out := "below ... \n"
 	scanner := bufio.NewScanner(strings.NewReader(st.Trace))