Update OSDK scorecard for 4.7

Author: adellape

modules/osdk-about-scorecard-tool.adoc

@@ -0,0 +1,34 @@
+// Module included in the following assemblies:
+//
+// * operators/operator_sdk/osdk-scorecard.adoc
+
+[id="osdk-about-scorecard_{context}"]
+= About the scorecard tool
+
+While the Operator SDK `bundle validate` subcommand can validate local bundle directories and remote bundle images for content and structure, you can use the `scorecard` subcommand to run tests on your Operator based on a configuration file and test images. These tests are implemented within test images that are configured and constructed to be executed by the scorecard.
+
+The scorecard assumes it is run with access to a configured Kubernetes cluster, such as {product-title}. The scorecard runs each test within a pod, from which pod logs are aggregated and test results are sent to the console. The scorecard has built-in basic and Operator Lifecycle Manager (OLM) tests and also provides a means to execute custom test definitions.
+
+.Scorecard workflow
+. Create all resources required by any related custom resources (CRs) and the Operator
+. Create a proxy container in the deployment of the Operator to record calls to the API server and run tests
+. Examine parameters in the CRs
+
+The scorecard tests make no assumptions as to the state of the Operator being tested. Creating Operators and CRs for an Operators are beyond the scope of the scorecard itself. Scorecard tests can, however, create whatever resources they require if the tests are designed for resource creation.
+
+The `scorecard` subcommand has the following syntax:
+
+[source,terminal]
+----
+$ operator-sdk scorecard <bundle_dir_or_image> [flags]
+----
+
+The scorecard requires a positional argument for either the on-disk path to
+your Operator bundle or the name of a bundle image.
+
+For further information about the flags, run:
+
+[source,terminal]
+----
+$ operator-sdk scorecard -h
+----

modules/osdk-running-scorecard.adoc

@@ -2,160 +2,63 @@
 //
 // * operators/operator_sdk/osdk-scorecard.adoc
 
+:osdk_ver: v1.3.0
+
 [id="osdk-scorecard-config_{context}"]
 = Scorecard configuration
 
-The scorecard tool uses a configuration file that allows you to configure internal plug-ins, as well as several global configuration options.
-
-[id="osdk-scorecard-config-file_{context}"]
-== Configuration file
-
-The default location for the scorecard tool configuration is the `<project_dir>/.osdk-scorecard.*`. The following is an example of a YAML-formatted configuration file:
+The scorecard tool uses a configuration that allows you to configure internal plug-ins, as well as several global configuration options. Tests are driven by a configuration file named `config.yaml`, which is generated by the `make bundle` command, located in your `bundle/` directory:
 
-.Scorecard configuration file
-[source,yaml]
+[source,terminal]
 ----
-scorecard:
-  output: json
-  plugins:
-    - basic: <1>
-        cr-manifest:
-          - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml"
-          - "deploy/crds/cache.example.com_v1alpha1_memcachedrs_cr.yaml"
-    - olm: <2>
-        cr-manifest:
-          - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml"
-          - "deploy/crds/cache.example.com_v1alpha1_memcachedrs_cr.yaml"
-        csv-path: "deploy/olm-catalog/memcached-operator/0.0.3/memcached-operator.v0.0.3.clusterserviceversion.yaml"
+./bundle
+...
+└── tests
+    └── scorecard
+        └── config.yaml
 ----
-<1> `basic` tests configured to test two custom resources (CRs).
-<2> `olm` tests configured to test two CRs.
-
-Configuration methods for global options take the following priority, highest to lowest:
-
-Command arguments (if available) -> configuration file -> default
-
-The configuration file must be in YAML format. As the configuration file might be extended to allow configuration of all `operator-sdk` subcommands in the future, the scorecard configuration must be under a `scorecard` subsection.
-
-[NOTE]
-====
-Configuration file support is provided by the `viper` package. For more info on how `viper` configuration works, see the link:https://github.com/spf13/viper/blob/master/README.md[README].
-====
-
-[id="osdk-scorecard-config-args_{context}"]
-== Command arguments
-
-While most of the scorecard tool configuration is done using a configuration file, you can also use the following arguments:
-
-.Scorecard tool arguments
-[cols="25%,20%,55%",options="header"]
-|===
-|Flag |Type |Description
-
-|`--bundle`, `-b`
-|string
-|The path to a bundle directory used for the bundle validation test.
-
-|`--config`
-|string
-|The path to the scorecard configuration file. The default is `<project_dir>/.osdk-scorecard`. The file type and extension must be `.yaml`. If a configuration file is not provided or found at the default location, the scorecard exits with an error.
-
-|`--output`, `-o`
-|string
-|Output format. Valid options are `text` and `json`. The default format is `text`, which is designed to be a human readable format. The `json` format uses the JSON schema output format used for plug-ins defined later.
-
-|`--kubeconfig`, `-o`
-|string
-|The path to the `kubeconfig` file. It sets the `kubeconfig` for internal plug-ins.
-
-|`--version`
-|string
-|The version of scorecard to run. The default and only valid option is `v1alpha2`.
-
-|`--selector`, `-l`
-|string
-|The label selector to filter tests on.
-
-|`--list`, `-L`
-|bool
-|If `true`, only print the test names that would be run based on selector filtering.
-|===
-
-[id="osdk-scorecard-config-file-options_{context}"]
-== Configuration file options
-
-The scorecard configuration file provides the following options:
-
-.Scorecard configuration file options
-[cols="25%,20%,55%",options="header"]
-|===
-|Option |Type |Description
-
-|`bundle`
-|string
-|Equivalent of the `--bundle` flag. Operator Lifecycle Manager (OLM) bundle directory path, when specified, runs bundle validation.
-
-|`output`
-|string
-|Equivalent of the `--output` flag. If this option is defined by both the configuration file and the flag, the flag value takes priority.
-
-|`kubeconfig`
-|string
-|Equivalent of the `--kubeconfig` flag. If this option is defined by both the configuration file and the flag, the flag value takes priority.
-
-|`plugins`
-|array
-|An array of plug-in names.
-|===
 
-[id="osdk-scorecard-config-plugins_{context}"]
-=== Basic and OLM plug-ins
+.Example scorecard configuration file
+[source,yaml,subs="attributes+"]
+----
+kind: Configuration
+apiversion: scorecard.operatorframework.io/v1alpha3
+metadata:
+  name: config
+stages:
+- parallel: true
+  tests:
+  - image: quay.io/operator-framework/scorecard-test:{osdk_ver}
+    entrypoint:
+    - scorecard-test
+    - basic-check-spec
+    labels:
+      suite: basic
+      test: basic-check-spec-test
+  - image: quay.io/operator-framework/scorecard-test:{osdk_ver}
+    entrypoint:
+    - scorecard-test
+    - olm-bundle-validation
+    labels:
+      suite: olm
+      test: olm-bundle-validation-test
+----
 
-The scorecard supports the internal `basic` and `olm` plug-ins, which are configured by a `plugins` section in the configuration file.
+The configuration file defines each test that scorecard can execute. The
+following fields of the scorecard configuration file define the test as follows:
 
-.Plug-in options
-[cols="25%,20%,55%",options="header"]
+[cols="3,7",options="header"]
 |===
-|Option |Type |Description
-
-|`cr-manifest`
-|[]string
-|The path(s) for CRs being tested. Required if `olm-deployed` is unset or `false`.
-
-|`csv-path`
-|string
-|The path to the cluster service version (CSV) for the Operator. Required for OLM tests or if `olm-deployed` is set to `true`.
-
-|`olm-deployed`
-|bool
-|Indicates that the CSV and relevant CRDs have been deployed onto the cluster by OLM.
-
-|`kubeconfig`
-|string
-|The path to the `kubeconfig` file. If both the global `kubeconfig` and this field are set, this field is used for the plug-in.
-
-|`namespace`
-|string
-|The namespace to run the plug-ins in. If unset, the default specified by the `kubeconfig` file is used.
-
-|`init-timeout`
-|int
-|Time in seconds until a timeout during initialization of the Operator.
+|Configuration field |Description
 
-|`crds-dir`
-|string
-|The path to the directory containing CRDs that must be deployed to the cluster.
+|`image`
+|Test container image name that implements a test
 
-|`namespaced-manifest`
-|string
-|The manifest file with all resources that run within a namespace. By default, the scorecard combines the `service_account.yaml`, `role.yaml`, `role_binding.yaml`, and `operator.yaml` files from the `deploy` directory into a temporary manifest to use as the namespaced manifest.
+|`entrypoint`
+|Command and arguments that are invoked in the test image to execute a test
 
-|`global-manifest`
-|string
-|The manifest containing required resources that run globally (not namespaced). By default, the scorecard combines all CRDs in the `crds-dir` directory into a temporary manifest to use as the global manifest.
+|`labels`
+|Scorecard-defined or custom labels that select which tests to run
 |===
 
-[NOTE]
-====
-Currently, using the scorecard with a CSV does not permit multiple CR manifests to be set through the CLI, configuration file, or CSV annotations. You must tear down your Operator in the cluster, re-deploy, and re-run the scorecard for each CR that is tested.
-====
+:!osdk_ver: