Scaffold a requirements.yml file for collection dependencies (#2652)

* Scaffold a requirements.yml file for collection dependencies

Users will now install the operator_sdk.util and community.kubernetes
collections at operator build time rather than them being baked into
the base image. This allows us to put version pins to ensure
compatibility of user Ansible content even when updating the base image.
This should make it safer and easier for users to update to newer
versions of the base image, as well as make it more transparent where
the Ansible content they execute is coming from. It also allows us to
push new features and bugfixes out to users via the collections without
having to continually re-release the base image with new content.

This will be a breaking change for existing users of the collection
however. In order to continue working, users will have to add a
requirements.yml to their projects and add an install step to their
build/Dockerfile.

* Update docs

* Ensure permissions on installed collections are properly set for execution in-cluster

* Update changelog and migration guide

Author: fabianvf

CHANGELOG.md

@@ -18,8 +18,10 @@
 - Add Prometheus metrics support to Helm-based operators. ([#2603](https://github.com/operator-framework/operator-sdk/pull/2603))
 
 ### Changed
-- Ansible scaffolding has been rewritten to be simpler and make use of newer features of Ansible and Molecule.
-    - The Kubernetes modules have migrated to the [Kubernetes Ansible collection](https://github.com/ansible-collections/kubernetes). All scaffolded code now references modules from this collection instead of Ansible Core. No immediate action is required for existing users of the modules from core, though it is recommended they switch to using the collection to continue to get non-critical bugfixes and features. The collection is now installed by default in the base image. ([#2646](https://github.com/operator-framework/operator-sdk/pull/2646))
+- The base image now includes version 0.10.3 of the OpenShift Python client. This should fix hanging in Python3
+- The Kubernetes modules have migrated to the [Kubernetes Ansible collection](https://github.com/ansible-collections/kubernetes). All scaffolded code now references modules from this collection instead of Ansible Core. No immediate action is required for existing users of the modules from core, though it is recommended they switch to using the collection to continue to get non-critical bugfixes and features. To install the collection, users will need to add the install step to their `build/Dockerfile`.  New projects will have a `requirements.yml` scaffolded that includes the `community.kubernetes` collection, as well as the corresponding install step in the `build/Dockerfile`. ([#2646](https://github.com/operator-framework/operator-sdk/pull/2646))
+- **Breaking change** `The operator_sdk.util` collection is no longer installed by default in the base image. Existing projects will need to install it in the `build/Dockerfile`. New projects will have a `requirements.yml` scaffolded that includes the `operator_sdk.util` collection, as well as the corresponding install step in the `build/Dockerfile`. ([#2652](https://github.com/operator-framework/operator-sdk/pull/2652))
+- Ansible scaffolding has been rewritten to be simpler and make use of newer features of Ansible and Molecule. ([#2425](https://github.com/operator-framework/operator-sdk/pull/2425))
     - No longer generates the build/test-framework directory or molecule/test-cluster scenario
     - Adds new `cluster` scenario that can be used to test against an existing cluster
     - There is no longer any Ansible templating done in the `deploy/` directory, any templates used for testing will be located in `molecule/templates/` instead.

cmd/operator-sdk/migrate/cmd.go

@@ -95,8 +95,9 @@ func migrateAnsible() error {
 	}
 
 	dockerfile := ansible.DockerfileHybrid{
-		Watches: true,
-		Roles:   true,
+		Watches:      true,
+		Roles:        true,
+		Requirements: true,
 	}
 	_, err := os.Stat(ansible.PlaybookYamlFile)
 	switch {

cmd/operator-sdk/new/cmd.go

@@ -268,6 +268,7 @@ func doAnsibleScaffold() error {
 		},
 		&ansible.DeployOperator{},
 		&ansible.Travis{},
+		&ansible.RequirementsYml{},
 		&ansible.MoleculeTestLocalMolecule{},
 		&ansible.MoleculeTestLocalPrepare{},
 		&ansible.MoleculeTestLocalVerify{},

doc/ansible/dev/developer_guide.md

@@ -36,6 +36,16 @@ Finally, a user must install the Ansible Kubernetes collection from ansible-gala
 $ ansible-galaxy collection install community.kubernetes
 ```
 
+Alternatively, if you've already initialized your operator, you will have a `requirements.yml`
+file at the top level of your project. This file specifies Ansible dependencies that
+need to be installed for your operator to function. By default it will install the
+`community.kubernetes` collection, which are used to interact with the Kubernetes API, as well
+as the `operator_sdk.util` collection, which provides modules and plugins for operator-specific
+operations. To install the Ansible modules from this file, run
+```bash
+$ ansible-galaxy collection install -r requirements.yml
+```
+
 ### Testing the k8s Ansible modules locally
 
 Sometimes it is beneficial for a developer to run the Ansible code from their
@@ -50,6 +60,7 @@ Create foo-operator/tmp/build/go-test.sh
 Rendering Ansible Galaxy role [foo-operator/roles/Foo]...
 Cleaning up foo-operator/tmp/init
 Create foo-operator/watches.yaml
+Create foo-operator/requirements.yml
 Create foo-operator/deploy/rbac.yaml
 Create foo-operator/deploy/crd.yaml
 Create foo-operator/deploy/cr.yaml
@@ -59,8 +70,8 @@ Initialized empty Git repository in /home/dymurray/go/src/github.com/dymurray/op
 Run git init done
 
 $ cd foo-operator
+$ ansible-galaxy collection install -r requirements.yml
 ```
-
 Modify `roles/Foo/tasks/main.yml` with desired Ansible logic. For this example
 we will create and delete a namespace with the switch of a variable:
 ```yaml
@@ -442,19 +453,6 @@ Please look over the following sections for help debugging an Ansible Operator:
 * [Additional Ansible debug](../user-guide.md#additional-ansible-debug)
 * [Testing Ansible Operators with Molecule](testing_guide.md#testing-ansible-operators-with-molecule)
 
-### Using k8s_status Ansible module with `run --local`
-This section covers the required steps to using the `k8s_status` Ansible module
-with `operator-sdk run --local`. If you are unfamiliar with managing status from
-the Ansible Operator, see the [proposal for user-driven status
-management][manage_status_proposal].
-
-If your operator takes advantage of the `k8s_status` Ansible module and you are
-interested in testing the operator with `operator-sdk run --local`, then
-you will need to install the collection locally.
-
-```sh
-$ ansible-galaxy collection install operator_sdk.util
- ```
 ## Extra vars sent to Ansible
 The extra vars that are sent to Ansible are managed by the operator. The `spec`
 section will pass along the key-value pairs as extra vars.  This is equivalent

doc/ansible/dev/testing_guide.md

@@ -15,13 +15,15 @@ To begin, you sould have:
   without modification. Your top-level project structure should look like this:
     ```
     .
-    ├── build
-    ├── deploy
-    ├── molecule
-    ├── roles
+    ├── build/
+    ├── deploy/
+    ├── molecule/
+    ├── roles/
     ├── playbook.yml (optional)
+    ├── requirements.yml
     └── watches.yaml
     ```
+- The Ansible content specified in `requirements.yml` will also need to be installed. You can install them with `ansible-galaxy collection install -r requirements.yml`
 
 ### Molecule scenarios
 If you look into the `molecule` directory, you will see four directories (`default`, `test-local`, `cluster`, `templates`).

doc/ansible/project_layout.md

@@ -10,4 +10,5 @@ After creating a new operator project using
 | roles/\<kind> | Contains an Ansible Role initialized using [Ansible Galaxy](https://docs.ansible.com/ansible/latest/galaxy/user_guide.html) |
 | build/ | Contains scripts that the `operator-sdk` uses for build and initialization. |
 | watches.yaml | Contains Group, Version, Kind, and the Ansible invocation method. |
+| requirements.yml | contains the Ansible collections and role dependencies to install during build. |
 | molecule/ | Contains [Molecule](https://molecule.readthedocs.io/) scenarios for end-to-end testing of your role and operator |

doc/migration/version-upgrade-guide.md

@@ -973,6 +973,25 @@ With:
 - name: {{your operator name which is the value of metadata.name in this file}}
 ```
 
+#### Migration to Ansible collections
+
+The core Ansible Kubernetes modules have been moved to the [`community.kubernetes` Ansible collection][kubernetes-ansible-collection]. Future development of the modules will occur there, with only critical bugfixes going into the modules in core. Additionally, the `operator_sdk.util` collection is no longer installed by default in the base image. Instead, users should add a `requirements.yml` to their project root, with the following content:
+
+```yaml
+collections:
+  - community.kubernetes
+  - operator_sdk.util
+```
+
+Users should then add the following stages to their `build/Dockerfile`:
+
+```
+COPY requirements.yml ${HOME}/requirements.yml
+RUN ansible-galaxy collection install -r ${HOME}/requirements.yml \
+ && chmod -R ug+rwx ${HOME}/.ansible
+```
+
+
 [legacy-kubebuilder-doc-crd]: https://book-v1.book.kubebuilder.io/beyond_basics/generating_crd.html
 [v0.8.2-go-mod]: https://github.com/operator-framework/operator-sdk/blob/28bd2b0d4fd25aa68e15d928ae09d3c18c3b51da/internal/pkg/scaffold/go_mod.go#L40-L94
 [activating-modules]: https://github.com/golang/go/wiki/Modules#how-to-install-and-activate-module-support
@@ -988,3 +1007,4 @@ With:
 [api-rules]: https://github.com/kubernetes/kubernetes/tree/36981002246682ed7dc4de54ccc2a96c1a0cbbdb/api/api-rules
 [generating-crd]: https://book.kubebuilder.io/reference/generating-crd.html
 [markers]: https://book.kubebuilder.io/reference/markers.html
+[kubernetes-ansible-collection]: https://github.com/ansible-collections/kubernetes

internal/scaffold/ansible/build_dockerfile.go

@@ -46,6 +46,10 @@ func (b *BuildDockerfile) GetInput() (input.Input, error) {
 
 const buildDockerfileAnsibleTmpl = `FROM quay.io/operator-framework/ansible-operator:[[.ImageTag]]
 
+COPY requirements.yml ${HOME}/requirements.yml
+RUN ansible-galaxy collection install -r ${HOME}/requirements.yml \
+ && chmod -R ug+rwx ${HOME}/.ansible
+
 COPY watches.yaml ${HOME}/watches.yaml
 
 COPY [[.RolesDir]]/ ${HOME}/[[.RolesDir]]/

internal/scaffold/ansible/dockerfilehybrid.go

@@ -34,6 +34,9 @@ type DockerfileHybrid struct {
 
 	// Watches - if true, include a COPY statement for watches.yaml
 	Watches bool
+
+	// Requirements - if true, include a COPY and RUN to install Ansible requirements
+	Requirements bool
 }
 
 // GetInput - gets the input
@@ -73,13 +76,12 @@ RUN yum clean all && rm -rf /var/cache/yum/* \
  && pip3 install --no-cache-dir --ignore-installed ipaddress \
       ansible-runner==1.3.4 \
       ansible-runner-http==1.0.0 \
-      openshift==0.9.2 \
+      openshift~=0.10.0 \
       ansible~=2.9 \
       jmespath \
  && yum remove -y gcc libffi-devel openssl-devel python36-devel \
  && yum clean all \
- && rm -rf /var/cache/yum \
- && ansible-galaxy collection install operator_sdk.util community.kubernetes
+ && rm -rf /var/cache/yum
 
 COPY build/_output/bin/[[.ProjectName]] ${OPERATOR}
 COPY bin /usr/local/bin
@@ -95,6 +97,10 @@ RUN TINIARCH=$(case $(arch) in x86_64) echo -n amd64 ;; ppc64le) echo -n ppc64el
   && curl -L -o /tini https://github.com/krallin/tini/releases/latest/download/tini-$TINIARCH \
   && chmod +x /tini
 
+[[- if .Requirements ]]
+COPY requirements.yml ${HOME}/requirements.yml
+RUN ansible-galaxy collection install -r ${HOME}/requirements.yml \
+ && chmod -R ug+rwx ${HOME}/.ansible[[ end ]]
 [[- if .Watches ]]
 COPY watches.yaml ${HOME}/watches.yaml[[ end ]]
 [[- if .Roles ]]

internal/scaffold/ansible/requirements.go

@@ -0,0 +1,40 @@
+// Copyright 2020 The Operator-SDK Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package ansible
+
+import (
+	"github.com/operator-framework/operator-sdk/internal/scaffold/input"
+)
+
+// RequirementsYml - A requirements file for Ansible collection dependencies
+type RequirementsYml struct {
+	StaticInput
+}
+
+// GetInput - gets the input
+func (r *RequirementsYml) GetInput() (input.Input, error) {
+	if r.Path == "" {
+		r.Path = "requirements.yml"
+	}
+	r.TemplateBody = requirementsYmlTmpl
+	return r.Input, nil
+}
+
+const requirementsYmlTmpl = `---
+collections:
+  - name: community.kubernetes
+    version: "<1.0.0"
+  - operator_sdk.util
+`