Make Cluster Roles Optional

This commit makes Cluster Roles optional within this PostgreSQL
Operator.  This is specifically accomplished by introducing the
concept of namespace operating modes, which represent various modes of
operation of the Postgres Operator that dictate which namespace
capabilities are enabled (if any at all).  The specific namespace mode
assigned to an Operator is determined based on the specific Cluster
Roles assigned to the postgres-operator Service Account.  Therefore,
users can choose to install the Cluster Roles required for a desired
operating mode, and the Operator will automatically detect those roles
and enable or disable the proper namespace capabilities.  This includes
allowing users to avoid the installation of any Cluster Roles
altogether, fully disabling namespace capabilities.

When the PostgreSQL Operator and all of its various components (i.e.
the apisever, scheduler, etc.) are run, the Kubernetes environment is
inspected to determine what cluster roles are currently assigned to the
pgo-operator Service Account (i.e. the Service Account running the Pod
the PostgreSQL Operator is running within).  Based on the Cluster Roles
identified, one of the following namespace operating modes will be
enabled for the installation:

- "dynamic": Enables full dynamic namespace capabilities, in which the
Operator can create, delete and update any namespaces within the
Kubernetes cluster, while then also having the ability to create the
roles, role bindings and service accounts within those namespaces as
required for the Operator to create PG clusters.  Additionally, while
in this mode the Operator can listen for namespace events (e.g.
namespace additions, updates and deletions), and then create or remove
controllers for various namespaces as those namespaces are added or
removed from the Kubernetes cluster and/or Operator install.

- "readOnly": In this mode the Operator is still able to listen for
namespace events within the  Kubernetetes cluster, and then create and
run and/or remove controllers as namespaces are added, updated and
deleted.  However, while in this mode the Operator is unable to create,
delete or update namespaces itself, nor can it create the RBAC it
requires in any of those namespaces to create PG clusters.  Therefore,
while in a "readonly" mode namespaces must be pre-configured with the
proper RBAC, since the Operator cannot create the RBAC itself.

- "disabled": Disables namespace capabilities within the Operator
altogether.  While in this mode the Operator will simply attempt to
work with the target namespaces specified during installation.  If no
target namespaces are specified, then the Operator will be configured
to work within the namespace in which it is deployed.  As with
"readonly", while in this mode namespaces must be pre-configured
with the proper RBAC, since the Operator cannot create the RBAC itself.

In support of this change, both the Ansible and Bash installers have
been updated to provide the user with a way of easily installing the
RBAC corresponding to each namespace operating mode.  The specific
variables used to install the proper Cluster Roles and therefore enable
the desired namespace operating mode are as follows:

- Bash: PGO_NAMESPACE_MODE
- Ansible: namespace_mode

When using the above variables with either installer, the valid options
available are:

- "dynamic"
- "readonly"
- "disabled"

When any of the values are selected during installation, the proper
Cluster roles will be installed to enable the corresponding namespace
operating mode as described above. The default value for either
installer is "dynamic", which maintains the current default behavior
when installing the Operator.

And finally this commit also updates various controller logic.  All
namespace informers now only listen for namespaces with the proper
"crunchydata" vendor label, as well as those labeled for the current
Operator install.  Additionally, all namespace controllers how have a
resync interval of 10 minutes.  Also, all controller manager functions
are now thread safe, while the Scheduler and Operator now share the
create the same type of namespace controller.  Otherwise various other
controller cleanup and/or reogranization is included.

Author: andrewlecuyer

apiserver/common.go

@@ -48,6 +48,11 @@ var (
 	// ErrStandbyNotAllowed contains the error message returned when an API call is not
 	// permitted because it involves a cluster that is in standby mode
 	ErrStandbyNotAllowed = errors.New("Action not permitted because standby mode is enabled")
+
+	// ErrMethodNotAllowed represents the error that is thrown when a feature is disabled within the
+	// current Operator install
+	ErrMethodNotAllowed = errors.New("This method has is not allowed in the current PostgreSQL " +
+		"Operator installation")
 )
 
 // ReplicaPodStatus stores the name of the node a replica pod is assigned to, as well

apiserver/namespaceservice/namespaceimpl.go

@@ -90,8 +90,8 @@ func CreateNamespace(clientset *kubernetes.Clientset, createdBy string, request
 	//iterate thru all the args (namespace names)
 	for _, namespace := range request.Args {
 
-		err := ns.CreateNamespace(clientset, apiserver.InstallationName, apiserver.PgoNamespace, createdBy, namespace)
-		if err != nil {
+		if err := ns.CreateNamespaceAndRBAC(clientset, apiserver.InstallationName,
+			apiserver.PgoNamespace, createdBy, namespace); err != nil {
 			resp.Status.Code = msgs.Error
 			resp.Status.Msg = err.Error()
 			return resp
@@ -140,8 +140,8 @@ func UpdateNamespace(clientset *kubernetes.Clientset, updatedBy string, request
 	//iterate thru all the args (namespace names)
 	for _, namespace := range request.Args {
 
-		err := ns.UpdateNamespace(clientset, apiserver.InstallationName, apiserver.PgoNamespace, updatedBy, namespace)
-		if err != nil {
+		if err := ns.UpdateNamespaceAndRBAC(clientset, apiserver.InstallationName,
+			apiserver.PgoNamespace, updatedBy, namespace); err != nil {
 			resp.Status.Code = msgs.Error
 			resp.Status.Msg = err.Error()
 			return resp

apiserver/namespaceservice/namespaceservice.go

@@ -17,10 +17,13 @@ limitations under the License.
 
 import (
 	"encoding/json"
+	"fmt"
+	"net/http"
+
 	"github.com/crunchydata/postgres-operator/apiserver"
 	msgs "github.com/crunchydata/postgres-operator/apiservermsgs"
+	"github.com/crunchydata/postgres-operator/ns"
 	log "github.com/sirupsen/logrus"
-	"net/http"
 )
 
 // ShowNamespaceHandler ...
@@ -54,6 +57,14 @@ func ShowNamespaceHandler(w http.ResponseWriter, r *http.Request) {
 
 	log.Debugf("ShowNamespaceHandler called [%v]", request)
 
+	// return 405 Method Not Allowed if all namespace functionality is disabled
+	if apiserver.NamespaceOperatingMode() == ns.NamespaceOperatingModeDisabled {
+		w.Header().Set("Allow", "")
+		http.Error(w, fmt.Errorf("Unable to show namespaces: %w",
+			apiserver.ErrMethodNotAllowed).Error(), http.StatusMethodNotAllowed)
+		return
+	}
+
 	username, err := apiserver.Authn(apiserver.SHOW_NAMESPACE_PERM, w, r)
 	if err != nil {
 		return
@@ -100,6 +111,14 @@ func CreateNamespaceHandler(w http.ResponseWriter, r *http.Request) {
 	var request msgs.CreateNamespaceRequest
 	_ = json.NewDecoder(r.Body).Decode(&request)
 
+	// return 405 Method Not Allowed if dynamic namespace functionality is not enabled
+	if apiserver.NamespaceOperatingMode() != ns.NamespaceOperatingModeDynamic {
+		w.Header().Set("Allow", "")
+		http.Error(w, fmt.Errorf("Unable to create namespaces: %w",
+			apiserver.ErrMethodNotAllowed).Error(), http.StatusMethodNotAllowed)
+		return
+	}
+
 	username, err := apiserver.Authn(apiserver.CREATE_NAMESPACE_PERM, w, r)
 	if err != nil {
 		return
@@ -144,6 +163,14 @@ func DeleteNamespaceHandler(w http.ResponseWriter, r *http.Request) {
 
 	log.Debugf("DeleteNamespaceHandler parameters [%v]", request)
 
+	// return 405 Method Not Allowed if dynamic namespace functionality is not enabled
+	if apiserver.NamespaceOperatingMode() != ns.NamespaceOperatingModeDynamic {
+		w.Header().Set("Allow", "")
+		http.Error(w, fmt.Errorf("Unable to delete namespaces: %w",
+			apiserver.ErrMethodNotAllowed).Error(), http.StatusMethodNotAllowed)
+		return
+	}
+
 	username, err := apiserver.Authn(apiserver.DELETE_NAMESPACE_PERM, w, r)
 	if err != nil {
 		return
@@ -194,6 +221,14 @@ func UpdateNamespaceHandler(w http.ResponseWriter, r *http.Request) {
 	var request msgs.UpdateNamespaceRequest
 	_ = json.NewDecoder(r.Body).Decode(&request)
 
+	// return 405 Method Not Allowed if dynamic namespace functionality is not enabled
+	if apiserver.NamespaceOperatingMode() != ns.NamespaceOperatingModeDynamic {
+		w.Header().Set("Allow", "")
+		http.Error(w, fmt.Errorf("Unable to update namespaces: %w",
+			apiserver.ErrMethodNotAllowed).Error(), http.StatusMethodNotAllowed)
+		return
+	}
+
 	username, err := apiserver.Authn(apiserver.UPDATE_NAMESPACE_PERM, w, r)
 	if err != nil {
 		return

apiserver/root.go

@@ -66,6 +66,9 @@ var BasicAuth bool
 var PgoNamespace string
 var InstallationName string
 
+// namespaceList is the list of namespaces identified at install time
+var namespaceList []string
+
 var CRUNCHY_DEBUG bool
 
 // TreeTrunk is for debugging only in this context
@@ -83,6 +86,11 @@ type CredentialDetail struct {
 
 var Pgo config.PgoConfig
 
+// NamespaceOperatingMode defines the namespace operating mode for the cluster,
+// e.g. "dynamic", "readonly" or "disabled".  See type NamespaceOperatingMode
+// for detailed explanations of each mode available.
+var namespaceOperatingMode ns.NamespaceOperatingMode
+
 func Initialize() {
 
 	PgoNamespace = os.Getenv("PGO_OPERATOR_NAMESPACE")
@@ -92,9 +100,6 @@ func Initialize() {
 	}
 	log.Info("Pgo Namespace is [" + PgoNamespace + "]")
 
-	//namespaceList := util.GetNamespaces()
-	//log.Debugf("watching the following namespaces: [%v]", namespaceList)
-
 	InstallationName = os.Getenv("PGO_INSTALLATION_NAME")
 	if InstallationName == "" {
 		log.Error("PGO_INSTALLATION_NAME environment variable is missng")
@@ -127,9 +132,19 @@ func Initialize() {
 
 	initConfig()
 
-	validateWithKube()
+	if err := setNamespaceOperatingMode(); err != nil {
+		log.Error(err)
+		os.Exit(2)
+	}
+
+	namespaceList, err = ns.GetNamespaceList(Clientset, NamespaceOperatingMode(),
+		InstallationName, PgoNamespace)
+	if err != nil {
+		log.Error(err)
+		os.Exit(2)
+	}
 
-	//validateUserCredentials()
+	log.Infof("Namespace operating mode is '%s'", NamespaceOperatingMode())
 }
 
 // ConnectToKube ...
@@ -279,12 +294,7 @@ func GetNamespace(clientset *kubernetes.Clientset, username, requestedNS string)
 		return requestedNS, errors.New(errMsg)
 	}
 
-	if ns.WatchingNamespace(clientset, requestedNS, InstallationName) {
-		return requestedNS, nil
-	}
-
-	log.Debugf("GetNamespace did not find the requested namespace %s", requestedNS)
-	return requestedNS, errors.New("requested Namespace was not found to be in the list of Namespaces being watched.")
+	return requestedNS, nil
 }
 
 // Authn performs HTTP Basic Authentication against a user if "BasicAuth" is set
@@ -380,37 +390,43 @@ func ValidateNodeLabel(nodeLabel string) error {
 	return nil
 }
 
-func validateWithKube() {
-	log.Debug("validateWithKube called")
-
-	err := ns.ValidateNamespaces(Clientset, InstallationName, PgoNamespace)
-	if err != nil {
-		log.Error(err)
-		os.Exit(2)
-	}
-
-}
-
-//returns installation access and user access
-//installation access means a namespace belongs to this Operator installation
-//user access means this user has access to a namespace
+// UserIsPermittedInNamespace returns installation access and user access.
+// Installation access means a namespace belongs to this Operator installation.
+// User access means this user has access to a namespace.
 func UserIsPermittedInNamespace(username, requestedNS string) (bool, bool) {
 
 	iAccess := false
 	uAccess := false
 
-	ns, found, err := kubeapi.GetNamespace(Clientset, requestedNS)
-	if !found {
-		log.Error(err)
-		log.Errorf("could not find namespace %s ", requestedNS)
-		return iAccess, uAccess
-	}
-
-	if ns.ObjectMeta.Labels[config.LABEL_VENDOR] == config.LABEL_CRUNCHY &&
+	// If the namespace operating mode isn't "disabled", then query the live Kube cluster
+	// for the namespace and then check its labels to determine if its part of the
+	// current Operator install.  Otherwise verify that it was a namespace specified
+	// during installation of the Operator.
+	if namespaceOperatingMode != ns.NamespaceOperatingModeDisabled {
+		ns, found, err := kubeapi.GetNamespace(Clientset, requestedNS)
+		if !found {
+			log.Error(err)
+			log.Errorf("could not find namespace %s ", requestedNS)
+			return iAccess, uAccess
+		}
 
-		ns.ObjectMeta.Labels[config.LABEL_PGO_INSTALLATION_NAME] == InstallationName {
-		iAccess = true
+		if ns.ObjectMeta.Labels[config.LABEL_VENDOR] == config.LABEL_CRUNCHY &&
+			ns.ObjectMeta.Labels[config.LABEL_PGO_INSTALLATION_NAME] == InstallationName {
+			iAccess = true
+		}
+	} else {
+		var exists bool
+		for _, namespace := range namespaceList {
+			if requestedNS == namespace {
+				exists = true
+				iAccess = true
+			}
+		}
 
+		if !exists {
+			log.Errorf("namespace %s is not included in this installation", requestedNS)
+			return iAccess, uAccess
+		}
 	}
 
 	//get the pgouser Secret for this username
@@ -528,3 +544,22 @@ func generateTLSCert(certPath, keyPath string) error {
 	return err
 
 }
+
+// setNamespaceOperatingMode set the namespace operating mode for the Operator by calling the
+// proper utility function to determine which mode is applicable based on the current
+// permissions assigned to the Operator Service Account.
+func setNamespaceOperatingMode() error {
+	nsOpMode, err := ns.GetNamespaceOperatingMode(Clientset)
+	if err != nil {
+		return err
+	}
+	namespaceOperatingMode = nsOpMode
+
+	return nil
+}
+
+// NamespaceOperatingMode returns the namespace operating mode for the current Operator
+// installation, which is stored in the "namespaceOperatingMode" variable
+func NamespaceOperatingMode() ns.NamespaceOperatingMode {
+	return namespaceOperatingMode
+}

conf/postgres-operator/pgo-target-role.json

@@ -50,18 +50,6 @@
                 "*"
             ]
         },
-        {
-            "apiGroups": [
-                ""
-            ],
-            "resources": [
-                "storageclasses"
-            ],
-            "verbs": [
-                "get",
-                "list"
-            ]
-        },
         {
             "apiGroups": [
                 "batch"

config/labels.go

@@ -169,7 +169,6 @@ const LABEL_CRUNCHY = "crunchydata"
 const LABEL_PGO_CREATED_BY = "pgo-created-by"
 const LABEL_PGO_UPDATED_BY = "pgo-updated-by"
 
-const LABEL_PGO_DEFAULT_SC = "pgo-default-sc"
 const LABEL_FAILOVER_STARTED = "failover-started"
 
 const GLOBAL_CUSTOM_CONFIGMAP = "pgo-custom-pg-config"

config/pgoconfig.go

@@ -542,14 +542,6 @@ func (c *PgoConfig) GetConfig(clientset *kubernetes.Clientset, namespace string)
 		return err
 	}
 
-	//determine the default storage class if necessary
-	if cMap == nil {
-		err = c.SetDefaultStorageClass(clientset)
-		if err != nil {
-			return err
-		}
-	}
-
 	c.CheckEnv()
 
 	//load up all the templates
@@ -789,49 +781,6 @@ func (c *PgoConfig) DefaultTemplate(path string) (string, error) {
 	return value, nil
 }
 
-func (c *PgoConfig) SetDefaultStorageClass(clientset *kubernetes.Clientset) error {
-
-	selector := LABEL_PGO_DEFAULT_SC + "=true"
-	scList, err := kubeapi.GetStorageClasses(clientset, selector)
-	if err != nil {
-		return err
-	}
-
-	if len(scList.Items) == 0 {
-		//no pgo default sc was found, so we will use 1st sc we find
-		scList, err = kubeapi.GetAllStorageClasses(clientset)
-		if err != nil {
-			return err
-		}
-		if len(scList.Items) == 0 {
-			return errors.New("no storage classes were found on this Kube system")
-		}
-		//configure with the 1st SC on the system
-	} else {
-		//configure with the default pgo sc
-	}
-
-	log.Infof("setting pgo-default-sc to %s", scList.Items[0].Name)
-
-	//add the storage class into the config
-	c.Storage[LABEL_PGO_DEFAULT_SC] = StorageStruct{
-		AccessMode:         "ReadWriteOnce",
-		Size:               "1G",
-		StorageType:        "dynamic",
-		StorageClass:       scList.Items[0].Name,
-		SupplementalGroups: "",
-		MatchLabels:        "",
-	}
-
-	//set the default storage configs to this new one
-	c.PrimaryStorage = LABEL_PGO_DEFAULT_SC
-	c.BackupStorage = LABEL_PGO_DEFAULT_SC
-	c.ReplicaStorage = LABEL_PGO_DEFAULT_SC
-	c.BackrestStorage = LABEL_PGO_DEFAULT_SC
-
-	return nil
-}
-
 // CheckEnv is mostly used for the OLM deployment use case
 // when someone wants to deploy with OLM, use the baked-in
 // configuration, but use a different set of images, by

controller/controllerutil.go

@@ -16,28 +16,35 @@ limitations under the License.
 */
 
 import (
+	"errors"
+
 	crv1 "github.com/crunchydata/postgres-operator/apis/crunchydata.com/v1"
 	"github.com/crunchydata/postgres-operator/config"
 	"github.com/crunchydata/postgres-operator/kubeapi"
 	log "github.com/sirupsen/logrus"
 	"k8s.io/client-go/rest"
 )
 
+// ErrControllerGroupExists is the error that is thrown when a controller group for a specific
+// namespace already exists
+var ErrControllerGroupExists = errors.New("A controller group for the namespace specified already" +
+	"exists")
+
 // WorkerRunner is an interface for controllers the have worker queues that need to be run
 type WorkerRunner interface {
 	RunWorker()
 }
 
 // ManagerInterface defines the interface for a controller manager
 type ManagerInterface interface {
-	AddControllerGroup(namespace string) error
-	AddAndRunControllerGroup(namespace string)
+	AddGroup(namespace string) error
+	AddAndRunGroup(namespace string) error
+	RemoveAll()
+	RemoveGroup(namespace string)
 	RunAll()
 	RunGroup(namespace string)
 	StopAll()
 	StopGroup(namespace string)
-	RemoveAll()
-	RemoveGroup(namespace string)
 }
 
 // InitializeReplicaCreation initializes the creation of replicas for a cluster.  For a regular