Add a type to the v1 model for the updated ClusterConfig (#489)

* Add a type to the v1 model for the updated ClusterConfig

ClusterConfiguration values can either be concrete ones, or
references - either to k8s resources (such as ConfigMap or
Secret) or an external secret.

Concrete items are represented as strings - the attribute in
a CR should have the yaml representation of the raw value.
This permits us to inject it verbatim into a reconstituted
`.bootstrap.yaml` file, and (with the aid of a schema from a
running Redpanda cluster) turn the representation into
concrete values without loss of fidelity.

We use json encoding to construct these, and yaml decoding
(which is a little more flexible in what it'll accept) to
decode them.

* Add the construction of a bootstrap template file

This is ugly at the moment - the intention here is to
add the bootstrap templating without too much change.

We'll follow up with commits to remove swathes of the v1
operator behaviour - such as removing drift detection -
after this has landed.

There's additional behaviour in the `configure` subcommand,
to template-expand environmant variables and/or external
secrets.

Internally, the operator uses the bootstrap template
file to detemine whether reconfiguration should be applied.
This is a short-term adjustment - it'll behave correctly on
concrete values, but won't necessarily pick up external sources
of values (such as config maps) changing. That's to come.

The operator takes some care to not unnecessarily trigger a
sts restart - if the sts has already been configured without
a tempalted bootstrap, then the old behaviour will be preserved
until such time as a restart would be required anyway.

* kuttl test for "ClusterConfiguration" override

Ensure the configuration is correctly stashed and updated.

kuttl test for "ClusterConfiguration" external ref override

Note: because this test causes an additional env var to be
added to an initContainer, it'll cause a restart of the
resulting statefulSet.

Author: jan-g

operator/api/vectorized/v1alpha1/cluster_types.go

@@ -170,6 +170,12 @@ type ClusterSpec struct {
 	// By default if Replicas is 3 or more and redpanda.default_topic_partitions is not set
 	// default webhook is setting redpanda.default_topic_partitions to 3.
 	AdditionalConfiguration map[string]string `json:"additionalConfiguration,omitempty"`
+	// Cluster configuration values may also be held here. A `keyName` entry in this
+	// attribute will override corresponding `redpanda.keyName` entries in AdditionalConfiguration;
+	// this is to permit the migration of those settings to this attribute.
+	// The configuration may contain references to values extracted from k8s ConfigMaps or Secrets;
+	// furthermore, we support the fetching of provider-specific secrets directly.
+	ClusterConfiguration ClusterConfiguration `json:"clusterConfiguration,omitempty"`
 	// DNSTrailingDotDisabled gives ability to turn off the fully-qualified
 	// DNS name.
 	// http://www.dns-sd.org/trailingdotsindomainnames.html
@@ -196,6 +202,53 @@ type ClusterSpec struct {
 	NodePools []NodePoolSpec `json:"nodePools,omitempty"`
 }
 
+// ClusterConfiguration holds values (or references to values) that should be used
+// to configure the cluster. Where the cluster schema defines a non-string type for a
+// given key, the corresponding values here should be string-encoded (according to yaml
+// rules)
+type ClusterConfiguration map[string]ClusterConfigValue
+
+// YAMLRepresentation holds a serialised form of a concrete value. We need this for
+// a couple of reasons: firstly, "stringifying" numbers avoids loss of accuracy and
+// rendering issues where intermediate values are represented as f64 values by
+// external tooling. Secondly, the initial configuration of a bootstrap file has
+// no running cluster - and therefore no online schema - available. Instead we use
+// representations that can be inserted verbatim into a YAML document.
+// Ideally, these will be JSON-encoded into a single line representation. They are
+// decoded using YAML deserialisation (which has a little more flexibility around
+// the representation of unambiguous string values).
+type YAMLRepresentation string
+
+// ClusterConfigValue represents a value of arbitrary type T. Values are string-encoded according to
+// YAML rules in order to preserve numerical fidelity.
+// Because these values must be embedded in a `.bootstrap.yaml` file - during the processing of
+// which, the AdminAPI's schema is unavailable - we endeavour to use yaml-compatible representations
+// throughout. The octet sequence of a Representation will be inserted into a bootstrap template
+// verbatim.
+// TODO: this type should be lifted to a shared module rather than duplicated in the `redpanda` CRD definition.
+// TODO: add CEL validation here.
+type ClusterConfigValue struct {
+	// If the value is directly known, its yaml representation can be embedded here.
+	// Use the string representation of a yaml-serialised value in order to preserve accuracy.
+	// Example:
+	// The string "foo" should be the five octets "\"foo\""
+	// A true value should be the four octets "true".
+	// The number -123456 should be a seven-octet sequence, "-123456".
+	Repr *YAMLRepresentation `json:"repr,omitempty"`
+	// If the value is supplied by an k8s object reference, coordinates are embedded here.
+	// For target values, the string value fetched from the source will be treated as
+	// a value encoded according to YAML rules; the string can then be embedded verbatim into
+	// the bootstrap file.
+	ConfigMapKeyRef *corev1.ConfigMapKeySelector `json:"configMapKeyRef,omitempty"`
+	// Should the value be contained in a k8s secret rather than configmap, we can refer
+	// to it here.
+	SecretKeyRef *corev1.SecretKeySelector `json:"secretKeyRef,omitempty"`
+	// If the value is supplied by an external source, coordinates are embedded here.
+	// Note: we interpret all fetched external secrets as string values and yam-encode them prior to embedding.
+	// TODO: This decision needs finalising and documenting.
+	ExternalSecretRef *string `json:"externalSecretRef,omitempty"`
+}
+
 // NodePoolSpec defines a NodePool. NodePools have their own:
 // NodeSelector, so they can be scheduled on specific cloud provider Node Pools.
 // Storage, as different NodePools may have different disk shapes.

operator/api/vectorized/v1alpha1/zz_generated.deepcopy.go

@@ -0,0 +1,39 @@
+package configurator
+
+import (
+	"encoding/json"
+	"fmt"
+	"maps"
+	"os"
+	"slices"
+	"strings"
+
+	"github.com/redpanda-data/redpanda-operator/operator/pkg/clusterconfiguration"
+)
+
+// Template out the bootstrap file
+// This takes an input template, resolves any remaining external references, then writes out the resulting bootstrap file
+func templateBootstrapYaml(inFile, outFile string) error {
+	var template map[string]clusterconfiguration.ClusterConfigTemplateValue
+	buf, err := os.ReadFile(inFile)
+	if err != nil {
+		return fmt.Errorf("cannot load bootstrap template file: %w", err)
+	}
+	if err := json.Unmarshal(buf, &template); err != nil {
+		return fmt.Errorf("cannot parse bootstrap template file: %w", err)
+	}
+
+	var config []string
+	keys := slices.Sorted(maps.Keys(template))
+	for _, k := range keys {
+		// Work out what the value should be and add it to the output.
+		repr, err := clusterconfiguration.ExpandValueForTemplate(template[k])
+		if err != nil {
+			return fmt.Errorf("cannot resolve value %s: %w", k, err)
+		}
+		config = append(config, fmt.Sprintf("%s: %s", k, repr))
+	}
+
+	output := strings.Join(config, "\n")
+	return os.WriteFile(outFile, []byte(output), 0o644)
+}

operator/cmd/configurator/bootstrap.go

@@ -59,6 +59,8 @@ const (
 	svcFQDNEnvVar                                        = "SERVICE_FQDN"
 	additionalListenersLegacyEnvVar                      = "ADDITIONAL_LISTENERS"
 	additionalListenersJSONEnvVar                        = "ADDITIONAL_LISTENERS_JSON"
+	bootstrapDestinationEnvVar                           = "BOOTSTRAP_DESTINATION"
+	bootstrapTemplateEnvVar                              = "BOOTSTRAP_TEMPLATE"
 )
 
 type brokerID int
@@ -83,6 +85,8 @@ type configuratorConfig struct {
 	additionalListenersLegacy                      string
 	additionalListenersJSON                        string
 	hostIndexOffset                                int
+	bootstrapDestination                           string
+	bootstrapTemplate                              string
 }
 
 func (c *configuratorConfig) String() string {
@@ -220,6 +224,14 @@ func run(cmd *cobra.Command, args []string) {
 		log.Fatalf("%s", fmt.Errorf("unable to write the destination configuration file: %w", err))
 	}
 
+	if c.bootstrapTemplate != "" && c.bootstrapDestination != "" {
+		// Perform the bootstrap templating
+		err := templateBootstrapYaml(c.bootstrapTemplate, c.bootstrapDestination)
+		if err != nil {
+			log.Fatalf("%s", fmt.Errorf("unable to template the .bootstrap.yaml file: %w", err))
+		}
+	}
+
 	log.Printf("Configuration saved to: %s", c.configDestination)
 }
 
@@ -582,6 +594,18 @@ func checkEnvVars() (configuratorConfig, error) {
 		log.Printf("additional listeners configured with the JSON format: %v", c.additionalListenersJSON)
 	}
 
+	// Handling the templating of the bootstrap file is optional
+	bootstrapTemplate, exist := os.LookupEnv(bootstrapTemplateEnvVar)
+	if exist && bootstrapTemplate != "" {
+		bootstrapDestination, exist := os.LookupEnv(bootstrapDestinationEnvVar)
+		if !exist || bootstrapDestination == "" {
+			result = errors.Join(result, fmt.Errorf("%s specified without corresponding %s", bootstrapTemplateEnvVar, bootstrapDestinationEnvVar))
+		} else {
+			c.bootstrapTemplate = bootstrapTemplate
+			c.bootstrapDestination = bootstrapDestination
+		}
+	}
+
 	return c, result
 }
 

operator/cmd/configurator/configurator.go

@@ -173,6 +173,91 @@ spec:
                 required:
                 - enabled
                 type: object
+              clusterConfiguration:
+                additionalProperties:
+                  description: |-
+                    ClusterConfigValue represents a value of arbitrary type T. Values are string-encoded according to
+                    YAML rules in order to preserve numerical fidelity.
+                    Because these values must be embedded in a `.bootstrap.yaml` file - during the processing of
+                    which, the AdminAPI's schema is unavailable - we endeavour to use yaml-compatible representations
+                    throughout. The octet sequence of a Representation will be inserted into a bootstrap template
+                    verbatim.
+                  properties:
+                    configMapKeyRef:
+                      description: |-
+                        If the value is supplied by an k8s object reference, coordinates are embedded here.
+                        For target values, the string value fetched from the source will be treated as
+                        a value encoded according to YAML rules; the string can then be embedded verbatim into
+                        the bootstrap file.
+                      properties:
+                        key:
+                          description: The key to select.
+                          type: string
+                        name:
+                          default: ""
+                          description: |-
+                            Name of the referent.
+                            This field is effectively required, but due to backwards compatibility is
+                            allowed to be empty. Instances of this type with an empty value here are
+                            almost certainly wrong.
+                            More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
+                          type: string
+                        optional:
+                          description: Specify whether the ConfigMap or its key must
+                            be defined
+                          type: boolean
+                      required:
+                      - key
+                      type: object
+                      x-kubernetes-map-type: atomic
+                    externalSecretRef:
+                      description: |-
+                        If the value is supplied by an external source, coordinates are embedded here.
+                        Note: we interpret all fetched external secrets as string values and yam-encode them prior to embedding.
+                      type: string
+                    repr:
+                      description: |-
+                        If the value is directly known, its yaml representation can be embedded here.
+                        Use the string representation of a yaml-serialised value in order to preserve accuracy.
+                        Example:
+                        The string "foo" should be the five octets "\"foo\""
+                        A true value should be the four octets "true".
+                        The number -123456 should be a seven-octet sequence, "-123456".
+                      type: string
+                    secretKeyRef:
+                      description: |-
+                        Should the value be contained in a k8s secret rather than configmap, we can refer
+                        to it here.
+                      properties:
+                        key:
+                          description: The key of the secret to select from.  Must
+                            be a valid secret key.
+                          type: string
+                        name:
+                          default: ""
+                          description: |-
+                            Name of the referent.
+                            This field is effectively required, but due to backwards compatibility is
+                            allowed to be empty. Instances of this type with an empty value here are
+                            almost certainly wrong.
+                            More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
+                          type: string
+                        optional:
+                          description: Specify whether the Secret or its key must
+                            be defined
+                          type: boolean
+                      required:
+                      - key
+                      type: object
+                      x-kubernetes-map-type: atomic
+                  type: object
+                description: |-
+                  Cluster configuration values may also be held here. A `keyName` entry in this
+                  attribute will override corresponding `redpanda.keyName` entries in AdditionalConfiguration;
+                  this is to permit the migration of those settings to this attribute.
+                  The configuration may contain references to values extracted from k8s ConfigMaps or Secrets;
+                  furthermore, we support the fetching of provider-specific secrets directly.
+                type: object
               configuration:
                 description: Configuration represent redpanda specific configuration
                 properties:

operator/config/crd/bases/redpanda.vectorized.io_clusters.yaml

@@ -422,6 +422,7 @@ func (a *attachedResources) statefulSet() error {
 			a.getServiceAccountName(),
 			a.reconciler.configuratorSettings,
 			cm.GetNodeConfigHash,
+			cm.CreateConfiguration,
 			a.reconciler.AdminAPIClientFactory,
 			a.reconciler.Dialer,
 			a.reconciler.DecommissionWaitInterval,