Improve documentation (#1975)

* Update docs and reword some sections

Author: johscheuer

README.md

@@ -79,6 +79,8 @@ To get this controller running in a local Kubernetes cluster:
    you can set the `BUILD_PLATFORM` env variable `BUILD_PLATFORM="linux/amd64" make rebuild-operator`.
 1. Run `kubectl apply -k ./config/tests/base` to create a new FoundationDB cluster with the operator.
 
+_NOTE_: FoundationDB currently only publishes container images for running on `amd64`/`x64` nodes.
+
 ### Running Locally with nerdctl
 
 Instead of Docker you can also use [nerdctl](https://github.com/containerd/nerdctl) to build and push your images.

docs/compatibility.md

@@ -40,7 +40,6 @@ in advance of the upgrade, through whatever process you need to update your
 clusters safely.
 After you updated the operator you should ensure that all clusters are in a reconciled state and all changes are applied.
 
-
 At this point, you can use the `kubectl-fdb` plugin to check your cluster specs for deprecated fields or defaults.
 For more information see the [kubectl-fdb plugin Readme](../kubectl-fdb/Readme.md) and the `deprecation` subcommand.
 

docs/manual/customization.md

@@ -326,7 +326,7 @@ The operator will add a special locality to the fdbserver processes called `dns_
 
 ## Using Multiple Namespaces
 
-Our [sample deployment](https://raw.githubusercontent.com/foundationdb/fdb-kubernetes-operator/master/config/samples/deployment.yaml) configures the operator to run in single-namespace mode, where it only manages resources in the namespace where the operator itself is running. If you want a single deployment of the operator to manage your FDB clusters across all of your namespaces, you will need to run it in global mode. Which mode is appropriate will depend on the constraints of your environment.
+Our [sample deployment](../../config/samples/deployment.yaml) configures the operator to run in single-namespace mode, where it only manages resources in the namespace where the operator itself is running. If you want a single deployment of the operator to manage your FDB clusters across all of your namespaces, you will need to run it in global mode. Which mode is appropriate will depend on the constraints of your environment.
 
 ### Single-Namespace Mode
 
@@ -339,7 +339,7 @@ To run the controller in single-namespace mode, you will need to configure the f
 * A service account for the controller
 * The serviceAccountName field in the controller's pod spec
 * A `WATCH_NAMESPACE` environment variable defined in the controller's pod spec or in the arguments of the container command
-* A Role that grants access to the necessary permissions to all of the resources that the controller manages. See the [sample role](https://raw.githubusercontent.com/FoundationDB/fdb-kubernetes-operator/master/config/samples/deployment/rbac_role.yaml) for the list of those permissions.
+* A Role that grants access to the necessary permissions to all of the resources that the controller manages. See the [sample role](../../config/samples/deployment/rbac_role.yaml) for the list of those permissions.
 * A RoleBinding that binds that role to the service account for the controller
 
 The sample deployment provides all of this configuration.
@@ -354,7 +354,7 @@ To run the controller in global mode, you will need to configure the following t
 
 * A service account for the controller
 * The serviceAccountName field in the controller's pod spec
-* A ClusterRole that grants access to the necessary permissions to all of the resources that the controller manages. See the [sample role](https://raw.githubusercontent.com/FoundationDB/fdb-kubernetes-operator/master/config/samples/deployment/rbac_role.yaml) for the list of those permissions.
+* A ClusterRole that grants access to the necessary permissions to all of the resources that the controller manages. See the [sample role](../../config/samples/deployment/rbac_role.yaml) for the list of those permissions.
 * A ClusterRoleBinding that binds that role to the service account for the controller
 
 You can build this kind of configuration easily from the sample deployment by changing the following things:

docs/manual/debugging.md

@@ -93,7 +93,8 @@ Remove [storage-1] from cluster default/sample-cluster with exclude: false and s
 
 **NOTE**: This is a very dangerous operation.
 This will delete the pod and the PVC without checking that the data has been re-replicated.
-You should only due this after checking that the database is available, has not had any data loss, and that the pod is currently not running. You can confirm the first and second check by looking at the cluster status.
+You should only due this after checking that the database is available, has not had any data loss, and that the pod is currently not running.
+You can confirm the first and second check by looking at the cluster status.
 
 ## Exclusions Not Starting Due to Missing Processes
 

docs/manual/fault_domains.md

@@ -134,8 +134,8 @@ This strategy uses the pod name as the fault domain, which allows each process t
 
 ## Three-Data-Hall Replication
 
-**NOTE**: The support for this redundancy mode is new and might have issues. Please make sure you test this configuration in our test/QA environment.
-The [three-data-hall](https://apple.github.io/foundationdb/configuration.html#single-datacenter-modes) replication can be use to replicate data across three data halls, or availability zones.
+**NOTE**: The support for this redundancy mode is new and might have issues. Please make sure you test this configuration in your test/QA environment.
+The [three-data-hall](https://apple.github.io/foundationdb/configuration.html#single-datacenter-modes) replication can be used to replicate data across three data halls, or availability zones.
 This requires that your fault domains are properly labeled on the Kubernetes nodes.
 Most cloud-providers will use the well-known label [topology.kubernetes.io/zone](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone) for this.
 When creating a three-data-hall replicated FoundationDBCluster on Kubernetes we have to create 3 `FoundationDBCluster` resources.
@@ -195,9 +195,9 @@ Operations across the different `FoundationDBCluster` resources are [coordinated
 
 ## Multi-Region Replication
 
-The replication strategies above all describe how data is replicated within a data center.
+The replication strategies above all describe how data is replicated within a data center or a single region.
 They control the `zoneid` field in the cluster's locality.
-If you want to run a cluster across multiple data centers, you can use FoundationDB's multi-region replication.
+If you want to run a cluster across multiple data centers or regions, you can use FoundationDB's multi-region replication.
 This can work with any of the replication strategies above.
 The data center will be a separate fault domain from whatever you provide for the zone.
 
@@ -286,15 +286,29 @@ spec:
 
 ## Coordinating Global Operations
 
-When running a FoundationDB cluster that is deployed across multiple Kubernetes clusters, each Kubernetes cluster will have its own instance of the operator working on the processes in its cluster. There will be some operations that cannot be scoped to a single Kubernetes cluster, such as changing the database configuration.
-The operator provides a locking system to ensure that only one instance of the operator can perform these operations at a time. You can enable this locking system by setting `lockOptions.disableLocks = false` in the cluster spec. The locking system is automatically enabled by default for any cluster that has multiple regions in its database configuration, a `zoneCount` greater than 1 in its fault domain configuration, or `redundancyMode` equal to `three_data_hall`.
+When running a FoundationDB cluster that is deployed across multiple Kubernetes clusters, each Kubernetes cluster will have its own instance of the operator working on the processes in its cluster.
+There will be some operations that cannot be scoped to a single Kubernetes cluster, such as changing the database configuration.
+The operator provides a locking system to reduce the risk of those independent operator instance performing the same action at the same time.
+All actions that the operator performs like changing the configuration or restarting processes will lead to the same desired state.
+The locking system is only intended to reduce the risk of frequent reoccurring recoveries.
+
+You can enable this locking system by setting `lockOptions.disableLocks = false` in the cluster spec.
+The locking system is automatically enabled by default for any cluster that has multiple regions in its database configuration, a `zoneCount` greater than 1 in its fault domain configuration, or `redundancyMode` equal to `three_data_hall`.
 
 The locking system uses the `processGroupIDPrefix` from the cluster spec to identify an process group of the operator.
 Make sure to set this to a unique value for each Kubernetes cluster, both to support the locking system and to prevent duplicate process group IDs.
 
-This locking system uses the FoundationDB cluster as its data source. This means that if the cluster is unavailable, no instance of the operator will be able to get a lock. If you hit a case where this becomes an issue, you can disable the locking system by setting `lockOptions.disableLocks = true` in the cluster spec.
+This locking system uses the FoundationDB cluster as its data source.
+This means that if the cluster is unavailable, no instance of the operator will be able to get a lock.
+If you hit a case where this becomes an issue, you can disable the locking system by setting `lockOptions.disableLocks = true` in the cluster spec.
 
-In most cases, restarts will be done independently in each Kubernetes cluster, and the locking system will be used to ensure a minimum time between the different restarts and avoid multiple recoveries in a short span of time. During upgrades, however, all instances must be restarted at the same time. The operator will use the locking system to coordinate this. Each instance of the operator will store records indicating what processes it is managing and what version they will be running after the restart. Each instance will then try to acquire a lock and confirm that every process reporting to the cluster is ready for the upgrade. If all processes are prepared, the operator will restart all of them at once. If any instance of the operator is stuck and unable to prepare its processes for the upgrade, the restart will not occur.
+In most cases, restarts will be done independently in each Kubernetes cluster, and the locking system will be used to try to ensure a minimum time between the different restarts and avoid multiple recoveries in a short span of time.
+During upgrades, however, all instances must be restarted at the same time.
+The operator will use the locking system to coordinate this.
+Each instance of the operator will store records indicating what processes it is managing and what version they will be running after the restart.
+Each instance will then try to acquire a lock and confirm that every process reporting to the cluster is ready for the upgrade.
+If all processes are prepared, the operator will restart all of them at once.
+If any instance of the operator is stuck and unable to prepare its processes for the upgrade, the restart will not occur.g
 
 ### Deny List
 
@@ -351,16 +365,17 @@ Depending on the requirements the operator can be configured to either prefer or
 The number of coordinators is currently a hardcoded mechanism based on the [following algorithm](https://github.com/FoundationDB/fdb-kubernetes-operator/blob/v0.49.2/api/v1beta1/foundationdbcluster_types.go#L1500-L1508):
 
 ```go
+// DesiredCoordinatorCount returns the number of coordinators to recruit for a cluster.
 func (cluster *FoundationDBCluster) DesiredCoordinatorCount() int {
-	if cluster.Spec.DatabaseConfiguration.UsableRegions > 1 {
-		return 9
-	}
+    if cluster.Spec.DatabaseConfiguration.UsableRegions > 1 || cluster.Spec.DatabaseConfiguration.RedundancyMode == RedundancyModeThreeDataHall {
+        return 9
+    }
 
-	return cluster.MinimumFaultDomains() + cluster.DesiredFaultTolerance()
+    return cluster.MinimumFaultDomains() + cluster.DesiredFaultTolerance()
 }
 ```
 
-For all clusters that use more than one region the operator will recruit 9 coordinators.
+For all clusters that use more than one region or uses `three_data_hall`, the operator will recruit 9 coordinators.
 If the number of regions is `1` the number of recruited coordinators depends on the redundancy mode.
 The number of coordinators is chosen based on the fact that the coordinators use a consensus protocol (Paxos) that needs a majority of processes to be up.
 A common pattern in majority based system is to run `n * 2 + 1` processes, where `n` defines the failures that should be tolerated.
@@ -412,7 +427,6 @@ The operator supports the following classes as coordinators:
 
 FoundationDB clusters that are spread across different DC's or Kubernetes clusters only support the same `coordinatorSelection`.
 The reason behind this is that the coordinator selection is a global process and different `coordinatorSelection` of the `FoundationDBCluster` resources can lead to an undefined behaviour or in the worst case flapping coordinators.
-There are plans to support this feature in the future.
 
 ## Next
 

docs/manual/getting_started.md

@@ -19,7 +19,7 @@ You can see logs from the operator by running `kubectl logs -f -l app=fdb-kubern
 
 The example below will cover creating a cluster. All subsequent examples will assume that you have just created this cluster, and will cover an operation on this cluster.
 
-For more information on the fields you can define on the cluster resource, see the [go docs](https://godoc.org/github.com/FoundationDB/fdb-kubernetes-operator/pkg/apis/apps/v1beta2#FoundationDBCluster).
+For more information on the fields you can define on the cluster resource, see the [cluster spec docs](../cluster_spec.md).
 
 For more information on version compatibility, see our [compatibility guide](/docs/compatibility.md).
 
@@ -36,16 +36,29 @@ spec:
   version: 7.1.26
 ```
 
-This will create a cluster with 3 storage processes, 4 log processes, and 7 stateless processes. Each fdbserver process will be in a separate pod, and the pods will have names of the form `sample-cluster-$role-$n`, where `$n` is the process group ID and `$role` is the role for the process.
+This will create a cluster with 3 storage processes, 4 log processes, and 7 stateless processes.
+Each `fdbserver` process will be in a separate pod, and the pods will have names of the form `sample-cluster-$role-$n`, where `$n` is the process group ID and `$role` is the role for the process.
 
-You can run `kubectl get foundationdbcluster sample-cluster` to check the progress of reconciliation. Once the reconciled generation appears in this output, the cluster should be up and ready. After creating the cluster, you can connect to the cluster by running `kubectl exec -it sample-cluster-log-1 -- fdbcli`.
+You can run `kubectl get foundationdbcluster sample-cluster` to check the progress of reconciliation.
+Once the reconciled generation appears in this output, the cluster should be up and ready.
+After creating the cluster, you can connect to the cluster by running `kubectl exec -it sample-cluster-log-1 -- fdbcli`.
 
-This example requires non-trivial resources, based on what a process will need in a production environment. This means that is too large to run in a local testing environment. It also requires disk I/O features that are not present in Docker for Mac. If you want to run these tests in that kind of environment, you can try bringing in the resource requirements, knobs, and fault domain information from a [local testing example](../../config/samples/cluster.yaml).
+This example requires non-trivial resources, based on what a process will need in a production environment.
+This means that it might be too large to run in a local testing environment.
+It also requires disk I/O features that are not present in Docker for Mac.
+If you want to run these tests in that kind of environment, you can try bringing in the resource requirements, knobs, and fault domain information from a [local testing example](../../config/samples/cluster.yaml).
+
+_NOTE_: FoundationDB currently only supports `amd64`/`x64`.
 
 In addition to the pods, the operator will create a Persistent Volume Claim for any stateful
 processes in the cluster. In this example, each volume will be 128 GB.
 
-By default each pod will have two containers and one init container. The `foundationdb` container will run fdbmonitor and fdbserver, and is the main container for the pod. The `foundationdb-kubernetes-sidecar` container will run a sidecar image designed to help run FDB on Kubernetes. It is responsible for managing the fdbmonitor conf files and providing FDB binaries to the `foundationdb` container. The operator will create a config map that contains a template for the monitor conf file, and the sidecar will interpolate instance-specific fields into the conf and make it available to the fdbmonitor process through a shared volume. The "Upgrading a Cluster" has more detail on we manage binaries. The init container will run the same sidecar image, and will ensure that the initial binaries and dynamic conf are ready before the fdbmonitor process starts.
+By default each pod will have two containers and one init container.
+The `foundationdb` container will run `fdbmonitor` and `fdbserver`, and is the main container for the pod. The `foundationdb-kubernetes-sidecar` container will run a sidecar image designed to help run FDB on Kubernetes.
+It is responsible for managing the `fdbmonitor` conf files and providing FDB binaries to the `foundationdb` container. 
+The operator will create a config map that contains a template for the monitor conf file, and the sidecar will interpolate instance-specific fields into the conf and make it available to the fdbmonitor process through a shared volume.
+The "Upgrading a Cluster" has more detail on we manage binaries.
+The init container will run the same sidecar image, and will ensure that the initial binaries and dynamic conf are ready before the `fdbmonitor` process starts.
 
 ## Accessing a Cluster
 
@@ -63,6 +76,22 @@ spec:
       template:
         spec:
           restartPolicy: OnFailure
+          initContainers:
+          - name: init-cluster-file
+            image: foundationdb/foundationdb-kubernetes-sidecar:7.1.26-1
+            args:
+              - --init-mode
+              - --input-dir
+              - /mnt/config-volume
+              - --copy-file
+              - cluster-file
+              - --require-not-empty
+              - cluster-file
+          volumeMounts:
+            - name: config-volume
+              mountPath: /mnt/config-volume
+            - name: shared-volume
+              mountPath: /out-dir
           containers:
           - name: fdbcli-status-cronjob
             image: foundationdb/foundationdb:7.1.26
@@ -74,18 +103,21 @@ spec:
             - name: FDB_CLUSTER_FILE
               value: /mnt/config-volume/cluster-file
             volumeMounts:
-            - name: config-volume
+            - name: shared-volume
               mountPath: /mnt/config-volume
           volumes:
           - name: config-volume
             configMap:
               name: sample-cluster-config
+          - name: shared-volume
+            emptyDir:
+              medium: Memory
 ```
 
 Note that:
 
 * The name of the config map will depend on the name of your cluster.
-* For long-running applications you should ensure that your cluster file is writeable by your application.
+* For long-running applications you should ensure that your cluster file is writeable by your application. You can achieve this by using the init container and copying the cluster-file inside a shared `emptyDir`.
 
 ## Next