[OCPBUGS-3341] Add RHOSP CCM options reference documentation (#53068)

Author: maxwelldb

installing/installing_openstack/installing-openstack-cloud-config-reference.adoc

@@ -6,4 +6,5 @@ include::_attributes/common-attributes.adoc[]
 
 toc::[]
 
-include::modules/nw-openstack-external-ccm.adoc[leveloffset=+1]
+include::modules/nw-openstack-external-ccm.adoc[leveloffset=+1]
+include::modules/cluster-cloud-controller-config-osp.adoc[leveloffset=+1]

modules/cluster-cloud-controller-config-osp.adoc

@@ -0,0 +1,328 @@
+// Module included in the following assemblies:
+//
+// * installing/installing_openstack/installing-openstack-cloud-config-reference.adoc
+
+:_content-type: REFERENCE
+[id="cluster-cloud-controller-config_{context}"]
+= The OpenStack Cloud Controller Manager (CCM) config map
+
+An OpenStack CCM config map defines how your cluster interacts with your {rh-openstack} cloud. By default, this configuration is stored under the `cloud.conf` key in the `cloud-conf` config map in the `openshift-cloud-controller-manager` namespace. 
+
+[IMPORTANT]
+====
+The `cloud-conf` config map is generated from the `cloud-provider-config` config map in the `openshift-config` namespace.
+
+To change the settings that are described by the `cloud-conf` config map, modify the `cloud-provider-config` config map. 
+
+As part of this synchronization, the CCM Operator overrides some options. For more information, see "The {rh-openstack} Cloud Controller Manager".
+====
+
+For example:
+
+.An example `cloud-conf` config map
+[source,yaml]
+----
+apiVersion: v1
+data:
+  cloud.conf: |
+    [Global] <1>
+    secret-name = openstack-credentials
+    secret-namespace = kube-system
+    region = regionOne
+    [LoadBalancer]
+    use-octavia = True
+kind: ConfigMap
+metadata:
+  creationTimestamp: "2022-12-20T17:01:08Z"
+  name: cloud-conf
+  namespace: openshift-cloud-controller-manager
+  resourceVersion: "2519"
+  uid: cbbeedaf-41ed-41c2-9f37-4885732d3677
+----
+<1> Set global options by using a `clouds.yaml` file rather than modifying the config map.
+
+The following options are present in the config map. Except when indicated otherwise, they are mandatory for clusters that run on {rh-openstack}. 
+
+// [id="ccm-config-global-options"]
+// == Global options
+
+// The following options are used for {rh-openstack} CCM authentication with the {rh-openstack} Identity service, also known as Keystone. They are similiar to the global options that you can set by using the `openstack` CLI.
+
+// |===
+// | Option | Description
+
+// | `ca-file`
+// | Optional. The CA certificate bundle file for communication with the {rh-openstack} Identity service. If you use the HTTPS protocol with The Identity service URL, this option is required.
+
+// | `cert-file`
+// | Optional. The client certificate path to use for client TLS authentication.
+
+// | `key-file`
+// | Optional. The client private key path to use for client TLS authentication.
+
+// | `region`
+// | The Identity service region name.
+
+// | `trust-id`
+// | The Identity service trust ID. A trust represents the authorization of a user, or trustor, to delegate roles to another user, or trustee. Optionally, a trust authorizes the trustee to impersonate the trustor. You can find available trusts by querying the `/v3/OS-TRUST/trusts` endpoint of the Identity service API.
+
+// | `trustee-id`
+// | The Identity service trustee user ID.
+
+// | `trustee-password`
+// | The Identity service trustee user password.
+
+// | `application-credential-id`
+// | The ID of an application credential to authenticate with. An `application-credential-secret` must be set along with this parameter.
+
+// | `application-credential-name`
+// | The name of an application credential to authenticate with. If `application-credential-id` is not set, the user name and domain must be set.
+
+// | `application-credential-secret`
+// | The secret of an application credential to authenticate with.
+
+// | `tls-insecure`
+// | Whether or not to verify the server's TLS certificate. If set to `true`, the certificate is not verified. By default, the certificate is verified.
+// |===
+
+
+// [id="ccm-config-networking-options"]
+// == Networking options
+
+// |===
+// | Option | Description
+
+// | `ipv6-support-disabled`
+// | Whether or not IPv6 is supported as indicated by a boolean value. By default, this option is `false`.
+
+// | `public-network-name`
+// | The name of an {rh-openstack} Networking service, or Neutron, external network. The CCM uses this option when retrieving the external IP address of a Kubernetes node. This value can contain multiple names. Specified networks are bitwise ORed. The default value is `""`.
+
+// | `internal-network-name`
+// | The name of a Networking service internal network. The CCM uses this option when retrieving the internal IP address of a Kubernetes node. This value can contain multiple names. Specified networks are bitwise ORed. The default value is `""`.
+
+// | `address-sort-order`
+// | This configuration key affects how the provider reports node addresses to Kubernetes node resources. The default order depends on the hard-coded order in which the provider queries addresses and what the cloud returns. A specific order is not guaranteed.
+
+//  To override this behavior, specify a comma-separated list of CIDR addresses. CCM sorts and groups all addresses that match the list in a prioritized manner, wherein the first retrieved item has  a higher priority than the last. Addresses that do not match the list remain in their default order. The default value is `""`.
+
+// This option can be useful if you have multiple or dual-stack interfaces attached to a node that need a user-controlled, deterministic way of sorting addresses.
+// |===
+
+[id="ccm-config-lb-options"]
+== Load balancer options
+
+CCM supports several load balancer options for deployments that use Octavia. 
+
+[NOTE]
+====
+Neutron-LBaaS support is deprecated. 
+====
+
+|===
+| Option | Description
+
+| `enabled`
+| Whether or not to enable the `LoadBalancer` type of services integration. The default value is `true`.
+
+// Always enforced.
+// | `use-octavia`
+// | Whether or not to use Octavia for the `LoadBalancer` type of service implementation rather than Neutron-LBaaS. The default value is `true`.
+
+| `floating-network-id`
+| Optional. The external network used to create floating IP addresses for load balancer virtual IP addresses (VIPs). If there are multiple external networks in the cloud, this option must be set or the user must specify `loadbalancer.openstack.org/floating-network-id` in the service annotation.
+
+| `floating-subnet-id`
+| Optional. The external network subnet used to create floating IP addresses for the load balancer VIP. Can be overridden by the service annotation `loadbalancer.openstack.org/floating-subnet-id`.
+
+| `floating-subnet`
+| Optional. A name pattern (glob or regular expression if starting with `~`) for the external network subnet used to create floating IP addresses for the load balancer VIP. Can be overridden by the service annotation `loadbalancer.openstack.org/floating-subnet`. If multiple subnets match the pattern, the first one with available IP addresses is used.
+
+| `floating-subnet-tags`
+| Optional. Tags for the external network subnet used to create floating IP addresses for the load balancer VIP. Can be overridden by the service annotation `loadbalancer.openstack.org/floating-subnet-tags`. If multiple subnets match these tags, the first one with available IP addresses is used.
+
+If the {rh-openstack} network is configured with sharing disabled, for example, with the `--no-share` flag used during creation, this option is unsupported. Set the network to share to use this option.
+
+| `lb-method`
+| The load balancing algorithm used to create the load balancer pool.
+For the Amphora provider the value can be `ROUND_ROBIN`, `LEAST_CONNECTIONS`, or `SOURCE_IP`. The default value is `ROUND_ROBIN`.
+
+For the OVN provider, only the `SOURCE_IP_PORT` algorithm is supported.
+
+For the Amphora provider, if using the `LEAST_CONNECTIONS` or `SOURCE_IP` methods, configure the `create-monitor` option as `true`  in the `cloud-provider-config` config map on the `openshift-config` namespace and `ETP:Local` on the load-balancer type service to allow balancing algorithm enforcement in the client to service endpoint connections.
+
+| `lb-provider`
+| Optional. Used to specify the provider of the load balancer, for example, `amphora` or `octavia`. Only the Amphora and Octavia providers are supported.
+
+| `lb-version`
+| Optional. The load balancer API version. Only `"v2"` is supported.
+
+| `subnet-id`
+| The ID of the Networking service subnet on which load balancer VIPs are created. 
+
+// This ID is also used to create pool members if `member-subnet-id` is not set.
+
+// | `member-subnet-id`
+// | ID of the Neutron network on which to create the members of the load balancer. The load balancer gets another network port on this subnet. Defaults to `subnet-id` if not set.
+
+| `network-id`
+| The ID of the Networking service network on which load balancer VIPs are created. Unnecessary if `subnet-id` is set.
+
+// | `manage-security-groups`
+// | If the Neutron security groups should be managed separately. Default: false
+
+| `create-monitor`
+| Whether or not to create a health monitor for the service load balancer. A health monitor is required for services that declare `externalTrafficPolicy: Local`. The default value is `false`.
+
+This option is unsupported if you use {rh-openstack} earlier than version 17 with the `ovn` provider.
+
+| `monitor-delay`
+| The interval in seconds by which probes are sent to members of the load balancer. The default value is `5`.
+
+| `monitor-max-retries`
+| The number of successful checks that are required to change the operating status of a load balancer member to `ONLINE`. The valid range is `1` to `10`, and the default value is `1`.
+
+| `monitor-timeout`
+| The time in seconds that a monitor waits to connect to the back end before it times out. The default value is `3`.
+
+| `internal-lb`
+| Whether or not to create an internal load balancer without floating IP addresses. The default value is `false`.
+
+// | `cascade-delete`
+// | Determines whether or not to perform cascade deletion of load balancers. Default: true.
+
+// | `flavor-id`
+// | The id of the loadbalancer flavor to use. Uses octavia default if not set.
+
+// | `availability-zone`
+// | The name of the loadbalancer availability zone to use. It is applicable if use-octavia is set to True and requires Octavia API version 2.14 or later (Ussuri release). The Octavia availability zone capabilities will not be used if it is not set. The parameter will be ignored if the Octavia version doesn't support availability zones yet.
+
+| `LoadBalancerClass "ClassName"`
+a| This is a config section that comprises a set of options:
+
+ * `floating-network-id`
+ * `floating-subnet-id`
+ * `floating-subnet`
+ * `floating-subnet-tags`
+ * `network-id`
+ * `subnet-id`
+
+//  * `member-subnet-id`
+
+The behavior of these options is the same as that of the identically named options in the load balancer section of the CCM config file.
+
+You can set the `ClassName` value by specifying the service annotation `loadbalancer.openstack.org/class`. 
+
+// | `enable-ingress-hostname`
+// | Used with proxy protocol (set by annotation `loadbalancer.openstack.org/proxy-protocol: "true"`) by adding a dns suffix (nip.io) to the load balancer IP address. Default false.
+
+//  This option is currently a workaround for the issue https://github.com/kubernetes/ingress-nginx/issues/3996, should be removed or refactored after the Kubernetes [KEP-1860](https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/1860-kube-proxy-IP-node-binding) is implemented.
+
+// | `ingress-hostname-suffix`
+// | The dns suffix to the load balancer IP address when using proxy protocol. Default nip.io
+
+//  This option is currently a workaround for the issue https://github.com/kubernetes/ingress-nginx/issues/3996, should be removed or refactored after the Kubernetes [KEP-1860](https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/1860-kube-proxy-IP-node-binding) is implemented.
+
+// | `default-tls-container-ref`
+// | Reference to a tls container. This option works with Octavia, when this option is set then the cloud provider will create an Octavia Listener of type TERMINATED_HTTPS for a TLS Terminated loadbalancer.
+
+//  Format for tls container ref: `https://{keymanager_host}/v1/containers/{uuid}`
+//  Check `container-store` parameter if you want to disable validation.
+
+// | `container-store`
+// | Optional. Used to specify the store of the tls-container-ref, e.g. "barbican" or "external" - other store will cause a warning log. Default value - `barbican` - existence of tls container ref would always be performed. If set to `external` format for tls container ref will not be validated.
+
+| `max-shared-lb`
+| The maximum number of services that can share a load balancer. The default value is `2`.
+|===
+
+// [id="ccm-config-metadata-options"]
+// == Metadata options
+
+// |===
+// | Option | Description
+
+// | `search-order`
+// | This configuration key affects the way that the provider retrieves metadata that relates to the instances in which it runs. The default value of `configDrive,metadataService` results in the provider retrieving metadata that relates to the instance from, if available, the config drive first,and then the metadata service. Alternative values are:
+//  * `configDrive`: Only retrieve instance metadata from the configuration drive.
+//  * `metadataService`: Only retrieve instance metadata from the metadata service.
+//  * `metadataService,configDrive`: Retrieve instance metadata from the metadata service first if available, and then retrieve instance metadata from the configuration drive.
+// |===
+
+// ### Multi region support (alpha)
+
+// | environment variable `OS_CCM_REGIONAL` is set to `true` - allow CCM to set ProviderID with region name `${ProviderName}://${REGION}/${instance-id}`. Default: false.
+
+[id="cluster-cloud-controller-config-overrides"]
+== Options that the Operator overrides
+
+The CCM Operator overrides the following options, which you might recognize from configuring {rh-openstack}. Do not configure them yourself. They are included in this document for informational purposes only.
+
+|===
+| Option | Description
+
+| `auth-url`
+| The {rh-openstack} Identity service URL. For example, `http://128.110.154.166/identity`.
+
+| `os-endpoint-type`
+| The type of endpoint to use from the service catalog. 
+
+// If unset, public endpoints are used.
+
+| `username`
+| The Identity service user name. 
+
+// Leave this option unset if you are using Identity service application credentials.
+
+| `password`
+| The Identity service user password. 
+
+// Leave this option unset if you are using Identity service application credentials.
+
+| `domain-id`
+| The Identity service user domain ID. 
+
+// Leave this option unset if you are using Identity service application credentials.
+
+| `domain-name`
+| The Identity service user domain name. 
+
+// This option is not required if you set `domain-id`.
+
+| `tenant-id`
+| The Identity service project ID. Leave this option unset if you are using Identity service application credentials.
+
+In version 3 of the Identity API, which changed the identifier `tenant` to `project`, the value of `tenant-id` is automatically mapped to the project construct in the API. 
+
+| `tenant-name`
+| The Identity service project name. 
+
+| `tenant-domain-id`
+| The Identity service project domain ID.
+
+| `tenant-domain-name`
+| The Identity service project domain name.
+
+| `user-domain-id`
+| The Identity service user domain ID.
+
+| `user-domain-name`
+| The Identity service user domain name.
+
+| `use-clouds`
+a| Whether or not to fetch authorization credentials from a `clouds.yaml` file. Options set in this section are prioritized over values read from the `clouds.yaml` file. 
+
+CCM searches for the file in the following places:
+
+. The value of the `clouds-file` option.
+. A file path stored in the environment variable `OS_CLIENT_CONFIG_FILE`.
+. The directory `pkg/openstack`.
+. The directory `~/.config/openstack`.
+. The directory `/etc/openstack`.
+
+| `clouds-file`
+| The file path of a `clouds.yaml` file. It is used if the `use-clouds` option is set to `true`.
+
+| `cloud`
+| The named cloud in the `clouds.yaml` file that you want to use. It is used if the `use-clouds` option is set to `true`.
+|===