diff --git a/README.md b/README.md index a2002d4c..9558695f 100644 --- a/README.md +++ b/README.md @@ -4,158 +4,37 @@ [![](https://img.shields.io/badge/license-Apache%202.0-blue.svg?color=%23282661&logo=apache&style=flat-square "Apache 2.0 license")](/LICENSE-2.0) [![](https://img.shields.io/badge/language-Go-%235adaff.svg?logo=go&style=flat-square "Go language")](https://golang.org) -A Cloud Controller Manager to facilitate Kubernetes deployments on Cloudstack. +A Kubernetes Cloud Controller Manager (CCM) for Apache CloudStack. It provides node metadata, lifecycle management, and load balancer integration for Kubernetes clusters running on CloudStack. -Based on the old in-tree CloudStack provider in Kubernetes that was removed. +## Quick Start -Refer: -* https://github.com/kubernetes/kubernetes/tree/release-1.15/pkg/cloudprovider/providers/cloudstack -* https://github.com/kubernetes/enhancements/issues/672 -* https://github.com/kubernetes/enhancements/issues/88 - -## Deployment - -The CloudStack Kubernetes Provider is automatically deployed when a Kubernetes Cluster is created on CloudStack 4.16+ - -In order to communicate with CloudStack, a separate service user **kubeadmin** is created in the same account as the cluster owner. -The provider uses this user's API keys to get the details of the cluster as well as update the networking rules. It is imperative that this user -is not altered or have its keys regenerated. - -The provider can also be manually deployed as follows : - -### Kubernetes - -Prebuilt containers are posted [here](https://github.com/leaseweb/cloudstack-kubernetes-provider/pkgs/container/cloudstack-kubernetes-provider). - -To configure API access to your CloudStack management server, you need to create a secret containing a `cloud-config` -that is suitable for your environment. - -`cloud-config` should look like this: -```ini -[Global] -api-url = -api-key = -secret-key = -project-id = -zone = -ssl-no-verify = -``` - -The access token needs to be able to fetch VM information and deploy load balancers in the project or domain where the nodes reside. - -To create the secret, use the following command: -```bash -kubectl -n kube-system create secret generic cloudstack-secret --from-file=cloud-config -``` - -You can then use the provided example [deployment.yaml](/deployment.yaml) to deploy the controller: ```bash -kubectl apply -f deployment.yaml -``` - -### Protocols - -This CCM supports TCP, UDP and [TCP-Proxy](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) LoadBalancer deployments. - -For UDP and Proxy Protocol support, CloudStack 4.6 or later is required. - -Since kube-proxy does not support the Proxy Protocol or UDP, you should connect this directly to pods, for example by deploying a DaemonSet and setting `hostPort: ` on the desired container port. -Important: The service running in the pod must support the chosen protocol. Do not try to enable TCP-Proxy when the service only supports regular TCP. - -[traefik-ingress-controller.yml](/traefik-ingress-controller.yml) contains a basic deployment for the Træfik ingress controller that illustrates how to use it with the proxy protocol. - -For the nginx ingress controller, please refer to the official documentation at [kubernetes.github.io/ingress-nginx/deploy](https://kubernetes.github.io/ingress-nginx/deploy/). After applying the deployment, patch it for proxy protocol support with the provided fragment: - -```bash -kubectl apply -f nginx-ingress-controller-patch.yml -``` - -### Node Labels - -:warning: **The node name must match the host name, so the controller can fetch and assign metadata from CloudStack.** - -It is recommended to launch `kubelet` with the following parameter: - +helm install cloud-controller-manager charts/cloud-controller-manager/ \ + --namespace kube-system \ + --set cloudConfig.global.api-url="https://cloudstack.example.com/client/api" \ + --set cloudConfig.global.api-key="YOUR_API_KEY" \ + --set cloudConfig.global.secret-key="YOUR_SECRET_KEY" ``` ---register-with-taints=node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule -``` - -This will treat the node as 'uninitialized' and cause the CCM to apply metadata labels from CloudStack automatically. - -Supported labels for Kubernetes versions up to 1.16 are: -* kubernetes.io/hostname (= the instance name) -* beta.kubernetes.io/instance-type (= the compute offering) -* failure-domain.beta.kubernetes.io/zone (= the zone) -* failure-domain.beta.kubernetes.io/region (also = the zone) - -Supported labels for Kubernetes versions 1.17 and later are: -* kubernetes.io/hostname (= the instance name) -* node.kubernetes.io/instance-type (= the compute offering) -* topology.kubernetes.io/zone (= the zone) -* topology.kubernetes.io/region (also = the zone) - -It is also possible to trigger this process manually by issuing the following command: - -``` -kubectl taint nodes node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule -``` - -## Migration Guide - -There are several notable differences to the old Kubernetes CloudStack cloud provider that need to be taken into -account when migrating from the old cloud provider to the standalone controller. - -### Load Balancer - -Load balancer rule names now include the protocol in addition to the LB name and service port. -This was added to distinguish tcp, udp and tcp-proxy services operating on the same port. -Without this change, it would not be possible to map a service that runs on both TCP and UDP port 8000, for example. - -:warning: **If you have existing rules, remove them before the migration, and add them back afterwards.** - -If you don't do this, you will end up with duplicate rules for the same service, which won't work. -### Metadata +## Documentation -Since the controller is now intended to be run inside a pod and not on the node, it will not be able to fetch metadata from the Virtual Router's DHCP server. - -Instead, it first obtains the name of the node from Kubernetes, then fetches information from the CloudStack API. +| Guide | Description | +|-------|-------------| +| [Getting Started](docs/getting-started.md) | Installation via Helm or Kubernetes manifests, node setup | +| [Configuration](docs/configuration.md) | Cloud config reference, Helm chart values | +| [Load Balancer](docs/load-balancer.md) | Protocols, annotations, IP management | +| [Development](docs/development.md) | Building, testing, local development | ## Development -### Building - -At least Go 1.21 is required to build cloudstack-ccm. - -To build the controller with correct versioning, some build flags need to be passed. -A Makefile is provided that sets these build flags to automatically derived values. - -```bash -go get github.com/apache/cloudstack-kubernetes-provider -cd ${GOPATH}/src/github.com/apache/cloudstack-kubernetes-provider -make -``` - -To build the cloudstack-cloud-controller-manager container, please use the provided Dockerfile. -The Makefile will also with that and properly tag the resulting container. - ```bash -make docker +make # Build +make docker # Build container image +make lint # Lint +make test # Test ``` -### Testing - -You need a local instance of the CloudStack Management Server or a 'real' one to connect to. -The CCM supports the same cloud-config configuration file format used by [the cs tool](https://github.com/exoscale/cs), -so you can simply point it to that. - -```bash -./cloudstack-ccm --cloud-provider cloudstack --cloud-config ~/.cloud-config --master k8s-apiserver -``` - -Replace k8s-apiserver with the host name of your Kubernetes development clusters's API server. - -If you don't have a 'real' CloudStack installation, you can also launch a local [simulator instance](https://hub.docker.com/r/cloudstack/simulator) instead. This is very useful for dry-run testing. +See [docs/development.md](docs/development.md) for details. ## Copyright diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 00000000..5c89478f --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,51 @@ +# Configuration + +## Cloud Config + +The CCM reads its CloudStack credentials from a `cloud-config` file in INI format. When using Kubernetes manifests, this file is mounted from a secret. When using the Helm chart, it is generated from the `cloudConfig` values. + +```ini +[Global] +api-url = +api-key = +secret-key = +project-id = +zone = +ssl-no-verify = +``` + +| Field | Required | Description | +|-------|----------|-------------| +| `api-url` | Yes | Full URL to the CloudStack API endpoint | +| `api-key` | Yes | API key for authentication | +| `secret-key` | Yes | Secret key for authentication | +| `project-id` | No | UUID of the CloudStack project. Required when nodes are in a project | +| `zone` | No | CloudStack zone name to scope operations to | +| `ssl-no-verify` | No | Set to `true` to skip TLS certificate verification | + +The API credentials need permission to fetch VM information and manage load balancers in the project or domain where the nodes reside. + +## Helm Chart Values + +The chart is located at [`charts/cloud-controller-manager/`](../charts/cloud-controller-manager/). Below are the key values. See [`values.yaml`](../charts/cloud-controller-manager/values.yaml) for the full reference. + +| Value | Default | Description | +|-------|---------|-------------| +| `replicaCount` | `1` | Number of CCM replicas. Leader election is automatically enabled when > 1 | +| `image.repository` | `ghcr.io/leaseweb/cloudstack-kubernetes-provider` | Container image repository | +| `image.tag` | Chart `appVersion` | Container image tag | +| `nodeSelector` | `node-role.kubernetes.io/control-plane: ""` | Node selector for scheduling | +| `tolerations` | Uninitialized, CriticalAddonsOnly, control-plane, not-ready | Pod tolerations | +| `enabledControllers` | `[cloud-node, cloud-node-lifecycle, route, service]` | List of controllers to enable | +| `cluster.name` | `kubernetes` | Cluster name passed to the CCM | +| `logVerbosityLevel` | `2` | klog verbosity level | +| `hostNetwork` | `true` | Run pods with host networking | +| `dnsPolicy` | `ClusterFirstWithHostNet` | DNS policy (should match `hostNetwork`) | +| `secret.enabled` | `true` | Mount cloud-config from a Kubernetes secret | +| `secret.create` | `true` | Create the secret from `cloudConfig` values. Set to `false` to use a pre-existing secret | +| `secret.name` | `cloud-config` | Name of the secret | +| `cloudConfig.global.api-url` | `""` | CloudStack API URL | +| `cloudConfig.global.api-key` | `""` | CloudStack API key | +| `cloudConfig.global.secret-key` | `""` | CloudStack secret key | +| `serviceMonitor` | `{}` | Prometheus ServiceMonitor configuration | +| `priorityClassName` | `system-node-critical` | Pod priority class | diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 00000000..986a19a8 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,29 @@ +# Development + +## Requirements + +- Go 1.24+ +- Docker (for building container images) + +## Building + +A Makefile is provided that sets build flags with automatically derived version information. + +```bash +make # Build the cloudstack-ccm binary +make docker # Build and tag the container image +make lint # Run linting (fmt, vet, golangci-lint) +make test # Run tests, vet, and format checks +``` + +## Local Testing + +You can run the CCM locally against a Kubernetes cluster and CloudStack API: + +```bash +./cloudstack-ccm --cloud-provider cloudstack --cloud-config /path/to/cloud-config --master +``` + +Replace `` with the hostname or address of your Kubernetes API server. + +If you don't have a CloudStack installation available, you can use the CloudStack [simulator image](https://hub.docker.com/r/cloudstack/simulator) for dry-run testing. diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 00000000..fb3c3bde --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,77 @@ +# Getting Started + +## Prerequisites + +- A Kubernetes cluster running on Apache CloudStack +- CloudStack API credentials (API key and secret key) with permissions to manage VMs and load balancers +- `kubectl` configured to access your cluster +- `helm` (optional, for Helm-based installation) + +## Installation + +### Helm Chart (recommended) + +Install from the bundled chart in [`charts/cloud-controller-manager/`](../charts/cloud-controller-manager/): + +```bash +helm install cloud-controller-manager charts/cloud-controller-manager/ \ + --namespace kube-system \ + --set cloudConfig.global.api-url="https://cloudstack.example.com/client/api" \ + --set cloudConfig.global.api-key="YOUR_API_KEY" \ + --set cloudConfig.global.secret-key="YOUR_SECRET_KEY" +``` + +See [`charts/cloud-controller-manager/values.yaml`](../charts/cloud-controller-manager/values.yaml) for the full list of configurable values, or refer to the [Configuration](configuration.md) guide. + +### Kubernetes Manifests + +1. Create a `cloud-config` file: + + ```ini + [Global] + api-url = https://cloudstack.example.com/client/api + api-key = YOUR_API_KEY + secret-key = YOUR_SECRET_KEY + ``` + + See [Configuration](configuration.md) for all available fields. + +2. Create the secret: + + ```bash + kubectl -n kube-system create secret generic cloudstack-secret --from-file=cloud-config + ``` + +3. Apply RBAC and deployment manifests: + + ```bash + kubectl apply -f deploy/k8s/rbac.yaml + kubectl apply -f deploy/k8s/deployment.yaml + ``` + +Prebuilt container images are available at [ghcr.io/leaseweb/cloudstack-kubernetes-provider](https://github.com/leaseweb/cloudstack-kubernetes-provider/pkgs/container/cloudstack-kubernetes-provider). + +## Node Setup + +**The node name must match the CloudStack VM hostname** so the controller can fetch and assign metadata. + +It is recommended to launch `kubelet` with the following flag: + +``` +--register-with-taints=node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule +``` + +This marks the node as uninitialized, causing the CCM to automatically apply metadata labels from CloudStack: + +| Label | Value | +|-------|-------| +| `kubernetes.io/hostname` | Instance name | +| `node.kubernetes.io/instance-type` | Compute offering | +| `topology.kubernetes.io/zone` | CloudStack zone | +| `topology.kubernetes.io/region` | CloudStack zone | + +To trigger this process manually on an existing node: + +```bash +kubectl taint nodes node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule +``` diff --git a/docs/load-balancer.md b/docs/load-balancer.md new file mode 100644 index 00000000..1b7b9298 --- /dev/null +++ b/docs/load-balancer.md @@ -0,0 +1,78 @@ +# Load Balancer + +## Overview + +When you create a Kubernetes `Service` with `type: LoadBalancer`, the CCM provisions CloudStack load balancer rules and associates a public IP address with the service. The load balancer name is derived from the service name, namespace, and protocol. + +## Protocols + +The CCM supports three protocols for load balancer rules: + +| Protocol | Description | +|----------|-------------| +| **TCP** | Standard TCP load balancing | +| **UDP** | UDP load balancing (CloudStack 4.6+) | +| **TCP-Proxy** | TCP with [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) header injection (CloudStack 4.6+) | + +Since kube-proxy does not support PROXY protocol or UDP forwarding, these protocols should target pods directly. Deploy your application as a DaemonSet and use `hostPort` on the container port to bypass kube-proxy. + +Example ingress controller configurations are provided in [`deploy/ingress-sample/`](../deploy/ingress-sample/): + +- `traefik-ingress-controller.yml` - Traefik with PROXY protocol +- `nginx-ingress-controller-patch.yml` - Patch for the nginx ingress controller to enable PROXY protocol + +> **Important:** The service running in the pod must support the chosen protocol. Do not enable TCP-Proxy when the service only supports regular TCP. + +## Annotations Reference + +All annotations use the prefix `service.beta.kubernetes.io/`. + +| Annotation | Type | Description | +|------------|------|-------------| +| `cloudstack-load-balancer-proxy-protocol` | string | Enable PROXY protocol on TCP ports. The value specifies which ports to enable it on | +| `cloudstack-load-balancer-hostname` | string | Hostname for in-cluster access when using PROXY protocol. Workaround for [kubernetes/kubernetes#66607](https://github.com/kubernetes/kubernetes/issues/66607) | +| `cloudstack-load-balancer-address` | string | Request a specific IP address for the load balancer. Replaces the deprecated `spec.loadBalancerIP` field | +| `cloudstack-load-balancer-keep-ip` | bool | When set to `"true"`, prevents the public IP from being released when the service is deleted | +| `cloudstack-load-balancer-id` | string | (Managed) CloudStack public IP UUID. Set automatically by the CCM for efficient ID-based lookups | +| `cloudstack-load-balancer-network-id` | string | (Managed) CloudStack network UUID. Set automatically by the CCM together with `load-balancer-id` | + +## IP Management + +### Requesting a specific IP + +Set the `cloudstack-load-balancer-address` annotation to request a specific public IP: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: my-service + annotations: + service.beta.kubernetes.io/cloudstack-load-balancer-address: "203.0.113.10" +spec: + type: LoadBalancer + ports: + - port: 80 + targetPort: 8080 +``` + +> This replaces the deprecated `spec.loadBalancerIP` field, which is still supported as a fallback. + +### Retaining an IP after service deletion + +To prevent the public IP from being released when the service is deleted, set: + +```yaml +metadata: + annotations: + service.beta.kubernetes.io/cloudstack-load-balancer-keep-ip: "true" +``` + +This is useful when you want to recreate a service with the same IP address. + +### Changing the IP of an existing service + +Live IP reassignment is not supported. To change the IP address of a load balancer: + +1. Delete the existing service +2. Create a new service with the desired IP in the `cloudstack-load-balancer-address` annotation