Skip to content

docs: Reorganize documentation into docs/ folder #116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hrak merged 1 commit into from
Mar 3, 2026
Merged

docs: Reorganize documentation into docs/ folder #116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
159 changes: 19 additions & 140 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,158 +4,37 @@
[![](https://img.shields.io/badge/license-Apache%202.0-blue.svg?color=%23282661&logo=apache&style=flat-square "Apache 2.0 license")](/LICENSE-2.0)
[![](https://img.shields.io/badge/language-Go-%235adaff.svg?logo=go&style=flat-square "Go language")](https://golang.org)

A Cloud Controller Manager to facilitate Kubernetes deployments on Cloudstack.
A Kubernetes Cloud Controller Manager (CCM) for Apache CloudStack. It provides node metadata, lifecycle management, and load balancer integration for Kubernetes clusters running on CloudStack.

Based on the old in-tree CloudStack provider in Kubernetes that was removed.
## Quick Start

Refer:
* https://github.com/kubernetes/kubernetes/tree/release-1.15/pkg/cloudprovider/providers/cloudstack
* https://github.com/kubernetes/enhancements/issues/672
* https://github.com/kubernetes/enhancements/issues/88

## Deployment

The CloudStack Kubernetes Provider is automatically deployed when a Kubernetes Cluster is created on CloudStack 4.16+

In order to communicate with CloudStack, a separate service user **kubeadmin** is created in the same account as the cluster owner.
The provider uses this user's API keys to get the details of the cluster as well as update the networking rules. It is imperative that this user
is not altered or have its keys regenerated.

The provider can also be manually deployed as follows :

### Kubernetes

Prebuilt containers are posted [here](https://github.com/leaseweb/cloudstack-kubernetes-provider/pkgs/container/cloudstack-kubernetes-provider).

To configure API access to your CloudStack management server, you need to create a secret containing a `cloud-config`
that is suitable for your environment.

`cloud-config` should look like this:
```ini
[Global]
api-url = <CloudStack API URL>
api-key = <CloudStack API Key>
secret-key = <CloudStack API Secret>
project-id = <CloudStack Project UUID (optional)>
zone = <CloudStack Zone Name (optional)>
ssl-no-verify = <Disable SSL certificate validation: true or false (optional)>
```

The access token needs to be able to fetch VM information and deploy load balancers in the project or domain where the nodes reside.

To create the secret, use the following command:
```bash
kubectl -n kube-system create secret generic cloudstack-secret --from-file=cloud-config
```

You can then use the provided example [deployment.yaml](/deployment.yaml) to deploy the controller:
```bash
kubectl apply -f deployment.yaml
```

### Protocols

This CCM supports TCP, UDP and [TCP-Proxy](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) LoadBalancer deployments.

For UDP and Proxy Protocol support, CloudStack 4.6 or later is required.

Since kube-proxy does not support the Proxy Protocol or UDP, you should connect this directly to pods, for example by deploying a DaemonSet and setting `hostPort: <TCP port>` on the desired container port.
Important: The service running in the pod must support the chosen protocol. Do not try to enable TCP-Proxy when the service only supports regular TCP.

[traefik-ingress-controller.yml](/traefik-ingress-controller.yml) contains a basic deployment for the Træfik ingress controller that illustrates how to use it with the proxy protocol.

For the nginx ingress controller, please refer to the official documentation at [kubernetes.github.io/ingress-nginx/deploy](https://kubernetes.github.io/ingress-nginx/deploy/). After applying the deployment, patch it for proxy protocol support with the provided fragment:

```bash
kubectl apply -f nginx-ingress-controller-patch.yml
```

### Node Labels

:warning: **The node name must match the host name, so the controller can fetch and assign metadata from CloudStack.**

It is recommended to launch `kubelet` with the following parameter:

helm install cloud-controller-manager charts/cloud-controller-manager/ \
--namespace kube-system \
--set cloudConfig.global.api-url="https://cloudstack.example.com/client/api" \
--set cloudConfig.global.api-key="YOUR_API_KEY" \
--set cloudConfig.global.secret-key="YOUR_SECRET_KEY"
```
--register-with-taints=node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
```

This will treat the node as 'uninitialized' and cause the CCM to apply metadata labels from CloudStack automatically.

Supported labels for Kubernetes versions up to 1.16 are:
* kubernetes.io/hostname (= the instance name)
* beta.kubernetes.io/instance-type (= the compute offering)
* failure-domain.beta.kubernetes.io/zone (= the zone)
* failure-domain.beta.kubernetes.io/region (also = the zone)

Supported labels for Kubernetes versions 1.17 and later are:
* kubernetes.io/hostname (= the instance name)
* node.kubernetes.io/instance-type (= the compute offering)
* topology.kubernetes.io/zone (= the zone)
* topology.kubernetes.io/region (also = the zone)

It is also possible to trigger this process manually by issuing the following command:

```
kubectl taint nodes <my-node-without-labels> node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
```

## Migration Guide

There are several notable differences to the old Kubernetes CloudStack cloud provider that need to be taken into
account when migrating from the old cloud provider to the standalone controller.

### Load Balancer

Load balancer rule names now include the protocol in addition to the LB name and service port.
This was added to distinguish tcp, udp and tcp-proxy services operating on the same port.
Without this change, it would not be possible to map a service that runs on both TCP and UDP port 8000, for example.

:warning: **If you have existing rules, remove them before the migration, and add them back afterwards.**

If you don't do this, you will end up with duplicate rules for the same service, which won't work.

### Metadata
## Documentation

Since the controller is now intended to be run inside a pod and not on the node, it will not be able to fetch metadata from the Virtual Router's DHCP server.

Instead, it first obtains the name of the node from Kubernetes, then fetches information from the CloudStack API.
| Guide | Description |
|-------|-------------|
| [Getting Started](docs/getting-started.md) | Installation via Helm or Kubernetes manifests, node setup |
| [Configuration](docs/configuration.md) | Cloud config reference, Helm chart values |
| [Load Balancer](docs/load-balancer.md) | Protocols, annotations, IP management |
| [Development](docs/development.md) | Building, testing, local development |

## Development

### Building

At least Go 1.21 is required to build cloudstack-ccm.

To build the controller with correct versioning, some build flags need to be passed.
A Makefile is provided that sets these build flags to automatically derived values.

```bash
go get github.com/apache/cloudstack-kubernetes-provider
cd ${GOPATH}/src/github.com/apache/cloudstack-kubernetes-provider
make
```

To build the cloudstack-cloud-controller-manager container, please use the provided Dockerfile.
The Makefile will also with that and properly tag the resulting container.

```bash
make docker
make # Build
make docker # Build container image
make lint # Lint
make test # Test
```

### Testing

You need a local instance of the CloudStack Management Server or a 'real' one to connect to.
The CCM supports the same cloud-config configuration file format used by [the cs tool](https://github.com/exoscale/cs),
so you can simply point it to that.

```bash
./cloudstack-ccm --cloud-provider cloudstack --cloud-config ~/.cloud-config --master k8s-apiserver
```

Replace k8s-apiserver with the host name of your Kubernetes development clusters's API server.

If you don't have a 'real' CloudStack installation, you can also launch a local [simulator instance](https://hub.docker.com/r/cloudstack/simulator) instead. This is very useful for dry-run testing.
See [docs/development.md](docs/development.md) for details.

## Copyright

Expand Down
51 changes: 51 additions & 0 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Configuration

## Cloud Config

The CCM reads its CloudStack credentials from a `cloud-config` file in INI format. When using Kubernetes manifests, this file is mounted from a secret. When using the Helm chart, it is generated from the `cloudConfig` values.

```ini
[Global]
api-url = <CloudStack API URL>
api-key = <CloudStack API Key>
secret-key = <CloudStack API Secret>
project-id = <CloudStack Project UUID (optional)>
zone = <CloudStack Zone Name (optional)>
ssl-no-verify = <Disable SSL certificate validation: true or false (optional)>
```

| Field | Required | Description |
|-------|----------|-------------|
| `api-url` | Yes | Full URL to the CloudStack API endpoint |
| `api-key` | Yes | API key for authentication |
| `secret-key` | Yes | Secret key for authentication |
| `project-id` | No | UUID of the CloudStack project. Required when nodes are in a project |
| `zone` | No | CloudStack zone name to scope operations to |
| `ssl-no-verify` | No | Set to `true` to skip TLS certificate verification |

The API credentials need permission to fetch VM information and manage load balancers in the project or domain where the nodes reside.

## Helm Chart Values

The chart is located at [`charts/cloud-controller-manager/`](../charts/cloud-controller-manager/). Below are the key values. See [`values.yaml`](../charts/cloud-controller-manager/values.yaml) for the full reference.

| Value | Default | Description |
|-------|---------|-------------|
| `replicaCount` | `1` | Number of CCM replicas. Leader election is automatically enabled when > 1 |
| `image.repository` | `ghcr.io/leaseweb/cloudstack-kubernetes-provider` | Container image repository |
| `image.tag` | Chart `appVersion` | Container image tag |
| `nodeSelector` | `node-role.kubernetes.io/control-plane: ""` | Node selector for scheduling |
| `tolerations` | Uninitialized, CriticalAddonsOnly, control-plane, not-ready | Pod tolerations |
| `enabledControllers` | `[cloud-node, cloud-node-lifecycle, route, service]` | List of controllers to enable |
| `cluster.name` | `kubernetes` | Cluster name passed to the CCM |
| `logVerbosityLevel` | `2` | klog verbosity level |
| `hostNetwork` | `true` | Run pods with host networking |
| `dnsPolicy` | `ClusterFirstWithHostNet` | DNS policy (should match `hostNetwork`) |
| `secret.enabled` | `true` | Mount cloud-config from a Kubernetes secret |
| `secret.create` | `true` | Create the secret from `cloudConfig` values. Set to `false` to use a pre-existing secret |
| `secret.name` | `cloud-config` | Name of the secret |
| `cloudConfig.global.api-url` | `""` | CloudStack API URL |
| `cloudConfig.global.api-key` | `""` | CloudStack API key |
| `cloudConfig.global.secret-key` | `""` | CloudStack secret key |
| `serviceMonitor` | `{}` | Prometheus ServiceMonitor configuration |
| `priorityClassName` | `system-node-critical` | Pod priority class |
29 changes: 29 additions & 0 deletions docs/development.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Development

## Requirements

- Go 1.24+
- Docker (for building container images)

## Building

A Makefile is provided that sets build flags with automatically derived version information.

```bash
make # Build the cloudstack-ccm binary
make docker # Build and tag the container image
make lint # Run linting (fmt, vet, golangci-lint)
make test # Run tests, vet, and format checks
```

## Local Testing

You can run the CCM locally against a Kubernetes cluster and CloudStack API:

```bash
./cloudstack-ccm --cloud-provider cloudstack --cloud-config /path/to/cloud-config --master <k8s-apiserver>
```

Replace `<k8s-apiserver>` with the hostname or address of your Kubernetes API server.

If you don't have a CloudStack installation available, you can use the CloudStack [simulator image](https://hub.docker.com/r/cloudstack/simulator) for dry-run testing.
77 changes: 77 additions & 0 deletions docs/getting-started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Getting Started

## Prerequisites

- A Kubernetes cluster running on Apache CloudStack
- CloudStack API credentials (API key and secret key) with permissions to manage VMs and load balancers
- `kubectl` configured to access your cluster
- `helm` (optional, for Helm-based installation)

## Installation

### Helm Chart (recommended)

Install from the bundled chart in [`charts/cloud-controller-manager/`](../charts/cloud-controller-manager/):

```bash
helm install cloud-controller-manager charts/cloud-controller-manager/ \
--namespace kube-system \
--set cloudConfig.global.api-url="https://cloudstack.example.com/client/api" \
--set cloudConfig.global.api-key="YOUR_API_KEY" \
--set cloudConfig.global.secret-key="YOUR_SECRET_KEY"
```

See [`charts/cloud-controller-manager/values.yaml`](../charts/cloud-controller-manager/values.yaml) for the full list of configurable values, or refer to the [Configuration](configuration.md) guide.

### Kubernetes Manifests

1. Create a `cloud-config` file:

```ini
[Global]
api-url = https://cloudstack.example.com/client/api
api-key = YOUR_API_KEY
secret-key = YOUR_SECRET_KEY
```

See [Configuration](configuration.md) for all available fields.

2. Create the secret:

```bash
kubectl -n kube-system create secret generic cloudstack-secret --from-file=cloud-config
```

3. Apply RBAC and deployment manifests:

```bash
kubectl apply -f deploy/k8s/rbac.yaml
kubectl apply -f deploy/k8s/deployment.yaml
```

Prebuilt container images are available at [ghcr.io/leaseweb/cloudstack-kubernetes-provider](https://github.com/leaseweb/cloudstack-kubernetes-provider/pkgs/container/cloudstack-kubernetes-provider).

## Node Setup

**The node name must match the CloudStack VM hostname** so the controller can fetch and assign metadata.

It is recommended to launch `kubelet` with the following flag:

```
--register-with-taints=node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
```

This marks the node as uninitialized, causing the CCM to automatically apply metadata labels from CloudStack:

| Label | Value |
|-------|-------|
| `kubernetes.io/hostname` | Instance name |
| `node.kubernetes.io/instance-type` | Compute offering |
| `topology.kubernetes.io/zone` | CloudStack zone |
| `topology.kubernetes.io/region` | CloudStack zone |

To trigger this process manually on an existing node:

```bash
kubectl taint nodes <node-name> node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
```
Loading
Loading