|
4 | 4 | [](/LICENSE-2.0) |
5 | 5 | [](https://golang.org) |
6 | 6 |
|
7 | | -A Cloud Controller Manager to facilitate Kubernetes deployments on Cloudstack. |
| 7 | +A Kubernetes Cloud Controller Manager (CCM) for Apache CloudStack. It provides node metadata, lifecycle management, and load balancer integration for Kubernetes clusters running on CloudStack. |
8 | 8 |
|
9 | | -Based on the old in-tree CloudStack provider in Kubernetes that was removed. |
| 9 | +## Quick Start |
10 | 10 |
|
11 | | -Refer: |
12 | | -* https://github.com/kubernetes/kubernetes/tree/release-1.15/pkg/cloudprovider/providers/cloudstack |
13 | | -* https://github.com/kubernetes/enhancements/issues/672 |
14 | | -* https://github.com/kubernetes/enhancements/issues/88 |
15 | | - |
16 | | -## Deployment |
17 | | - |
18 | | -The CloudStack Kubernetes Provider is automatically deployed when a Kubernetes Cluster is created on CloudStack 4.16+ |
19 | | - |
20 | | -In order to communicate with CloudStack, a separate service user **kubeadmin** is created in the same account as the cluster owner. |
21 | | -The provider uses this user's API keys to get the details of the cluster as well as update the networking rules. It is imperative that this user |
22 | | -is not altered or have its keys regenerated. |
23 | | - |
24 | | -The provider can also be manually deployed as follows : |
25 | | - |
26 | | -### Kubernetes |
27 | | - |
28 | | -Prebuilt containers are posted [here](https://github.com/leaseweb/cloudstack-kubernetes-provider/pkgs/container/cloudstack-kubernetes-provider). |
29 | | - |
30 | | -To configure API access to your CloudStack management server, you need to create a secret containing a `cloud-config` |
31 | | -that is suitable for your environment. |
32 | | - |
33 | | -`cloud-config` should look like this: |
34 | | -```ini |
35 | | -[Global] |
36 | | -api-url = <CloudStack API URL> |
37 | | -api-key = <CloudStack API Key> |
38 | | -secret-key = <CloudStack API Secret> |
39 | | -project-id = <CloudStack Project UUID (optional)> |
40 | | -zone = <CloudStack Zone Name (optional)> |
41 | | -ssl-no-verify = <Disable SSL certificate validation: true or false (optional)> |
42 | | -``` |
43 | | - |
44 | | -The access token needs to be able to fetch VM information and deploy load balancers in the project or domain where the nodes reside. |
45 | | - |
46 | | -To create the secret, use the following command: |
47 | | -```bash |
48 | | -kubectl -n kube-system create secret generic cloudstack-secret --from-file=cloud-config |
49 | | -``` |
50 | | - |
51 | | -You can then use the provided example [deployment.yaml](/deployment.yaml) to deploy the controller: |
52 | 11 | ```bash |
53 | | -kubectl apply -f deployment.yaml |
54 | | -``` |
55 | | - |
56 | | -### Protocols |
57 | | - |
58 | | -This CCM supports TCP, UDP and [TCP-Proxy](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) LoadBalancer deployments. |
59 | | - |
60 | | -For UDP and Proxy Protocol support, CloudStack 4.6 or later is required. |
61 | | - |
62 | | -Since kube-proxy does not support the Proxy Protocol or UDP, you should connect this directly to pods, for example by deploying a DaemonSet and setting `hostPort: <TCP port>` on the desired container port. |
63 | | -Important: The service running in the pod must support the chosen protocol. Do not try to enable TCP-Proxy when the service only supports regular TCP. |
64 | | - |
65 | | -[traefik-ingress-controller.yml](/traefik-ingress-controller.yml) contains a basic deployment for the Træfik ingress controller that illustrates how to use it with the proxy protocol. |
66 | | - |
67 | | -For the nginx ingress controller, please refer to the official documentation at [kubernetes.github.io/ingress-nginx/deploy](https://kubernetes.github.io/ingress-nginx/deploy/). After applying the deployment, patch it for proxy protocol support with the provided fragment: |
68 | | - |
69 | | -```bash |
70 | | -kubectl apply -f nginx-ingress-controller-patch.yml |
71 | | -``` |
72 | | - |
73 | | -### Node Labels |
74 | | - |
75 | | -:warning: **The node name must match the host name, so the controller can fetch and assign metadata from CloudStack.** |
76 | | - |
77 | | -It is recommended to launch `kubelet` with the following parameter: |
78 | | - |
| 12 | +helm install cloud-controller-manager charts/cloud-controller-manager/ \ |
| 13 | + --namespace kube-system \ |
| 14 | + --set cloudConfig.global.api-url="https://cloudstack.example.com/client/api" \ |
| 15 | + --set cloudConfig.global.api-key="YOUR_API_KEY" \ |
| 16 | + --set cloudConfig.global.secret-key="YOUR_SECRET_KEY" |
79 | 17 | ``` |
80 | | ---register-with-taints=node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule |
81 | | -``` |
82 | | - |
83 | | -This will treat the node as 'uninitialized' and cause the CCM to apply metadata labels from CloudStack automatically. |
84 | | - |
85 | | -Supported labels for Kubernetes versions up to 1.16 are: |
86 | | -* kubernetes.io/hostname (= the instance name) |
87 | | -* beta.kubernetes.io/instance-type (= the compute offering) |
88 | | -* failure-domain.beta.kubernetes.io/zone (= the zone) |
89 | | -* failure-domain.beta.kubernetes.io/region (also = the zone) |
90 | | - |
91 | | -Supported labels for Kubernetes versions 1.17 and later are: |
92 | | -* kubernetes.io/hostname (= the instance name) |
93 | | -* node.kubernetes.io/instance-type (= the compute offering) |
94 | | -* topology.kubernetes.io/zone (= the zone) |
95 | | -* topology.kubernetes.io/region (also = the zone) |
96 | | - |
97 | | -It is also possible to trigger this process manually by issuing the following command: |
98 | | - |
99 | | -``` |
100 | | -kubectl taint nodes <my-node-without-labels> node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule |
101 | | -``` |
102 | | - |
103 | | -## Migration Guide |
104 | | - |
105 | | -There are several notable differences to the old Kubernetes CloudStack cloud provider that need to be taken into |
106 | | -account when migrating from the old cloud provider to the standalone controller. |
107 | | - |
108 | | -### Load Balancer |
109 | | - |
110 | | -Load balancer rule names now include the protocol in addition to the LB name and service port. |
111 | | -This was added to distinguish tcp, udp and tcp-proxy services operating on the same port. |
112 | | -Without this change, it would not be possible to map a service that runs on both TCP and UDP port 8000, for example. |
113 | | - |
114 | | -:warning: **If you have existing rules, remove them before the migration, and add them back afterwards.** |
115 | | - |
116 | | -If you don't do this, you will end up with duplicate rules for the same service, which won't work. |
117 | 18 |
|
118 | | -### Metadata |
| 19 | +## Documentation |
119 | 20 |
|
120 | | -Since the controller is now intended to be run inside a pod and not on the node, it will not be able to fetch metadata from the Virtual Router's DHCP server. |
121 | | - |
122 | | -Instead, it first obtains the name of the node from Kubernetes, then fetches information from the CloudStack API. |
| 21 | +| Guide | Description | |
| 22 | +|-------|-------------| |
| 23 | +| [Getting Started](docs/getting-started.md) | Installation via Helm or Kubernetes manifests, node setup | |
| 24 | +| [Configuration](docs/configuration.md) | Cloud config reference, Helm chart values | |
| 25 | +| [Load Balancer](docs/load-balancer.md) | Protocols, annotations, IP management | |
| 26 | +| [Development](docs/development.md) | Building, testing, local development | |
123 | 27 |
|
124 | 28 | ## Development |
125 | 29 |
|
126 | | -### Building |
127 | | - |
128 | | -At least Go 1.21 is required to build cloudstack-ccm. |
129 | | - |
130 | | -To build the controller with correct versioning, some build flags need to be passed. |
131 | | -A Makefile is provided that sets these build flags to automatically derived values. |
132 | | - |
133 | | -```bash |
134 | | -go get github.com/apache/cloudstack-kubernetes-provider |
135 | | -cd ${GOPATH}/src/github.com/apache/cloudstack-kubernetes-provider |
136 | | -make |
137 | | -``` |
138 | | - |
139 | | -To build the cloudstack-cloud-controller-manager container, please use the provided Dockerfile. |
140 | | -The Makefile will also with that and properly tag the resulting container. |
141 | | - |
142 | 30 | ```bash |
143 | | -make docker |
| 31 | +make # Build |
| 32 | +make docker # Build container image |
| 33 | +make lint # Lint |
| 34 | +make test # Test |
144 | 35 | ``` |
145 | 36 |
|
146 | | -### Testing |
147 | | - |
148 | | -You need a local instance of the CloudStack Management Server or a 'real' one to connect to. |
149 | | -The CCM supports the same cloud-config configuration file format used by [the cs tool](https://github.com/exoscale/cs), |
150 | | -so you can simply point it to that. |
151 | | - |
152 | | -```bash |
153 | | -./cloudstack-ccm --cloud-provider cloudstack --cloud-config ~/.cloud-config --master k8s-apiserver |
154 | | -``` |
155 | | - |
156 | | -Replace k8s-apiserver with the host name of your Kubernetes development clusters's API server. |
157 | | - |
158 | | -If you don't have a 'real' CloudStack installation, you can also launch a local [simulator instance](https://hub.docker.com/r/cloudstack/simulator) instead. This is very useful for dry-run testing. |
| 37 | +See [docs/development.md](docs/development.md) for details. |
159 | 38 |
|
160 | 39 | ## Copyright |
161 | 40 |
|
|
0 commit comments