Skip to content

Commit 970891b

Browse files
mrwulfscotty-c
mrwulf
authored and
scotty-c
committed
Expose a lot of params - mostly to ease deployments in a different overlay network range (#82)
* Exposing some cluster cidr and service ip params. * Reorder yaml * Fixing puppet types and using a better taint. * Pass in the kube-proxy version and fix some formatting * Inherit the package version from the kubernetes version * Removing some noise and adding some spacing * Changing order of hiera file to move editable fields to the top. Adding more hosts to the api-server's cert. * Adding tolerations for master, switching to iptables mode for kube-proxy (the recommended/default now), and easing initial access to kube-controller-manager * Pass in the desired dns service ip * Exposing the docker version * Make some tests pass * Make even more tests pass * Make even more tests pass +1 * Trigger another travis build * Make even more tests pass +1 * Cleaning up a duplicate param * Filling out the documentation and adding an ENV for the docker pass-thru * Really pass cni_cluster_cidr and cni_node_cidr as nil/undef if the cni plugin doesn't override it * Adding a new param for cluster_service_cidr * Stupid syntax * copy and paste strikes again * Fixed the condition, but forgot the actual value
1 parent 85e1900 commit 970891b

25 files changed

+300
-147
lines changed

‎README.md‎

Lines changed: 55 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ If you do not already have Docker installed on your workstation, install it [her
5252

5353
The kubetool docker image takes each of the parameters as environment variables. When run as follows it will output a `kubernetes.yaml` file in your current working directory:
5454
```
55-
docker run --rm -v $(pwd):/mnt -e OS=debian -e VERSION=1.9.2 -e CONTAINER_RUNTIME=docker -e CNI_PROVIDER=weave -e FQDN=kubernetes -e IP=172.17.10.101 -e BOOTSTRAP_CONTROLLER_IP=172.17.10.101 -e ETCD_INITIAL_CLUSTER="etcd-kube-master=http://172.17.10.101:2380" -e ETCD_IP="%{::ipaddress_enp0s8}" -e KUBE_API_ADVERTISE_ADDRESS="%{::ipaddress_enp0s8}" -e INSTALL_DASHBOARD=true puppet/kubetool
55+
docker run --rm -v $(pwd):/mnt -e OS=debian -e VERSION=1.9.2 -e CONTAINER_RUNTIME=docker -e CNI_PROVIDER=weave -e FQDN=kubernetes -e IP=172.17.10.101 -e BOOTSTRAP_CONTROLLER_IP=172.17.10.101 -e ETCD_INITIAL_CLUSTER="etcd-kube-master=http://172.17.10.101:2380" -e ETCD_IP="%{::ipaddress_enp0s8}" -e KUBE_API_ADVERTISE_ADDRESS="%{::ipaddress_enp0s8}" -e SERVICE_API_IP=10.96.0.1 -e INSTALL_DASHBOARD=true puppet/kubetool
5656
```
5757

5858
The parameters are:
@@ -61,10 +61,12 @@ The parameters are:
6161
* `VERSION`: the version of kubernetes you want to deploy
6262
* `CONTAINER_RUNTIME`: the container runtime kubernetes will use, this can only be set to `docker` or `cri_containerd`
6363
* `CNI_PROVIDER` : This is the CNI network to install. This can be set to `weave` or `flannel`
64-
* `FQDN`: the cluster fqdn.
64+
* `FQDN`: the cluster api fqdn. Should resolve to `IP`.
65+
* `IP`: the cluster api IP. When in production, should be load balanced between the controllers.
6566
* `BOOTSTRAP_CONTROLLER_IP`: the ip address of the controller puppet will use to create things like cluster role bindings, kube dns, and the Kubernetes dashboard.
6667
* `ETCD_INITIAL_CLUSTER`: the server addresses. When in production, include three, five, or seven nodes for etcd.
67-
* `ETCD_IP` and `ETCD_IP KUBE_API_ADVERTISE_ADDRESS`: we recommend passing the fact for the interface to be used by the cluster.
68+
* `ETCD_IP` and `KUBE_API_ADVERTISE_ADDRESS`: the IP each etcd/apiserver instance will use on each controller. We recommend passing the fact for the interface to be used by the cluster.
69+
* `SERVICE_API_IP`: the IP that the kubernetes service will be available on inside the cluster. Dependent on overlay network range.
6870
* `INSTALL_DASHBOARD`: a boolean to install the dashboard or not.
6971

7072
The kubetool creates a `kubernetes.yaml` file. To view the file contents on
@@ -88,7 +90,7 @@ After your `kubernetes.yaml` file has been added to the Hiera directory on your
8890

8991
#### Bootstrap Controller
9092

91-
A bootstrap controller is the node a cluster uses to add cluster addons (such as kube dns, cluster role bindings etc). After the cluster is bootstrapped, the bootstrap controller becomes a normal controller.
93+
A bootstrap controller is the node a cluster uses to add cluster addons (such as kube dns, cluster role bindings etc). *After the cluster is bootstrapped, the bootstrap controller should be changed to a normal controller.*
9294

9395
To make a node a bootstrap controller, add the following code to the manifest:
9496

@@ -437,6 +439,55 @@ Allows the user to override the label of a node.
437439

438440
Defaults for hostname
439441

442+
#### `docker_version`
443+
444+
This is the version of the docker runtime that you want to install.
445+
446+
Defaults to `1.12.6` on RedHat
447+
Defaults to `1.12.0-0~xenial` on Debian
448+
449+
#### `kube_dns_version`
450+
451+
The version of kube DNS you would like to install
452+
453+
Defaults to `1.14.2`
454+
455+
#### `kube_proxy_version`
456+
457+
The version of kube-proxy you would like to install
458+
459+
Defaults to match the `kubernetes_version`
460+
461+
#### `cni_cluster_cidr`
462+
463+
The overlay (internal) network range to use.
464+
465+
Defaults to `undef` (don't specify for kube-controller-manager). kube_tool sets this per cni provider.
466+
467+
#### `cni_node_cidr`
468+
469+
This triggers `allocate-node-cidrs=true` to be added to the controller-manager.
470+
471+
Defaults to `false`.
472+
473+
#### `cluster_service_cidr`
474+
475+
The overlay (internal) network range to use for cluster services. This should be a subset of the `cni_cluster_cidr`. `kube_api_ip` and `kube_dns_ip` should be in this range.
476+
477+
Defaults to `undef` (don't specify for kube-apiserver). kube_tool sets this per cni provider.
478+
479+
#### `kube_dns_ip`
480+
481+
The cluster service IP to use for kube-dns.
482+
483+
Defaults to `10.96.0.10`
484+
485+
#### `kube_api_ip`
486+
487+
The cluster service IP to use for the kube api.
488+
489+
Defaults to `10.96.0.1`
490+
440491
## Limitations
441492

442493
This module supports [Kubernetes 1.6](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#v160) and above.

‎manifests/cluster_roles.pp‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,9 @@
3434

3535
if $kubernetes_version =~ /1[.](8|9)[.]\d/ {
3636

37-
exec { 'Create role biniding for system nodes':
37+
exec { 'Create role binding for system nodes':
3838
command => 'kubectl set subject clusterrolebinding system:node --group=system:nodes',
39+
unless => 'kubectl describe clusterrolebinding system:node | tr -s \' \' | grep \'Group system:nodes\'',
3940
}
4041
}
4142
}

‎manifests/config.pp‎

Lines changed: 17 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,11 @@
55
String $kubernetes_version = $kubernetes::kubernetes_version,
66
String $container_runtime = $kubernetes::container_runtime,
77
Optional[String] $cni_cluster_cidr = $kubernetes::cni_cluster_cidr,
8-
Optional[String] $cni_node_cidr = $kubernetes::cni_node_cidr,
8+
Optional[Boolean] $cni_node_cidr = $kubernetes::cni_node_cidr,
9+
Optional[String] $cluster_service_cidr = $kubernetes::cluster_service_cidr,
10+
String $kube_dns_ip = $kubernetes::kube_dns_ip,
911
String $kube_dns_version = $kubernetes::kube_dns_version,
12+
String $kube_proxy_version = $kubernetes::kube_proxy_version,
1013
String $kubernetes_fqdn = $kubernetes::kubernetes_fqdn,
1114
Boolean $controller = $kubernetes::controller,
1215
Boolean $bootstrap_controller = $kubernetes::bootstrap_controller,
@@ -107,24 +110,24 @@
107110

108111
#TODO fix secuirty issue that the bootstarp token is left on the server.
109112

110-
file {'/etc/kubernetes/secrets/bootstraptoken.yaml':
111-
ensure => present,
112-
content => template('kubernetes/secrets/bootstraptoken.yaml.erb'),
113-
require => File['/etc/kubernetes/secrets/'],
114-
}
115-
116-
$kube_addons_files.each | String $addons_file | {
117-
file { "/etc/kubernetes/addons/${addons_file}":
113+
file {'/etc/kubernetes/secrets/bootstraptoken.yaml':
118114
ensure => present,
119-
content => template("kubernetes/addons/${addons_file}.erb"),
120-
require => File['/etc/kubernetes/addons'],
121-
}
122-
}
115+
content => template('kubernetes/secrets/bootstraptoken.yaml.erb'),
116+
require => File['/etc/kubernetes/secrets/'],
117+
}
118+
119+
$kube_addons_files.each | String $addons_file | {
120+
file { "/etc/kubernetes/addons/${addons_file}":
121+
ensure => present,
122+
content => template("kubernetes/addons/${addons_file}.erb"),
123+
require => File['/etc/kubernetes/addons'],
124+
}
125+
}
123126

124127
file {'/root/admin.conf':
125128
ensure => present,
126129
content => template('kubernetes/admin.conf.erb'),
127-
}
130+
}
128131

129132
file { '/etc/profile.d/kubectl.sh':
130133
mode => '0644',

‎manifests/init.pp‎

Lines changed: 31 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,11 @@
2020
# It can only be set to "cri_containerd" or "docker"
2121
# Defaults to docker
2222
#
23+
# [*docker_version]
24+
# This is the version of the docker runtime that you want to install.
25+
# Defaults to 1.12.6 on RedHat
26+
# Defaults to 1.12.0-0~xenial on Debian
27+
#
2328
# [*cni_version*]
2429
# The version of the cni package you would like to install
2530
# Defaults to 0.6.0
@@ -201,10 +206,30 @@
201206
# We will support any networking provider that supports cni
202207
# This defaults to https://git.io/weave-kube-1.6
203208
#
209+
# [*cni_cluster_cidr*]
210+
# The overlay (internal) network range to use.
211+
# Defaults to undef. kube_tool sets this per cni provider.
212+
#
213+
# [*cni_node_cidr*]
214+
# This triggers 'allocate-node-cidrs=true' to be added to the controller-manager.
215+
# Defaults to false.
216+
#
204217
# [*install_dashboard*]
205218
# This is a bool that determines if the kubernetes dashboard is installed.
206219
# Defaults to false
207220
#
221+
# [*kube_dns_ip*]
222+
# The service IP to use for kube-dns.
223+
# Defaults to 10.96.0.10
224+
#
225+
# [*kube_api_ip*]
226+
# The service IP to use for the kube api.
227+
# Defaults to 10.96.0.1
228+
#
229+
# [*kube_proxy_version*]
230+
# The version of kube-proxy you would like to install
231+
# Defaults to $kubernetes_version
232+
#
208233
#
209234
# Authors
210235
# -------
@@ -213,15 +238,16 @@
213238
#
214239
#
215240
#
216-
217241
class kubernetes (
218242
String $kubernetes_version = $kubernetes::params::kubernetes_version,
219243
Optional[String] $kubernetes_package_version = $kubernetes::params::kubernetes_package_version,
220244
String $kubernetes_fqdn = $kubernetes::params::kubernetes_fqdn,
221245
String $container_runtime = $kubernetes::params::container_runtime,
246+
Optional[String] $docker_version = $kubernetes::params::docker_version,
222247
Optional[String] $cni_version = $kubernetes::params::cni_version,
223248
Optional[String] $cni_cluster_cidr = $kubernetes::params::cni_cluster_cidr,
224-
Optional[String] $cni_node_cidr = $kubernetes::params::cni_node_cidr,
249+
Optional[Boolean] $cni_node_cidr = $kubernetes::params::cni_node_cidr,
250+
Optional[String] $cluster_service_cidr = $kubernetes::params::cluster_service_cidr,
225251
String $kube_dns_version = $kubernetes::params::kube_dns_version,
226252
Boolean $controller = $kubernetes::params::controller,
227253
Boolean $bootstrap_controller = $kubernetes::params::bootstrap_controller,
@@ -267,6 +293,9 @@
267293
Boolean $install_dashboard = $kubernetes::params::install_dashboard,
268294
Boolean $taint_master = $kubernetes::params::taint_master,
269295
String $node_label = $kubernetes::params::node_label,
296+
String $kube_dns_ip = $kubernetes::params::kube_dns_ip,
297+
String $kube_api_ip = $kubernetes::params::kube_api_ip,
298+
String $kube_proxy_version = $kubernetes::params::kube_proxy_version,
270299

271300
) inherits kubernetes::params {
272301

‎manifests/kube_addons.pp‎

Lines changed: 50 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
class kubernetes::kube_addons (
33

44
Boolean $bootstrap_controller = $kubernetes::bootstrap_controller,
5-
Optional[String]$cni_network_provider = $kubernetes::cni_network_provider,
5+
Optional[String] $cni_network_provider = $kubernetes::cni_network_provider,
66
Boolean $install_dashboard = $kubernetes::install_dashboard,
77
String $kubernetes_version = $kubernetes::kubernetes_version,
88
Boolean $controller = $kubernetes::controller,
@@ -20,64 +20,65 @@
2020

2121
if $bootstrap_controller {
2222

23-
$addon_dir = '/etc/kubernetes/addons'
23+
$addon_dir = '/etc/kubernetes/addons'
2424

25-
exec { 'Install cni network provider':
26-
command => "kubectl apply -f ${cni_network_provider}",
27-
onlyif => 'kubectl get nodes',
28-
}
29-
30-
exec { 'Create kube proxy service account':
31-
command => 'kubectl create -f kube-proxy-sa.yaml',
32-
cwd => $addon_dir,
33-
subscribe => File['/etc/kubernetes/addons/kube-proxy-sa.yaml'],
34-
refreshonly => true,
35-
require => Exec['Install cni network provider'],
36-
}
37-
38-
exec { 'Create kube proxy ConfigMap':
39-
command => 'kubectl create -f kube-proxy.yaml',
40-
cwd => $addon_dir,
41-
subscribe => File['/etc/kubernetes/addons/kube-proxy.yaml'],
42-
refreshonly => true,
43-
require => Exec['Create kube proxy service account'],
44-
}
45-
46-
exec { 'Create kube proxy daemonset':
47-
command => 'kubectl create -f kube-proxy-daemonset.yaml',
48-
cwd => $addon_dir,
49-
subscribe => File['/etc/kubernetes/addons/kube-proxy-daemonset.yaml'],
50-
refreshonly => true,
51-
require => Exec['Create kube proxy ConfigMap'],
52-
}
25+
exec { 'Install cni network provider':
26+
command => "kubectl apply -f ${cni_network_provider}",
27+
onlyif => 'kubectl get nodes',
28+
}
5329

54-
exec { 'Create kube dns service account':
55-
command => 'kubectl create -f kube-dns-sa.yaml',
56-
cwd => $addon_dir,
57-
subscribe => File['/etc/kubernetes/addons/kube-dns-sa.yaml'],
58-
refreshonly => true,
30+
exec { 'Create kube proxy service account':
31+
command => 'kubectl apply -f kube-proxy-sa.yaml',
32+
cwd => $addon_dir,
33+
subscribe => File['/etc/kubernetes/addons/kube-proxy-sa.yaml'],
34+
refreshonly => true,
35+
require => Exec['Install cni network provider'],
5936
}
6037

61-
exec { 'Create kube dns service':
62-
command => 'kubectl create -f kube-dns-service.yaml',
63-
cwd => $addon_dir,
64-
subscribe => File['/etc/kubernetes/addons/kube-dns-service.yaml'],
65-
refreshonly => true,
66-
require => Exec['Create kube dns service account'],
38+
exec { 'Create kube proxy ConfigMap':
39+
command => 'kubectl apply -f kube-proxy.yaml',
40+
cwd => $addon_dir,
41+
subscribe => File['/etc/kubernetes/addons/kube-proxy.yaml'],
42+
refreshonly => true,
43+
require => Exec['Create kube proxy service account'],
6744
}
6845

69-
exec { 'Create kube dns deployment':
70-
command => 'kubectl create -f kube-dns-deployment.yaml',
71-
cwd => $addon_dir,
72-
subscribe => File['/etc/kubernetes/addons/kube-dns-deployment.yaml'],
73-
refreshonly => true,
74-
require => Exec['Create kube dns service account'],
46+
exec { 'Create kube proxy daemonset':
47+
command => 'kubectl apply -f kube-proxy-daemonset.yaml',
48+
cwd => $addon_dir,
49+
subscribe => File['/etc/kubernetes/addons/kube-proxy-daemonset.yaml'],
50+
refreshonly => true,
51+
require => Exec['Create kube proxy ConfigMap'],
7552
}
53+
54+
exec { 'Create kube dns service account':
55+
command => 'kubectl apply -f kube-dns-sa.yaml',
56+
cwd => $addon_dir,
57+
subscribe => File['/etc/kubernetes/addons/kube-dns-sa.yaml'],
58+
refreshonly => true,
59+
}
60+
61+
exec { 'Create kube dns service':
62+
command => 'kubectl apply -f kube-dns-service.yaml',
63+
cwd => $addon_dir,
64+
subscribe => File['/etc/kubernetes/addons/kube-dns-service.yaml'],
65+
refreshonly => true,
66+
require => Exec['Create kube dns service account'],
67+
}
68+
69+
exec { 'Create kube dns deployment':
70+
command => 'kubectl apply -f kube-dns-deployment.yaml',
71+
cwd => $addon_dir,
72+
subscribe => File['/etc/kubernetes/addons/kube-dns-deployment.yaml'],
73+
refreshonly => true,
74+
require => Exec['Create kube dns service account'],
75+
}
7676
}
7777

7878
if $controller {
7979
exec { 'Assign master role to controller':
8080
command => "kubectl label node ${node_label} node-role.kubernetes.io/master=",
81+
onlyif => 'kubectl get nodes',
8182
unless => "kubectl describe nodes ${node_label} | tr -s ' ' | grep 'Roles: master'",
8283
}
8384

@@ -93,9 +94,9 @@
9394
}
9495

9596
exec { 'Taint master node':
96-
command => "kubectl taint nodes ${node_label} key=value:NoSchedule",
97+
command => "kubectl taint nodes ${node_label} node-role.kubernetes.io/master=value:NoSchedule",
9798
onlyif => 'kubectl get nodes',
98-
unless => "kubectl describe nodes ${node_label} | tr -s ' ' | grep 'Taints: key=value:NoSchedule'"
99+
unless => "kubectl describe nodes ${node_label} | tr -s ' ' | grep 'Taints: node-role.kubernetes.io/master=value:NoSchedule'"
99100
}
100101
}
101102
}

‎manifests/packages.pp‎

Lines changed: 3 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
Optional[String] $kubernetes_package_version = $kubernetes::kubernetes_package_version,
66
String $container_runtime = $kubernetes::container_runtime,
77
String $cni_version = $kubernetes::cni_version,
8-
8+
String $docker_version = $kubernetes::docker_version,
99
) {
1010

1111
$kube_packages = ['kubelet', 'kubectl']
@@ -23,15 +23,9 @@
2323

2424
if $container_runtime == 'docker' {
2525
case $::osfamily {
26-
'Debian' : {
27-
package { 'docker-engine':
28-
ensure => '1.12.0-0~xenial',
29-
}
30-
}
31-
32-
'RedHat' : {
26+
'Debian','RedHat' : {
3327
package { 'docker-engine':
34-
ensure => '1.12.6',
28+
ensure => $docker_version,
3529
}
3630
}
3731

0 commit comments

Comments
 (0)