Skip to content

Commit 9938156

Browse files
openshift-merge-bot[bot]openshift-merge-bot[bot]
authored
Merge pull request #7699 from vr4manta/SPLAT-1208
SPLAT-1208: replace UPI terraform with powercli
2 parents 94fec8e + cbe9796 commit 9938156

File tree

7 files changed

+1332
-2
lines changed

7 files changed

+1332
-2
lines changed

‎images/installer/Dockerfile.upi.ci‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ RUN yum update -y && \
3939
unzip \
4040
openssh-clients \
4141
openssl \
42+
powershell \
4243
python3-pyOpenSSL \
4344
python2-pyyaml \
4445
python3-pyyaml \
@@ -93,6 +94,11 @@ RUN mkdir /output && HOME=/output && \
9394
ibmcloud version && \
9495
ibmcloud plugin list
9596

97+
# Install VMware plugin for powershell. Create settings directory /output/.local/share/VMware/PowerCLI
98+
RUN pwsh -Command 'Install-Module VMware.PowerCLI -Force -Scope AllUsers' && \
99+
pwsh -Command 'Install-Module -Name EPS -RequiredVersion 1.0 -Force -Scope AllUsers' && \
100+
mkdir -p /output/.local/share/VMware/PowerCLI && chmod -R 777 /output/.local
101+
96102
RUN chown 1000:1000 /output && chmod -R g=u "/output/.bluemix/"
97103
USER 1000:1000
98104
ENV PATH /bin

‎upi/vsphere/README.md‎

Lines changed: 213 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,220 @@
1-
# Pre-Requisites
1+
This project shows two ways to install an UPI cluster. We will discuss how to install using one of these two techniques:
2+
- Terraform
3+
- PowerShell
4+
5+
# Table of Contents
6+
- [PowerShell](#PowerShell)
7+
- [Pre-Requisites](#pre-requisites)
8+
- [PowerShell Setup](#powershell-setup)
9+
- [VMware.PowerCLI](#vmwarepowercli)
10+
- [EPS](#eps)
11+
- [Script Configuration]
12+
- [OpenShift Installation Configuration]()
13+
- [Terraform](#Terraform)
14+
- [Pre-Requisites](#pre-requisites-1)
15+
- [Build a Cluster](#build-a-cluster-1)
16+
17+
# PowerShell
18+
This section will describe the process to generate the vSphere VMs using PowerShell and the supplied scripts in this module.
19+
20+
## Pre-requisites
21+
* PowerShell
22+
* PowerShell VMware.PowerCLI Module
23+
* PowerShell EPS Module
24+
25+
## PowerShell Setup
26+
27+
PowerShell will need to have a couple of plugin installed in order for our script to work. The plugins we need to install are VMware.PowerCLI and EPS.
28+
29+
### VMware.PowerCLI
30+
31+
To install the VMware.PowerCLI, you can run the following command:
32+
33+
```shell
34+
pwsh -Command 'Install-Module VMware.PowerCLI -Force -Scope CurrentUser'
35+
```
36+
37+
### EPS
38+
39+
To install the EPS module, you can run the following command:
40+
41+
```shell
42+
pwsh -Command 'Install-Module -Name EPS -RequiredVersion 1.0 -Force -Scope CurrentUser'
43+
```
44+
45+
### Generating CLI Credentials
46+
47+
The PowerShell scripts require that a credentials file be generated with the credentials to be used for generating the vSphere resources. This does not have to be the credentials used by the OpenShift cluster via the install-config.yaml, but must have all permissions to create folders, tags, templates, and vms. To generate the credentials files, run:
48+
49+
```shell
50+
pwsh -command "\$User='<username>';\$Password=ConvertTo-SecureString -String '<password>' -AsPlainText -Force;\$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList \$User, \$Password;\$Credential | Export-Clixml secrets/vcenter-creds.xml"
51+
```
52+
53+
Be sure to modify `<username>` to be the username for vCenter and `<password>` to the your password. The output of this needs to go into `secrets/vcenter-creds.xml`. Make sure the secrets directory exists before running the credentials generation command above.
54+
55+
## Script Configuration
56+
57+
The PowerShell script provided by this project provides examples on how to do several aspects to creating a UPI cluster environment. It is configurable to do as much or as little as you need. For the CI build process, we will handle all install-config.yaml configuration, uploading of templates, and monitoring of installation progress. This project can handle doing all that as well if configured appropriately.
58+
59+
### Behavioral Configurations
60+
61+
These properties are used to control how the script works. All of the properties will dictate how much or little work the script will do.
62+
63+
| Property | Description |
64+
|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
65+
| createInstallConfig | Enable script to create install config based on configuration of variables.ps1. If set to false, the script will not generate install-config.yaml. Note, install-config.yaml will be needed if you want this script to generate the ignition files for vm creation. |
66+
| downloadInstaller | Enable script to download installer to be used. If not downloading the installer, the installer must be placed in the same directory as this script. The installer is needed to determine what template to upload to the cluster for vm cloning. |
67+
| uploadTemplateOva | Enable script to upload OVA template to be used for all VM being created. |
68+
| generateIgnitions | Enable script to generate ignition configs. This is normally used when install-config.yaml is provided to script, but need script to generate the ignition configs for VMs. |
69+
| waitForComplete | This option has the script step through the process of waiting for installation complete. Most of this functionality is provided by `openshift-install wait-for`. The script will will check for when api is ready, bootstrap complete, accept CSRs and then for all COs to be done installing. |
70+
| delayVMStart | This option has the script delay the start of the VMs after their creation. |
71+
72+
73+
### vCenter Configuration
74+
75+
These properties are related to vCenter. These will be used by script to create the install-config.yaml as well as define a single failure domain if no failure domains are configured in the variables.ps1 file.
76+
77+
| Property | Description |
78+
|-------------|------------------------------------------------------------------------------------------------------------------------------------------------|
79+
| vcenter | This property specifies the vCenter URL |
80+
| username | Specifies the username used to connect to vCenter. This is only used in generating the install-config.yaml information for use by the cluster |
81+
| password | The password to be used to connect to vCenter. This is only used in generating the install-config.yaml information for use by the cluster |
82+
| portgroup | The default port group to use. This property will be used when no failure domains are defined. |
83+
| datastore | The default data store to use. This property will be used when no failure domains are defined. |
84+
| datacenter | The default data center to use. This property will be used when no failure domains are defined. |
85+
86+
87+
### Cluster Configuration
88+
89+
These properties are used to configure the new cluster. These properties will also be used to determine how to create the VMs.
90+
91+
| Property | Description |
92+
|-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
93+
| clustername | The name of the cluster to create. |
94+
| basedomain | The base domain to use for the cluster. |
95+
| sshkeypath | The path of the ssh key to use when generating ignition configs. |
96+
| failure_domains | Contains the failure domains for the cluster. This needs to be a JSON formatted configuration following what is provided in the example below table. |
97+
98+
Example of a failure domain configuration:
99+
100+
```powershell
101+
$failure_domains = @"
102+
[
103+
{
104+
"datacenter": "IBMCloud",
105+
"cluster": "vcs-mdcnc-workload-1",
106+
"datastore": "mdcnc-ds-1",
107+
"network": "ocp-ci-seg-14"
108+
},{
109+
"datacenter": "IBMCloud",
110+
"cluster": "vcs-mdcnc-workload-2",
111+
"datastore": "mdcnc-ds-2",
112+
"network": "ocp-ci-seg-14"
113+
},{
114+
"datacenter": "IBMCloud",
115+
"cluster": "vcs-mdcnc-workload-3",
116+
"datastore": "mdcnc-ds-3",
117+
"network": "ocp-ci-seg-14"
118+
},{
119+
"datacenter": "datacenter-2",
120+
"cluster": "vcs-mdcnc-workload-4",
121+
"datastore": "mdcnc-ds-4",
122+
"network": "ocp-ci-seg-14"
123+
}
124+
]
125+
"@
126+
```
127+
128+
### VM Configuration
129+
130+
| Property | Description |
131+
|----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------|
132+
| vm_template | Name of the existing vm template in vCenter to use. Use this option to prevent script from uploading a template and use an existing template. |
133+
| dns | DNS server for network configuration of all VMs. |
134+
| gateway | Gateway for network configuration of all VMs. |
135+
| netwask | The network mask for all VMs. |
136+
| lb_ip_address | The IP address to use for the load balancer VM. |
137+
| bootstrap_ip_address | The IP address to use for the bootstrap VM. |
138+
| control_plane_memory | Amount of memory to assign each control plane VM. This value is in MB. |
139+
| control_plane_num_cpus | Number of CPUs to assign to each control plane VM. |
140+
| control_plane_count | Number of control plane VMs. |
141+
| control_plane_ip_addresses | The IP addresses to assign all control plane VMs. |
142+
| control_plane_hostnames | The host names to assign to all control plane VMs. |
143+
| compute_memory | Amount of memory to assign each compute VM. This value is in MB. |
144+
| compute_num_cpus | Number of CPUs to assign to each compute VM. |
145+
| compute_count | Number of compute VMs. |
146+
| compute_ip_addresses | The IP addresses to assign all compute VMs. |
147+
| compute_hostnames | The host names to assign to all compute VMs. |
148+
149+
150+
## Build a Cluster
151+
152+
### Manual Method
153+
154+
1. Create an install-config.yaml. The machine CIDR for the dev cluster is 139.178.89.192/26.
155+
156+
```
157+
apiVersion: v1
158+
baseDomain: devcluster.openshift.com
159+
metadata:
160+
name: mstaeble
161+
networking:
162+
machineNetwork:
163+
- cidr: "139.178.89.192/26"
164+
platform:
165+
vsphere:
166+
vCenter: vcsa.vmware.devcluster.openshift.com
167+
username: YOUR_VSPHERE_USER
168+
password: YOUR_VSPHERE_PASSWORD
169+
datacenter: dc1
170+
defaultDatastore: nvme-ds1
171+
pullSecret: YOUR_PULL_SECRET
172+
sshKey: YOUR_SSH_KEY
173+
```
174+
175+
2. Run `openshift-install create manifest`.
176+
177+
3. Update any configurations you need before generating ignition configs. It is recommended to remove the master machine CR and the worker machineset CR. This can be accomplished by running:
178+
179+
```
180+
rm -f openshift/99_openshift-cluster-api_master-machines-*.yaml openshift/99_openshift-cluster-api_worker-machineset-*.yaml
181+
```
182+
183+
3. Run `openshift-install create ignition-configs`.
184+
185+
4. Create and configure a variables.ps1 file.
186+
There is an example variables.ps1 file in this directory named variables.ps1.example. The example file contains all properties that are available to be configured. At a minimum, you need to set values for the following variables.
187+
* clustername
188+
* basedomain
189+
* username
190+
* password
191+
* vcenter
192+
* vcentercredpath
193+
194+
The bootstrap ignition config must be placed in a location that will be accessible by the bootstrap machine. For example, you could store the bootstrap ignition config in a gist.
195+
196+
Even if declaring static IPs a DHCP server is still required early in the boot process to download the ignition files.
197+
198+
5. Run `pwsh -f upi.ps1`.
199+
200+
The script will now attempt to create all VMs based on the config provided. The script will start the VMs and will say `Install Comlete` when all VMs are cloned and started.
201+
202+
6. Run `openshift-install wait-for install-complete`. Wait for the cluster install to finish.
203+
204+
7. Enjoy your new OpenShift cluster.
205+
206+
8. Run `pwsh -f upi-destroy.ps1` to clean up all folder, VMs, and tags.
207+
208+
# Terraform
209+
This section will walk you through generating a cluster using Terraform.
210+
211+
<a id="terraform-pre-requisites"></a>
212+
## Pre-Requisites
2213

3214
* terraform
4215
* jq
5216

6-
# Build a Cluster
217+
## Build a Cluster
7218

8219
1. Create an install-config.yaml.
9220
The machine CIDR for the dev cluster is 139.178.89.192/26.

‎upi/vsphere/lb/haproxy.erb.tmpl‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
defaults
2+
maxconn 20000
3+
mode tcp
4+
log /var/run/haproxy/haproxy-log.sock local0
5+
option dontlognull
6+
retries 3
7+
timeout http-request 10s
8+
timeout queue 1m
9+
timeout connect 10s
10+
timeout client 86400s
11+
timeout server 86400s
12+
timeout tunnel 86400s
13+
14+
frontend api-server
15+
bind <%= $lb_ip_address %>:6443
16+
default_backend api-server
17+
18+
frontend machine-config-server
19+
bind <%= $lb_ip_address %>:22623
20+
default_backend machine-config-server
21+
22+
frontend router-http
23+
bind <%= $lb_ip_address %>:80
24+
default_backend router-http
25+
26+
frontend router-https
27+
bind <%= $lb_ip_address %>:443
28+
default_backend router-https
29+
30+
backend api-server
31+
option httpchk GET /readyz HTTP/1.0
32+
option log-health-checks
33+
balance roundrobin
34+
<% foreach ($addr in $api) { -%>
35+
server <%= $addr %> <%= $addr %>:6443 weight 1 verify none check check-ssl inter 1s fall 2 rise 3
36+
<% } -%>
37+
38+
backend machine-config-server
39+
balance roundrobin
40+
<% foreach ($addr in $api) { -%>
41+
server <%= $addr %> <%= $addr %>:22623 check
42+
<% } -%>
43+
44+
backend router-http
45+
balance source
46+
mode tcp
47+
<% foreach ($addr in $ingress) { -%>
48+
server <%= $addr %> <%= $addr %>:80 check
49+
<% } -%>
50+
51+
backend router-https
52+
balance source
53+
mode tcp
54+
<% foreach ($addr in $ingress) { -%>
55+
server <%= $addr %> <%= $addr %>:443 check
56+
<% } -%>
57+

‎upi/vsphere/powercli/upi-destroy.ps1‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
#!/usr/bin/pwsh
2+
3+
. .\variables.ps1
4+
5+
$ErrorActionPreference = "Stop"
6+
7+
# since we do not have ca for vsphere certs, we'll just set insecure
8+
Set-PowerCLIConfiguration -InvalidCertificateAction:Ignore -Confirm:$false | Out-Null
9+
$Env:GOVC_INSECURE = 1
10+
11+
# Connect to vCenter
12+
Connect-VIServer -Server $vcenter -Credential (Import-Clixml $vcentercredpath)
13+
14+
# Convert the installer metadata to a powershell object
15+
$metadata = Get-Content -Path ./metadata.json | ConvertFrom-Json
16+
17+
# Get tag for all resources we created
18+
$tagCategory = Get-TagCategory -Name "openshift-$($metadata.infraID)"
19+
$tag = Get-Tag -Category $tagCategory -Name "$($metadata.infraID)"
20+
21+
# Clean up all VMs
22+
$vms = Get-VM -Tag $tag
23+
foreach ($vm in $vms) {
24+
if ($vm.PowerState -eq "PoweredOn") {
25+
Write-Output "Stopping VM $vm"
26+
Stop-VM -VM $vm -confirm:$false > $null
27+
}
28+
Write-Output "Removing VM $vm"
29+
Remove-VM -VM $vm -DeletePermanently -confirm:$false
30+
}
31+
32+
# Clean up all templates
33+
$templates = Get-TagAssignment -Tag $tag -Entity (Get-Template)
34+
foreach ($template in $templates) {
35+
Write-Output "Removing template $($template.Entity)"
36+
Remove-Template -Template $($template.Entity) -DeletePermanently -confirm:$false
37+
}
38+
39+
# Clean up storage policy
40+
$storagePolicies = Get-SpbmStoragePolicy -Tag $tag
41+
42+
foreach ($policy in $storagePolicies) {
43+
44+
$clusterInventory = @()
45+
$splitResults = @($policy.Name -split "openshift-storage-policy-")
46+
47+
if ($splitResults.Count -eq 2) {
48+
$clusterId = $splitResults[1]
49+
if ($clusterId -ne "") {
50+
Write-Host $clusterId
51+
$clusterInventory = @(Get-Inventory -Name "$($clusterId)*" -ErrorAction Continue)
52+
53+
if ($clusterInventory.Count -eq 0) {
54+
Write-Host "Removing policy: $($policy.Name)"
55+
$policy | Remove-SpbmStoragePolicy -Confirm:$false
56+
}
57+
else {
58+
Write-Host "not deleting: $($clusterInventory)"
59+
}
60+
}
61+
}
62+
}
63+
64+
# Clean up all resource pools
65+
$rps = Get-TagAssignment -Tag $tag -Entity (Get-ResourcePool)
66+
foreach ($rp in $rps) {
67+
Write-Output "Removing resource pool $($rp.Entity)"
68+
Remove-ResourcePool -ResourcePool $($rp.Entity) -confirm:$false
69+
}
70+
71+
# Clean up all folders
72+
$folders = Get-TagAssignment -Tag $tag -Entity (Get-Folder)
73+
foreach ($folder in $folders) {
74+
Write-Output "Removing folder $($folder.Entity)"
75+
Remove-Folder -Folder $($folder.Entity) -DeletePermanently -confirm:$false
76+
}
77+
78+
# Clean up tags
79+
Remove-Tag -Tag $tag -confirm:$false
80+
Remove-TagCategory -Category $tagCategory -confirm:$false

0 commit comments

Comments
 (0)