Skip to content

Commit e5097e4

Browse files
codyhoagcodyhoag
authored
Merge pull request #28832 from codyhoag/vsphere-winc
OSDOCS-1766 WINC vSphere support
2 parents 4094e0d + ea7a51f commit e5097e4

9 files changed

+293
-1
lines changed

‎_topic_map.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1632,6 +1632,8 @@ Topics:
16321632
File: creating-windows-machineset-aws
16331633
- Name: Creating a Windows MachineSet object on Azure
16341634
File: creating-windows-machineset-azure
1635+
- Name: Creating a Windows MachineSet object on vSphere
1636+
File: creating-windows-machineset-vsphere
16351637
- Name: Scheduling Windows container workloads
16361638
File: scheduling-windows-workloads
16371639
- Name: Windows node upgrades

‎modules/configuring-hybrid-ovnkubernetes.adoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,11 +75,13 @@ spec: <1>
7575
hybridClusterNetwork: <4>
7676
- cidr: 10.132.0.0/14
7777
hostPrefix: 23
78+
hybridOverlayVXLANPort: 9898 <5>
7879
status: {}
7980
----
8081
<1> The parameters for the `spec` parameter are only an example. Specify your configuration for the Cluster Network Operator in the custom resource.
8182
<2> Specify the CIDR configuration used when adding nodes.
8283
<3> Specify `OVNKubernetes` as the Container Network Interface (CNI) cluster network provider.
8384
<4> Specify the CIDR configuration used for nodes on the additional overlay network. The `hybridClusterNetwork` CIDR cannot overlap with the `clusterNetwork` CIDR.
85+
<5> Specify a custom VXLAN port for the additional overlay network. This is required for running Windows nodes in a cluster installed on vSphere; the custom port can be any open port excluding the default `4789` port. For more information on this requirement, see the Microsoft documentation on link:https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/common-problems#pod-to-pod-connectivity-between-hosts-is-broken-on-my-kubernetes-cluster-running-on-vsphere[Pod to pod connectivity between hosts is broken].
8486

8587
. Optional: Back up the `<installation_directory>/manifests/cluster-network-03-config.yml` file. The installation program deletes the `manifests/` directory when creating the cluster.

‎modules/creating-the-vsphere-windows-vm-golden-image.adoc‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * windows_containers/creating_windows_machinesets/creating-windows-machineset-vsphere.adoc
4+
5+
[id="creating-the-vsphere-windows-vm-golden-image_{context}"]
6+
= Creating the vSphere Windows VM golden image
7+
8+
Create a vSphere Windows virtual machine (VM) golden image.
9+
10+
.Prerequisites
11+
12+
* You have installed a cluster on vSphere.
13+
14+
.Procedure
15+
16+
. Create the VM from an updated version of the Windows Server 1909 VM image that includes the following link:https://support.microsoft.com/en-us/help/4565351/windows-10-update-kb4565351[Microsoft patch].
17+
18+
. Create the `C:\Users\Administrator.ssh\authorized_keys` file in the Windows VM containing the public key that corresponds to the private key that resides in the secret you created in the `openshift-windows-machine-config-operator` namespace. The private key of the secret was created when first installing the Windows Machine Config Operator (WMCO) to give {product-title} access to Windows VMs. The `authorized_keys` file is used to configure SSH in the Windows VM.
19+
20+
. Configure SSH on the Windows VM by running the following Powershell script:
21+
+
22+
[source,posh]
23+
----
24+
# Powershell script to configure SSH on vSphere Windows VMs
25+
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
26+
$firewallRuleName = "ContainerLogsPort"
27+
$containerLogsPort = "10250"
28+
New-NetFirewallRule -DisplayName $firewallRuleName -Direction Inbound -Action Allow -Protocol TCP -LocalPort $containerLogsPort -EdgeTraversalPolicy Allow
29+
Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force
30+
Install-Module -Force OpenSSHUtils
31+
Set-Service -Name ssh-agent -StartupType 'Automatic'
32+
Set-Service -Name sshd -StartupType 'Automatic'
33+
Start-Service ssh-agent
34+
Start-Service sshd
35+
$pubKeyConf = (Get-Content -path C:\ProgramData\ssh\sshd_config) -replace '#PubkeyAuthentication yes','PubkeyAuthentication yes'
36+
$pubKeyConf | Set-Content -Path C:\ProgramData\ssh\sshd_config
37+
$passwordConf = (Get-Content -path C:\ProgramData\ssh\sshd_config) -replace '#PasswordAuthentication yes','PasswordAuthentication yes'
38+
$passwordConf | Set-Content -Path C:\ProgramData\ssh\sshd_config
39+
$authFileConf = (Get-Content -path C:\ProgramData\ssh\sshd_config) -replace 'AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys','#AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys'
40+
$authFileConf | Set-Content -Path C:\ProgramData\ssh\sshd_config
41+
$pubKeyLocationConf = (Get-Content -path C:\ProgramData\ssh\sshd_config) -replace 'Match Group administrators','#Match Group administrators'
42+
$pubKeyLocationConf | Set-Content -Path C:\ProgramData\ssh\sshd_config
43+
Restart-Service sshd
44+
New-item -Path $env:USERPROFILE -Name .ssh -ItemType Directory -force
45+
----
46+
47+
. Install and configure VMware Tools version 11.0.6 or greater on the Windows VM. See the link:https://docs.vmware.com/en/VMware-Tools/index.html[VMware Tools documentation] for more information.
48+
49+
. After installing VMware Tools on the Windows VM, verify the following:
50+
.. The `C:\ProgramData\VMware\VMware Tools\tools.conf` file has the following entry:
51+
+
52+
[source,ini]
53+
----
54+
exclude-nics=
55+
----
56+
+
57+
This entry ensures the following:
58+
+
59+
* The cloned vNIC generated on the Windows VM by the hybrid-overlay is not ignored.
60+
* The VM has an IP address in vCenter.
61+
62+
.. The VMTools Windows service is running.
63+
64+
. Pull all of the required Windows container base images needed for your applications. The images you pull
65+
are dependent on the Windows kernel you are using. See Microsoft's documentation on link:https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/container-base-images[pulling Windows container base images] for more information.
66+
67+
. Run the link:https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/sysprep--generalize--a-windows-installation[Windows Sysprep tool] on the Windows VM:
68+
+
69+
[source,terminal]
70+
----
71+
C:\> sysprep.exe /generalize /oobe /shutdown /unattend:<path_to_unattend.xml>
72+
----
73+
+
74+
An example `unattend.xml` is provided, which maintains all the changes needed for the WMCO. For example, the `unattend.xml` file must ensure the Administrator's home directory stays intact with the SSH public key. You must customize the example to fit your needs.
75+
+
76+
.Example `unattend.xml`
77+
[%collapsible]
78+
====
79+
[source,xml]
80+
----
81+
<?xml version="1.0" encoding="UTF-8"?>
82+
<!--A sample unattend.xml which helps in setting admin password and running scripts on first boot-->
83+
<unattend xmlns="urn:schemas-microsoft-com:unattend">
84+
<settings pass="specialize">
85+
<component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http:// www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-International-Core" processorArchitecture="am d64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
86+
<InputLocale>0409:00000409</InputLocale>
87+
<SystemLocale>en-US</SystemLocale>
88+
<UILanguage>en-US</UILanguage>
89+
<UILanguageFallback>en-US</UILanguageFallback>
90+
<UserLocale>en-US</UserLocale>
91+
</component>
92+
<component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-Security-SPP-UX" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
93+
<SkipAutoActivation>true</SkipAutoActivation>
94+
</component>
95+
<component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-SQMApi" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
96+
<CEIPEnabled>0</CEIPEnabled>
97+
</component>
98+
<component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
99+
<ComputerName>windows-host</ComputerName>
100+
<ProductKey>My_Product_key</ProductKey>
101+
</component>
102+
</settings>
103+
<settings pass="oobeSystem">
104+
<component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
105+
<AutoLogon>
106+
<Password>
107+
<Value>MyPassword</Value>
108+
<PlainText>true</PlainText>
109+
</Password>
110+
<Enabled>true</Enabled>
111+
<Username>Administrator</Username>
112+
</AutoLogon>
113+
<OOBE>
114+
<HideEULAPage>true</HideEULAPage>
115+
<HideLocalAccountScreen>true</HideLocalAccountScreen>
116+
<HideOEMRegistrationScreen>true</HideOEMRegistrationScreen>
117+
<HideOnlineAccountScreens>true</HideOnlineAccountScreens>
118+
<HideWirelessSetupInOOBE>true</HideWirelessSetupInOOBE>
119+
<NetworkLocation>Work</NetworkLocation>
120+
<ProtectYourPC>1</ProtectYourPC>
121+
<SkipMachineOOBE>true</SkipMachineOOBE>
122+
<SkipUserOOBE>true</SkipUserOOBE>
123+
</OOBE>
124+
<RegisteredOrganization>Organization</RegisteredOrganization>
125+
<RegisteredOwner>Owner</RegisteredOwner>
126+
<DisableAutoDaylightTimeSet>false</DisableAutoDaylightTimeSet>
127+
<TimeZone>Eastern Standard Time</TimeZone>
128+
<UserAccounts>
129+
<AdministratorPassword>
130+
<Value>MyPassword</Value>
131+
<PlainText>true</PlainText>
132+
</AdministratorPassword>
133+
<LocalAccounts>
134+
<LocalAccount wcm:action="add">
135+
<Description>Administrator</Description>
136+
<DisplayName>Administrator</DisplayName>
137+
<Group>Administrators</Group>
138+
<Name>Administrator</Name>
139+
</LocalAccount>
140+
</LocalAccounts>
141+
</UserAccounts>
142+
</component>
143+
</settings>
144+
</unattend>
145+
----
146+
====

‎modules/enabling-internal-api-server-vsphere.adoc‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * windows_containers/creating_windows_machinesets/creating-windows-machineset-vsphere.adoc
4+
5+
[id="enabling-internal-api-server-vsphere_{context}"]
6+
= Enabling communication with the internal API server for the WMCO on vSphere
7+
8+
The Windows Machine Config Operator (WMCO) downloads the Ignition config files from the internal API server endpoint. You must enable communication with the internal API server so that your Windows virtual machine (VM) can download the Ignition config files, and the kubelet on the configured VM can communicate with the internal API server.
9+
10+
.Prerequisites
11+
12+
* You have installed a cluster on vSphere.
13+
14+
.Procedure
15+
16+
* Add a new DNS entry for `api-int.<cluster_name>.<base_domain>` that points to the external API server URL `api.<cluster_name>.<base_domain>`. This can be a CNAME or an additional A record.
17+
18+
[NOTE]
19+
====
20+
The external API endpoint was already created as part of the initial cluster installation on vSphere.
21+
====

‎modules/machine-api-overview.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
// * machine_management/creating_machinesets/creating-machineset-vsphere.adoc
88
// * windows_containers/creating_windows_machinesets/creating-windows-machineset-aws.adoc
99
// * windows_containers/creating_windows_machinesets/creating-windows-machineset-azure.adoc
10+
// * windows_containers/creating_windows_machinesets/creating-windows-machineset-vsphere.adoc
1011

1112
[id="machine-api-overview_{context}"]
1213
= Machine API overview

‎modules/machineset-creating.adoc‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,13 +9,17 @@
99
// * post_installation_configuration/cluster-tasks.adoc
1010
// * windows_containers/creating_windows_machinesets/creating-windows-machineset-aws.adoc
1111
// * windows_containers/creating_windows_machinesets/creating-windows-machineset-azure.adoc
12+
// * windows_containers/creating_windows_machinesets/creating-windows-machineset-vsphere.adoc
1213

1314
ifeval::["{context}" == "creating-windows-machineset-aws"]
1415
:win:
1516
endif::[]
1617
ifeval::["{context}" == "creating-windows-machineset-azure"]
1718
:win:
1819
endif::[]
20+
ifeval::["{context}" == "creating-windows-machineset-vsphere"]
21+
:win:
22+
endif::[]
1923

2024
ifeval::["{context}" == "creating-machineset-vsphere"]
2125
:vsphere:
@@ -202,3 +206,6 @@ endif::[]
202206
ifeval::["{context}" == "creating-windows-machineset-azure"]
203207
:!win:
204208
endif::[]
209+
ifeval::["{context}" == "creating-windows-machineset-vsphere"]
210+
:!win:
211+
endif::[]

‎modules/windows-machineset-vsphere.adoc‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * windows_containers/creating_windows_machinesets/creating-windows-machineset-vsphere.adoc
4+
5+
[id="windows-machineset-vsphere_{context}"]
6+
= Sample YAML for a Windows MachineSet object on vSphere
7+
8+
This sample YAML defines a Windows `MachineSet` object running on VMware vSphere that the Windows Machine Config Operator (WMCO) can react upon.
9+
10+
[source,yaml]
11+
----
12+
apiVersion: machine.openshift.io/v1beta1
13+
kind: MachineSet
14+
metadata:
15+
labels:
16+
machine.openshift.io/cluster-api-cluster: <infrastructure_id> <1>
17+
name: <windows_machine_set_name> <2>
18+
namespace: openshift-machine-api
19+
spec:
20+
replicas: 1
21+
selector:
22+
matchLabels:
23+
machine.openshift.io/cluster-api-cluster: <infrastructure_id> <1>
24+
machine.openshift.io/cluster-api-machineset: <windows_machine_set_name> <2>
25+
template:
26+
metadata:
27+
labels:
28+
machine.openshift.io/cluster-api-cluster: <infrastructure_id> <1>
29+
machine.openshift.io/cluster-api-machine-role: worker
30+
machine.openshift.io/cluster-api-machine-type: worker
31+
machine.openshift.io/cluster-api-machineset: <windows_machine_set_name> <2>
32+
machine.openshift.io/os-id: Windows <3>
33+
spec:
34+
metadata:
35+
labels:
36+
node-role.kubernetes.io/worker: "" <4>
37+
providerSpec:
38+
value:
39+
apiVersion: vsphereprovider.openshift.io/v1beta1
40+
credentialsSecret:
41+
name: vsphere-cloud-credentials
42+
diskGiB: 128
43+
kind: VSphereMachineProviderSpec
44+
memoryMiB: 16384
45+
network:
46+
devices:
47+
- networkName: "<vm_network_name>" <5>
48+
numCPUs: 4
49+
numCoresPerSocket: 1
50+
snapshot: ""
51+
template: <windows_vm_template_name> <6>
52+
userDataSecret:
53+
name: windows-user-data <7>
54+
workspace:
55+
datacenter: <vcenter_datacenter_name> <8>
56+
datastore: <vcenter_datastore_name> <9>
57+
folder: <vcenter_vm_folder_path> <10>
58+
resourcePool: <vsphere_resource_pool> <11>
59+
server: <vcenter_server_ip> <12>
60+
----
61+
<1> Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. You can obtain the infrastructure ID by running the following command:
62+
+
63+
[source,terminal]
64+
----
65+
$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster
66+
----
67+
<2> Specify the Windows machine set name. The machine set name cannot be more than 9 characters long, due to the way machine names are generated in vSphere.
68+
<3> Configure the machine set as a Windows machine.
69+
<4> Configure the Windows node as a compute machine.
70+
<5> Specify the vSphere VM network to deploy the machine set to.
71+
<6> Specify the full path of the Windows vSphere VM template to use, such as `/Datacenter/vm/ocp4-llplx/windows-golden-image`. The name must be unique.
72+
<7> The `windows-user-data` is created by the WMCO when the first Windows machine is configured. After that, the `windows-user-data` is available for all subsequent machine sets to consume.
73+
<8> Specify the vCenter Datacenter to deploy the machine set on.
74+
<9> Specify the vCenter Datastore to deploy the machine set on.
75+
<10> Specify the path to the vSphere VM folder in vCenter, such as `/dc1/vm/user-inst-5ddjd`.
76+
<11> Optional: Specify the vSphere resource pool for your Windows VMs.
77+
<12> Specify the vCenter server IP or fully qualified domain name.

‎windows_containers/creating_windows_machinesets/creating-windows-machineset-vsphere.adoc‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
[id="creating-windows-machineset-vsphere"]
2+
= Creating a Windows MachineSet object on vSphere
3+
include::modules/common-attributes.adoc[]
4+
:context: creating-windows-machineset-vsphere
5+
6+
toc::[]
7+
8+
You can create a Windows `MachineSet` object to serve a specific purpose in your {product-title} cluster on VMware vSphere. For example, you might create infrastructure Windows machine sets and related machines so that you can move supporting Windows workloads to the new Windows machines.
9+
10+
[discrete]
11+
== Prerequisites
12+
13+
* You installed the Windows Machine Config Operator (WMCO) using Operator Lifecycle Manager (OLM).
14+
* You are using a supported Windows Server as the operating system image with the Docker-formatted container runtime add-on enabled.
15+
16+
[IMPORTANT]
17+
====
18+
Currently, the Docker-formatted container runtime is used in Windows nodes. Kubernetes is deprecating Docker as a container runtime; you can reference the Kubernetes documentation for more information on link:https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/[Docker deprecation]. Containerd will be the new supported container runtime for Windows nodes in a future release of Kubernetes.
19+
====
20+
21+
include::modules/machine-api-overview.adoc[leveloffset=+1]
22+
23+
[id="preparing-vsphere-for-windows-containers"]
24+
== Preparing your vSphere environment for Windows container workloads
25+
26+
You must prepare your vSphere environment for Windows container workloads by creating the vSphere Windows VM golden image and enabling communication with the internal API server for the WMCO.
27+
28+
include::modules/creating-the-vsphere-windows-vm-golden-image.adoc[leveloffset=+2]
29+
include::modules/enabling-internal-api-server-vsphere.adoc[leveloffset=+2]
30+
31+
include::modules/windows-machineset-vsphere.adoc[leveloffset=+1]
32+
include::modules/machineset-creating.adoc[leveloffset=+1]
33+
34+
== Additional resources
35+
36+
* For more information on managing machine sets, see the _Machine management_ section.

‎windows_containers/understanding-windows-container-workloads.adoc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Windows container workloads are supported for clusters running on the following
1111

1212
* Amazon Web Services (AWS)
1313
* Microsoft Azure
14-
//* VMware vSphere
14+
* VMware vSphere
1515

1616
The following Windows Server operating systems are supported for {product-title} {product-version}:
1717

0 commit comments

Comments
 (0)