Skip to content

Commit 8dfd24f

Browse files
agadekaragadekar
authored
Adds support for arc-enabled virtual machines in HCI setup (#206)
* Adds support for arc-enabled virtual machines in HCI setup * uses LogOutput, slight modifications in identification of existing Arc resources
1 parent 863ed1e commit 8dfd24f

File tree

3 files changed

+1218
-260
lines changed

3 files changed

+1218
-260
lines changed

‎Frameworks/VMFleet/ArcVM-Setup.md‎

Lines changed: 329 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,329 @@
1+
# From Setup to Start-FleetSweep for Arc Enabled Virtual Machines in HCI clusters
2+
3+
This is the traditional path of setting up VMFleet to deploy Arc enabled VMs on HCI clusters and running it using your desired DiskSpd parameters/flags.
4+
5+
6+
7+
## Prerequisites
8+
9+
Before we begin setting up VMFleet, there are a few prerequisites that you should have ready.
10+
11+
Ensure an HCI cluster is setup.
12+
13+
Ensure that you have 1 CSV per node
14+
15+
Within Storage Spaces Direct, CPU usage is based on the host. Therefore, it is recommended that you split the storage load by creating as many CSVs as there are host nodes. We can go ahead and create a CSV per node in the cluster.
16+
17+
You may run the following:
18+
19+
20+
```
21+
Get-ClusterNode |% {
22+
New-Volume -StoragePoolFriendlyName SU1_Pool* -FriendlyName $_ -FileSystem CSVFS_ReFS -Size <DESIRED SIZE>
23+
}
24+
25+
```
26+
27+
Ensure that you create a “collect” volume.
28+
29+
You may run the following:
30+
31+
```
32+
33+
New-Volume -StoragePoolFriendlyName SU1_Pool* -FriendlyName collect -FileSystem CSVFS_ReFS -Size 200GB
34+
35+
```
36+
37+
If you have ran VMFleet in the past, please ensure that prior VMFleet directories are completely removed from existing volumes.
38+
39+
Retrieve or install a Server Core VHDX file. If you do not have one handy, we can create a new one by following the below instructions.
40+
41+
Download WS2019 Server Core ISO from the public website.
42+
43+
Open Hyper-V Manager.
44+
45+
Click “New”, then “Virtual Machine”.
46+
47+
Navigate through the prompts and pick a location to store your VM.
48+
49+
Once your VM is created, boot up the VM and follow the instructions. This is where you will decide your VM password, or what we will later call, “adminpass”.
50+
51+
This is important as we will use this later, so make sure you write this down.
52+
53+
Log out of the VM, and navigate to where you stored your VM. You should find a “Virtual Hard Disks” folder. Inside, you should find your new Server Core VHDX file.
54+
55+
Rename it to “Base1.vhdx”.
56+
57+
Copy or move the file to the cluster environment that you want to run VMFleet in.
58+
59+
Done!
60+
61+
### Deployment
62+
63+
1. First, we need to install the new PowerShell Module from the PowerShell Gallery and then load it into our terminal. We also need to disable cache as it is not supported currently for ArcVMs. Run the following:
64+
65+
```
66+
Install-Module -Name “VMFleet”
67+
Import-Module VMFleet
68+
Set-ClusterStorageSpacesDirect -CacheState Disabled;
69+
70+
```
71+
72+
73+
2. Sanity Check:
74+
Run "Get-Module VMFleet" to confirm the module exists.
75+
76+
Run "Get-Command -Module VMFleet" to obtain a list of functions included in the module.
77+
78+
We will now set up the directory structure within the “Collect” CSV created earlier. Run "Install-Fleet"
79+
80+
This creates the necessary VMFleet directories which include:
81+
82+
* Collect/control
83+
84+
Contains arc.json which stores Arc configuration and the scripts that the Virtual Machines continuously monitor.
85+
86+
* Control.ps1: the control script the VMs use to implement the control loop (what used to be called “master.ps1”).
87+
88+
* Run.ps1: The VMs continuously look for the most recent version of run.ps1 and runs the newly updated script (parameters).
89+
90+
* Collect/flag
91+
92+
Location where the control script drops the “go”, “pause”, and “done” flag files. Users should not need to look at these files.
93+
94+
* Collect/result
95+
96+
Location of the output files from the VMFleet test run.
97+
98+
* Collect/tools
99+
100+
DiskSpd will be preinstalled in this folder.
101+
102+
3. You need to create a new or use existing Resource group under the same subscription as the Resource bridge VM is under. Set-ArcConfig will take care of creating one if not already present.
103+
104+
4. Setup configuration required for creating Arc enabled Virtual Machines.
105+
106+
```
107+
Set-ArcConfig -ResourceGroup [ENTER_RESOURCE_GROUP] -AzureRegistrationUser [ENTER_AZURE_REGUSER] -AzureRegistrationPassword [ENTER_AZURE_REGPASS]-StoragePathCsv [ENTER_CSV_PATH] -Enabled $true -StoragePathName [ENTER_StoragePath_Name] -ImageName [ENTER_Image_Name] -ResetSalt
108+
```
109+
110+
-"ResourceGroup" is the resource group where ARC VMs will be deployed.
111+
112+
-"AzureRegistrationUser" and "AzureRegistrationPassword" are the azure account credentials.
113+
114+
-"StoragePathCsv" (optional) is the csv path where storage path resource will be created. If not provided, one of the existing csvs which were created earlier will be used by default.
115+
116+
-"StoragePathName" (optional) is the storage path name which will be used to create gallery image. If not provided, default name will be used.
117+
118+
-"ImageName" (optional) is the gallery image name which will be used to create Arc-enabled virtual machines. If not provided, default name will be used. Currently, only windows gallery image is the supported image type for Arc-Enabled VMs.
119+
120+
-"Enabled" (optional) is set to $false by default. Set to $true if Arc-enabled virtual machines are to be created.
121+
122+
-"ResetSalt" (optional) is a flag used to reset salt. Salt is a random 4 character (alphanumeric) used in Arc resource names for arc enabled virtual machines (for eg: vm-group-node-salt-001). In case, user wants to regenerate the salt, they can run "Set-ArcConfig -ResetSalt" which will override existing salt with a new one in the arc.json and this will be used to create new VMs. This is useful in case of multiple clusters under same subscription using same resource group on a virtual setup. A 4 character (alphanumeric) Salt will be generated by default when user runs Set-ArcConfig the first time and stored in arc.json along with other arc configs. Within Set-ArcConfig, a quick test is done to check if atleast one vm exists with same salt in the resource group. If it does, it is regenerated.
123+
124+
5. By default, VMs with 2GB static memory and 1 processor count will be created.
125+
126+
Note: Please move the VHDX file into the collect folder. CSV Cache is also turned off by default.
127+
128+
We will now create our “fleet” of VMs by running:
129+
130+
```
131+
New-Fleet -basevhd <PATH TO VHDX> -vms [ENTER_NUM_VMS] -adminpass [ENTER_ADMINPASS] -connectuser [ENTER_NODE_USER] -connectpass [ENTER_NODE_PASS]
132+
```
133+
134+
-"adminpass" is the administrator password for the Server Core Image. This is the password you set on your Virtual Machine earlier.
135+
136+
-"connectuser" is a domain account with access to the cluster.
137+
138+
-"connectpass " is the password for the above domain account.
139+
140+
-"vms" is the number of VM's to create per node.
141+
142+
6. If "vms" parameter is not provided, the default is a 1:1 subscription ratio where the Number of VMs = Number of physical cores.
143+
144+
[Optional] You can consider modifying the VM hardware configuration. Run
145+
146+
```
147+
Set-Fleet -ProcessorCount 1 -MemoryStartupBytes 2048mb -MemoryMaximumBytes 2048mb -MemoryMinimumBytes 2048mb
148+
```
149+
150+
Note:
151+
152+
If you specify “MemoryMaximumBytes”, you must specify “MemoryMinimumBytes”, which implies that your VMs will have dynamic memory.
153+
154+
If you omit “MemoryMaximumBytes” or “MemoryMinimumBytes”, it implies that your VMs will have static memory.
155+
156+
If MemoryStartupBytes = MemoryMinimumBytes = MemoryMaximumBytes, that also denotes static memory.
157+
158+
“MemoryStartupBytes” is a mandatory parameter.
159+
160+
161+
### Start Running VMFleet!
162+
163+
7. Open 2 PowerShell terminals. In the first one, run Watch-Cluster and in the second one, run Start-Fleet. This second function will turn on all the VMs in a “paused” state.
164+
165+
8. At this point you can run Start-FleetSweep [ENTER_PARAMETERS] or take this time to explore and run any of the other functions!
166+
167+
Here is a sample sweep command to help you get started: "Start-FleetSweep -b 4 -t 8 -o 8 -w 0 -d 300 -p r"
168+
169+
9. Done!
170+
171+
### Aftermath
172+
173+
Once you are done running VMFleet you can run Stop-Fleet to shut down all the virtual machines or run Remove-Fleet to completely delete all the virtual machines on your environment.
174+
175+
# From Setup to Measure-FleetCoreWorkload
176+
177+
This is a new workflow for setting up VMFleet and the predefined profile workloads (General, Peak, VDI, SQL).
178+
179+
180+
181+
## Prerequisites
182+
183+
Before we begin setting up VMFleet, there are a few prerequisites that you should have ready.
184+
185+
Ensure an HCI cluster is setup.
186+
187+
Ensure that you have 1 CSV per node
188+
189+
Within Storage Spaces Direct, CPU usage is based on the host. Therefore, it is recommended that you split the storage load by creating as many CSVs as there are host nodes. We can go ahead and create a CSV per node in the cluster.
190+
191+
In order to be precise about the CSV size, please use our new VMFleet command: "Get-FleetVolumeEstimate"
192+
193+
This will output a prescribed CSV size based on different resiliency types. We recommend you select a 2-way mirrored value or 3-way mirrored value depending on your node count.
194+
195+
1. You may run the following: (use the CSV size from Get-FleetVolumeEstimate)
196+
197+
```
198+
Get-ClusterNode |% {
199+
New-Volume -StoragePoolFriendlyName SU1_Pool* -FriendlyName $_ -FileSystem CSVFS_ReFS -Size <DESIRED SIZE>
200+
}
201+
```
202+
203+
Ensure that you create a “collect” volume.
204+
205+
2. You may run the following:
206+
207+
```
208+
New-Volume -StoragePoolFriendlyName SU1_Pool* -FriendlyName collect -FileSystem CSVFS_ReFS -Size 200GB
209+
```
210+
211+
3. If you have ran VMFleet in the past, please ensure that prior VMFleet directories are completely removed from existing volumes.
212+
213+
Retrieve or install a Server Core VHDX file. If you do not have one handy, we can create a new one by following the below instructions.
214+
215+
Download WS2019 Server Core ISO from the public website.
216+
217+
Open Hyper-V Manager.
218+
219+
Click “New”, then “Virtual Machine”.
220+
221+
Navigate through the prompts and pick a location to store your VM.
222+
223+
Once your VM is created, boot up the VM and follow the instructions. This is where you will decide your VM password, or what we will later call, “adminpass”.
224+
225+
This is important as we will use this later, so make sure you write this down.
226+
227+
Log out of the VM, and navigate to where you stored your VM. You should find a “Virtual Hard Disks” folder. Inside, you should find your new Server Core VHDX file.
228+
Rename it to “Base1.vhdx”.
229+
230+
Copy or move the file to the cluster environment that you want to run VMFleet in.
231+
232+
4. Done!
233+
234+
## Deployment
235+
5. Let’s begin deploying VMFleet. First, we need to install the new PowerShell Module from the PowerShell Gallery and then load it into the terminal. Run the following:
236+
237+
```
238+
Install-Module -Name “VMFleet”
239+
Import-Module VMFleet
240+
Set-ClusterStorageSpacesDirect -CacheState Disabled;
241+
```
242+
243+
244+
245+
## Sanity Check:
246+
247+
6. Run "Get-Module VMFleet" to confirm the module exists.
248+
249+
7. Run "Get-Command -Module VMFleet" to obtain a list of commands included in the module.
250+
251+
8. We will now set up the directory structure within the “Collect” CSV that we created earlier. Run "Install-Fleet"
252+
253+
9. You need to create a new or use existing Resource group under the same subscription as the Resource bridge VM is under. Set-ArcConfig will take care of creating one if not already present.
254+
255+
10. Setup configuration required for creating Arc enabled Virtual Machines.
256+
257+
```
258+
Set-ArcConfig -ResourceGroup [ENTER_RESOURCE_GROUP] -AzureRegistrationUser [ENTER_AZURE_REGUSER] -AzureRegistrationPassword [ENTER_AZURE_REGPASS]-StoragePathCsv [ENTER_CSV_PATH] -Enabled $true -StoragePathName [ENTER_StoragePath_Name] -ImageName [ENTER_Image_Name] -ResetSalt
259+
```
260+
261+
-"ResourceGroup" is the resource group where ARC VMs will be deployed.
262+
263+
-"AzureRegistrationUser" and "AzureRegistrationPassword" are the azure account credentials.
264+
265+
-"StoragePathCsv" (optional) is the csv path where storage path resource will be created. If not provided, one of the existing csvs which were created earlier will be used by default.
266+
267+
-"StoragePathName" (optional) is the storage path name which will be used to create gallery image. If not provided, default name will be used.
268+
269+
-"ImageName" (optional) is the gallery image name which will be used to create Arc-enabled virtual machines. If not provided, default name will be used.
270+
271+
-"Enabled" (optional) is set to $false by default. Set to $true if Arc-enabled virtual machines are to be created.
272+
273+
-"ResetSalt" (optional) is a flag used to reset salt. Salt is a random 4 character (alphanumeric) used in Arc resource names for arc enabled virtual machines (for eg: vm-group-node-salt-001). In case, user wants to regenerate the salt, they can run "Set-ArcConfig -ResetSalt" which will override existing salt with a new one in the arc.json and this will be used to create new VMs. This is useful in case of multiple clusters under same subscription using same resource group on a virtual setup. A 4 character (alphanumeric) Salt will be generated by default when user runs Set-ArcConfig the first time and stored in arc.json along with other arc configs. Within Set-ArcConfig, a quick test is done to check if atleast one vm exists with same salt in the resource group. If it does, it is regenerated.
274+
275+
11. By default, VMs with 2GB static memory and 1 processor count will be created.
276+
277+
Note: Please move the VHDX file into the collect folder. CSV Cache is also turned off by default.
278+
279+
We will now create our “fleet” of VMs by running:
280+
281+
```
282+
New-Fleet -basevhd <PATH TO VHDX> -vms [ENTER_NUM_VMS] -adminpass [ENTER_ADMINPASS] -connectuser [ENTER_NODE_USER] -connectpass [ENTER_NODE_PASS]
283+
```
284+
285+
-"adminpass" is the administrator password for the Server Core Image. This is the password you set on your Virtual Machine earlier.
286+
287+
-"connectuser" is a domain account with access to the cluster.
288+
289+
-"connectpass " is the password for the above domain account.
290+
291+
-"vms" is the number of VM's to create per node.
292+
293+
294+
12. Measure-FleetCoreWorkload also collects diagnostic data (Get-SDDCDiagnosticInfo). Therefore, before running the command, we must also install the NuGet Package if you have not previously done so. In doing so, we also need to temporairly set the PSGallery as a trusted repository source (Note: This will temporarily relax the security boundary).
295+
296+
297+
```
298+
$repo = Get-PSRepository -Name PSGallery
299+
if ($null -eq $repo) { Write-Host "The PSGallery is not configured on this system, please address this before continuing" }
300+
else {
301+
if ($repo.InstallationPolicy -ne 'Trusted') {
302+
Write-Host "Setting the PSGallery repository to Trusted, original InstallationPolicy: $($repo.InstallationPolicy)"
303+
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
304+
}
305+
306+
### Installing the pre-requisite modules
307+
308+
Install-PackageProvider NuGet -Force
309+
Install-Module -Name PrivateCloud.DiagnosticInfo -Force
310+
Install-Module -Name MSFT.Network.Diag -Force
311+
312+
if ($repo.InstallationPolicy -ne 'Trusted') {
313+
Write-Host "Resetting the PSGallery repository to $($repo.InstallationPolicy)"
314+
Set-PSRepository -Name PSGallery -InstallationPolicy $repo.InstallationPolicy
315+
}
316+
}
317+
```
318+
319+
## Run Measure-FleetCoreWorkload!
320+
13. We can now run Measure-FleetCoreWorkload. Running the command below will automatically run all 4 workloads (General, Peak, VDI, SQL) and place the individual outputs in the result directory. IMPORTANT: If you plan on running another test, please clear the result directory.
321+
322+
323+
```
324+
Measure-FleetCoreWorkload
325+
```
326+
327+
14. Congratulations! You’re done! All you need to do is wait for the test to complete.
328+
329+
Note: If you ever run into an error and need to rerun Measure-FleetCoreWorkload, don’t be afraid to do so! It is smart enough to pick up from where it last stopped and continue the test without starting from scratch.

‎Frameworks/VMFleet/VMFleet.psd1‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,11 +120,13 @@ FunctionsToExport = @(
120120
'New-Fleet',
121121
'Remove-Fleet',
122122
'Repair-Fleet',
123+
'Set-ArcConfig',
123124
'Set-Fleet',
124125
'Set-FleetPause',
125126
'Set-FleetPowerScheme',
126127
'Set-FleetProfile',
127128
'Set-FleetQoS',
129+
'Set-FleetRunProfileScript',
128130
'Show-Fleet',
129131
'Show-FleetCpuSweep',
130132
'Show-FleetPause',
@@ -138,7 +140,8 @@ FunctionsToExport = @(
138140
'Test-FleetResultRun',
139141
'Use-FleetPolynomialFit',
140142
'Watch-FleetCluster',
141-
'Watch-FleetCPU'
143+
'Watch-FleetCPU',
144+
'Initialize-ArcVMs'
142145
)
143146

144147
# Cmdlets to export from this module, for best performance, do not use wildcards and do not delete the entry, use an empty array if there are no cmdlets to export.

0 commit comments

Comments
 (0)