Skip to content

Commit 3791bdd

Browse files
zulinx86zulinx86
committed
docs(tool): Add public documentation for cpu-template-helper
Add public documentation for the `cpu-template-helper` tool that explains its usage and step-by-step instructions for a sample typical scenario. Signed-off-by: Takahiro Itazuri <[email protected]>
1 parent 62441ce commit 3791bdd

File tree

2 files changed

+243
-2
lines changed

2 files changed

+243
-2
lines changed

‎docs/cpu_templates/cpu-template-helper.md‎

Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
# CPU template helper tool
2+
3+
The `cpu-template-helper` tool is a program designed to assist users with
4+
creating and managing their custom CPU templates.
5+
6+
## Usage
7+
8+
The `cpu-template-helper` tool has two sets of commands: template-related
9+
commands and fingerprint-related commands.
10+
11+
### Template-related commands
12+
13+
#### Dump command
14+
15+
This command dumps guest CPU configuration in the custom CPU template JSON
16+
format.
17+
18+
```
19+
cpu-template-helper template dump --config <firecracker-config> --output <cpu-config>
20+
```
21+
22+
Users can utilize this as an entry point of a custom CPU template creation
23+
to comprehend what CPU configuration are exposed to guests.
24+
25+
The guest CPU configuration consists of the following entities:
26+
27+
- x86_64
28+
- CPUID
29+
- MSRs (Model Specific Registers)
30+
- aarch64
31+
- ARM registers
32+
33+
It retrieves the above entities exposed to a guest by applying the same preboot
34+
process as Firecacker and capturing them in the state just before booting a
35+
guest. More details about the preboot process can be found
36+
[here](boot-protocol.md) and [here](cpuid-normalization.md).
37+
38+
> **Note**
39+
Some MSRs and ARM registers are not included in the output, since they are not
40+
reasonable to modify with CPU templates. The full list of them can be found in
41+
[Appendix](#appendix).
42+
43+
> **Note**
44+
Since the output depends on underlying hardware and software stack (BIOS, CPU,
45+
kernel, Firecracker), it is required to dump guest CPU configuration on each
46+
combination when creating a custom CPU template targetting them all.
47+
48+
#### Strip command
49+
50+
This command strips identical entries from multiple guest CPU configuration
51+
files generated with the dump command.
52+
53+
```
54+
cpu-template-helper template strip \
55+
--paths <cpu-config-1> <cpu-config-2> [..<cpu-config-N>] \
56+
--suffix <suffix>
57+
```
58+
59+
One practical use case of the CPU template feature is to provide a consistent
60+
CPU feature set to guests running on multiple CPU models. When creating a custom
61+
CPU template for this purpose, it is efficient to focus on the differences
62+
in guest CPU configurations across those CPU models. Given that a dumped guest
63+
CPU configuration typically amounts to approximately 1,000 lines, this command
64+
considerably narrows down the scope to consider.
65+
66+
#### Verify command
67+
68+
This command verifies that the given custom CPU template is applied correctly.
69+
70+
```
71+
cpu-template-helper template verify \
72+
--config <firecracker-config>
73+
```
74+
75+
Firecracker modifies the guest CPU configuration after the CPU template is
76+
applied. Occasionally, due to hardware and/or software limitations, KVM might
77+
not set the given configuration. Since Firecracker does not check them at
78+
runtime, it is required to ensure that these situations don't happen with their
79+
custom CPU templates before deploying it.
80+
81+
The command uses the same configuration file as Firecracker and the path to the
82+
custom CPU template file should be specified in the "cpu-config" field.
83+
84+
> **Note**
85+
This command does not ensure that the contents of the template are sensible.
86+
Thus, users need to make sure that the template does not have any inconsistent
87+
entries and does not crash guests.
88+
89+
### Fingerprint-related commands
90+
91+
#### Dump command
92+
93+
This command not only dumps the guest CPU configuration, but also host
94+
information that could affect the validity of custom CPU templates.
95+
96+
```
97+
cpu-template-helper fingerprint dump \
98+
--config <firecracker-config> \
99+
--output <output-path>
100+
```
101+
102+
Keeping the underlying hardware and software stack updated is essential for
103+
maintaining security and leveraging new technologies. On the other hand, since
104+
the guest CPU configuration can vary depending on the infrastructure, updating
105+
it could lead to a situation where a custom CPU template loses its validity.
106+
In addition, even if values of the guest CPU configuration don't change, its
107+
internal behavior or semantics could still change. For instance, a kernel
108+
version update may introduce changes to KVM emulation and a microcode update may
109+
alter the behavior of CPU instructions.
110+
111+
To ensure awareness of these changes, it is strongly recommended to store the
112+
fingerprint file at the time of creating a custom CPU template and to
113+
continuously compare it with the current one.
114+
115+
#### Compare command
116+
117+
This command compares two fingerprint files: one was taken at the time of custom
118+
CPU template creation and the other is taken currently.
119+
120+
```
121+
cpu-template-helper fingerprint compare \
122+
--prev <prev-fingerprint> \
123+
--curr <curr-fingerprint> \
124+
--filters <field-1> [..<field-N>]
125+
```
126+
127+
By continously comparing fingerprint files, users can ensure they are aware of
128+
any changes that could require revising the custom CPU template. However, it is
129+
worth noting that not all of these changes necessarily require a revision, and
130+
some changes could be inconsequential to the custom CPU template depending on
131+
its use case. To provide users with flexibility in comparing fingerprint files
132+
based on situations or use cases, the `--filters` option allows users to select
133+
which fields to compare.
134+
135+
As examples of when to compare fingerprint files:
136+
137+
- When bumping the Firecracker version up
138+
- When bumping the kernel version up
139+
- When applying a microcode update (or launching a new host (e.g. AWS EC2 metal
140+
instance))
141+
142+
## Sample scenario
143+
144+
This section gives steps of creating and managing a custom CPU template in a
145+
sample scenario where the template is designed to provide a consistent set of
146+
CPU features to a heterogeneous fleet consisting of multiple CPU models.
147+
148+
### Custom CPU template creation
149+
150+
1. Run the `cpu-template-helper template dump` command on each CPU model to
151+
retrieve guest CPU configuration.
152+
1. Run the `cpu-template-helper template strip` command to remove identical
153+
entries across the dumped guest CPU configuration files.
154+
1. Examine the differences of guest CPU configuration in details, determine
155+
which CPU features should be presented to guests and draft a custom CPU
156+
template.
157+
1. Run the `cpu-template-helper template verify` command to check the created
158+
custom CPU template is applied correctly.
159+
1. Conduct thorough testing of the template as needed to ensure that it does
160+
not contain any inconsistent entries and does not lead to guest crashes.
161+
162+
### Custom CPU template management
163+
164+
1. Run the `cpu-template-helper fingerprint dump` command on each CPU model
165+
at the same time when creating a custom CPU template.
166+
1. Store the dumped fingerprint files together with the custom CPU template.
167+
1. Run the `cpu-template-helper fingerprint dump` command to ensure the
168+
template's validity whenever you expect changes to the underlying hardware
169+
and software stack.
170+
1. Run the `cpu-template-helper fingerprint compare` command to identify
171+
changes of the underlying environment introduced after creating the template.
172+
1. (if changes are detected) Review the identified changes, make necessary
173+
revisions to the CPU template, and replace the fingerprint file with the new
174+
one.
175+
176+
> **Note**
177+
It is recommended to review the update process of the underlying stack on
178+
your infrastructure. This can help identify points that may require the above
179+
validation check.
180+
181+
## Appendix
182+
183+
### MSRs excluded from guest CPU configuration dump
184+
185+
| Register name | Index |
186+
|-----------------------------------------|-------------------------|
187+
| MSR_IA32_TSC | 0x00000010 |
188+
| MSR_ARCH_PERFMON_PERFCTRn | 0x000000c1 - 0x000000d2 |
189+
| MSR_ARCH_PERFMON_EVENTSELn | 0x00000186 - 0x00000197 |
190+
| MSR_ARCH_PERFMON_FIXED_CTRn | 0x00000309 - 0x0000030b |
191+
| MSR_CORE_PERF_FIXED_CTR_CTRL | 0x0000038d |
192+
| MSR_CORE_PERF_GLOBAL_STATUS | 0x0000038e |
193+
| MSR_CORE_PERF_GLOBAL_CTRL | 0x0000038f |
194+
| MSR_CORE_PERF_GLOBAL_OVF_CTRL | 0x00000390 |
195+
| MSR_K7_EVNTSELn | 0xc0010000 - 0xc0010003 |
196+
| MSR_K7_PERFCTR0 | 0xc0010004 - 0xc0010007 |
197+
| MSR_F15H_PERF_CTLn + MSR_F15H_PERF_CTRn | 0xc0010200 - 0xc001020c |
198+
| MSR_IA32_VMX_BASIC | 0x00000480 |
199+
| MSR_IA32_VMX_PINBASED_CTLS | 0x00000481 |
200+
| MSR_IA32_VMX_PROCBASED_CTLS | 0x00000482 |
201+
| MSR_IA32_VMX_EXIT_CTLS | 0x00000483 |
202+
| MSR_IA32_VMX_ENTRY_CTLS | 0x00000484 |
203+
| MSR_IA32_VMX_MISC | 0x00000485 |
204+
| MSR_IA32_VMX_CR0_FIXEDn | 0x00000486 - 0x00000487 |
205+
| MSR_IA32_VMX_CR4_FIXEDn | 0x00000488 - 0x00000489 |
206+
| MSR_IA32_VMX_VMCS_ENUM | 0x0000048a |
207+
| MSR_IA32_VMX_PROCBASED_CTLS2 | 0x0000048b |
208+
| MSR_IA32_VMX_EPT_VPID_CAP | 0x0000048c |
209+
| MSR_IA32_VMX_TRUE_PINBASED_CTLS | 0x0000048d |
210+
| MSR_IA32_VMX_TRUE_PROCBASED_CTLS | 0x0000048e |
211+
| MSR_IA32_VMX_TRUE_EXIT_CTLS | 0x0000048f |
212+
| MSR_IA32_VMX_TRUE_ENTRY_CTLS | 0x00000490 |
213+
| MSR_IA32_VMX_VMFUNC | 0x00000491 |
214+
| MSR_IA32_MCG_STATUS | 0x0000017a |
215+
| MSR_IA32_MCG_CTL | 0x0000017b |
216+
| MSR_IA32_MCG_EXT_CTL | 0x000004d0 |
217+
| HV_X64_MSR_GUEST_OS_ID | 0x40000000 |
218+
| HV_X64_MSR_HYPERCALL | 0x40000001 |
219+
| HV_X64_MSR_VP_INDEX | 0x40000002 |
220+
| HV_X64_MSR_RESET | 0x40000003 |
221+
| HV_X64_MSR_VP_RUNTIME | 0x40000010 |
222+
| HV_X64_MSR_VP_ASSIST_PAGE | 0x40000073 |
223+
| HV_X64_MSR_SCONTROL | 0x40000080 |
224+
| HV_X64_MSR_STIMER0_CONFIG | 0x400000b0 |
225+
| HV_X64_MSR_CRASH_Pn | 0x40000100 - 0x40000104 |
226+
| HV_X64_MSR_CRASH_CTL | 0x40000105 |
227+
| HV_X64_MSR_REENLIGHTENMENT_CONTROL | 0x40000106 |
228+
| HV_X64_MSR_TSC_EMULATION_CONTROL | 0x40000107 |
229+
| HV_X64_MSR_TSC_EMULATION_STATUS | 0x40000108 |
230+
| HV_X64_MSR_SYNDBG_CONTROL | 0x400000f1 |
231+
| HV_X64_MSR_SYNDBG_STATUS | 0x400000f2 |
232+
| HV_X64_MSR_SYNDBG_SEND_BUFFER | 0x400000f3 |
233+
| HV_X64_MSR_SYNDBG_RECV_BUFFER | 0x400000f4 |
234+
| HV_X64_MSR_SYNDBG_PENDING_BUFFER | 0x400000f5 |
235+
| HV_X64_MSR_SYNDBG_OPTIONS | 0x400000ff |
236+
237+
### ARM registers excluded from guest CPU configuration dump
238+
239+
| Register name | ID |
240+
|----------------------|--------------------|
241+
| KVM_REG_ARM_TIMER_CNT| 0x603000000013df1a |

‎docs/cpu_templates/cpu-templates.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -112,8 +112,8 @@ This is user's responsibility to make sure that their custom CPU template
112112
is applied as expected even if Firecracker does not report an error.
113113

114114
In order to assist with creation and usage of CPU templates, there exists
115-
a CPU template helper tool.
116-
TODO: add a link to the CPU template helper tool when it is available.
115+
a CPU template helper tool. More details can be found
116+
[here](cpu-template-helper.md).
117117

118118
### Configuring custom CPU templates
119119

0 commit comments

Comments
 (0)