Multicore support

[dimakuv]

Work in progress.

Useful resources:

• https://linux-kernel-labs.github.io/refs/heads/master/lectures/smp.html
• http://viewer.media.bitpipe.com/1090331926_980/1182181756_632/a3.pdf
• https://www.sigops.org/s/conferences/sosp/2009/papers/baumann-sosp09.pdf
• https://www.usenix.org/legacy/events/osdi08/tech/full_papers/boyd-wickizer/boyd_wickizer.pdf
• https://kernel.org/doc/ols/2003/ols2003-pages-76-88.pdf
• https://www.kernel.org/doc/ols/2004/ols2004v1-pages-133-148.pdf
• https://www.kernel.org/doc/ols/2007/ols2007v1-pages-153-166.pdf
• http://halobates.de/lk09-scalability.pdf
• http://www.osdever.net/tutorials/view/multiprocessing-support-for-hobby-oses-explained

SMP booting

• VM:

  • Intel SDM, Vol. 3A, Chapter 8.4 ("Multiple-Processor (MP) Initialization")
  • Intel SDM, Vol. 3A, Chapter 10.6 ("Issuing Interprocessor Interrupts")
  • https://stackoverflow.com/questions/56384291/what-happens-to-a-startup-ipi-sent-to-an-active-ap-that-is-not-in-a-wait-for-sip -- pseudo-codes how people do INIT-SIPI-SIPI
  • https://wiki.osdev.org/Symmetric_Multiprocessing -- examples of assembly for BSP and AP
  • https://gitlab.com/qemu-project/qboot/-/blob/master/mptable.c#L43 -- here is how QBoot gets the info about CPUs and APICs on QEMU; we can use the same tricks
  • https://binarydebt.wordpress.com/2018/10/06/how-does-an-x86-processor-boot/
  • https://stackoverflow.com/questions/55316726/how-is-the-bootstrap-processor-bsp-selected-on-intel-ring-and-mesh-architectur -- comments here are very interesting
  • https://stackoverflow.com/questions/980999/what-does-multicore-assembly-language-look-like
  • https://stackoverflow.com/questions/69024372/updating-the-pte-bits-from-multiple-cores-how-do-they-avoid-stepping-on-each-o
  • https://stackoverflow.com/questions/48763883/apic-multi-core-startup-protocol-and-icr-starting-up-address
  • https://stackoverflow.com/questions/40083533/what-is-the-effect-of-startup-ipi-on-application-processor

• TDX:

  • https://github.com/confidential-containers/td-shim/blob/main/doc/tdshim_spec.md#multiple-processor-support
  • https://www.intel.com/content/dam/develop/external/us/en/documents/tdx-virtual-firmware-design-guide-rev-1.pdf (Section 4.3)
  • https://www.intel.com/content/dam/develop/external/us/en/documents/intel-tdx-guest-hypervisor-communication-interface.pdf (Section 4.1)

• In case of VM:

  • If the BSP flag is set in the IA32_APIC_BASE MSR, I am a BSP.
  • If not, I am an AP.
  • Note: each logical CPU has its own local APIC (LAPIC) with a unique APIC ID.
  • Table 8.1 in Intel SDM, Vol. 3A shows pseudo-code for BSP and AP initialization INIT-SIPI-SIPI.
  • Note: According to this, enough to have INIT-SIPI.

• In case of TDX:

  • On each vCPU, perform TDCALL[TDG.VP.INFO] and learn my VCPU_INDEX.
  • If VCPU_INDEX == 0, I am a BSP, continue with normal initialization flow (init page tables, interrupts, etc.)
  • If VCPU_INDEX == 1, I am an AP, I spin using mwait or pause instruction, waiting for a change in a well-defined "mailbox" physical memory address (which has zeros by default).
  • This well-defined address can be in a RW 4KB page of the PAL ELF binary.

CPU/NUMA topology

Must follow this document: https://cdrdv2-public.intel.com/759067/intel-64-architecture-processor-topology-enumeration.pdf

Additional resources:

• https://www.kernel.org/doc/html/v5.5/x86/topology.html
• https://elixir.bootlin.com/linux/latest/source/arch/x86/kernel/cpu/topology.c
• https://stackoverflow.com/questions/21422433/how-does-the-linux-kernel-get-info-about-the-processors-and-the-cores