The unified modular hypervisor based on ArceOS

[hky1999]

About the overall architecture of arceos-hypervisor,
a unified modular hypervisor based on ArceOS.

Design Goal

This project originated from the discussion/13 of rCore-OS community.

In general, this project hopes to add virtualization support crates/modules based on ArceOS unikernel,
and build a modular hypervisor that supports multiple architectures based on the basic OS functions provided by ArceOS unikernel.

We hope to make the hypervisor as modular as possible and minimize modifications to the arceos kernel code.

Components

ArceOS-hypervisor is mainly composed of the following independent components:

• vmm-app: a user app of ArceOS, acts like a VMM (Virtual Machine Monitor)

  • completely architecture-independent
  • responsible for VM management (configuration & runtime)

• axvm: a module of ArceOS, responsible for resource management within each VM

  • partially architecture-independent
  • resources:
    • address space of guest VM
    • axvcpu list
    • axdevice list

• axvcpu: a module of ArceOS, providing CPU virtualization support

  • highly architecture-dependent
  • stores exception context frame of different architecture
  • basic scheduling item

• axdevice (unimplement): a module of ArceOS, providing device emulation support

  • partially architecture-independent
  • different emulated device implementations need to be separated into separate crates

VCpu Scheduling

axvcpu is just and only reponsible virtualization function support, e.g. enter/exit guest VM through vmlaunch/vmexit.

Since ArceOS already provides axtask for runtime control flow mangement under single privilege level,
we can reuse its scheduler and evolve with it.

VCpu scheduling upon ArceOS may looks like this:

    for vcpu in vm.vcpu_list() {
        axtask::spawn(|| {
                let curr = axtask::current();
                let vcpu = unsafe { curr.task_ext().vcpu };
                let vm = unsafe { curr.task_ext().vm };

                loop {
                    let exit_reason = vcpu.run();
                    match exit_reason {
                        MMIO(emu_ctx) => vm.handle_device(emu_ctx),
                        HVC(emu_ctx) => vm.handle_hvc(emu_ctx),
                        YIELD => axtask::yield_now(),
                        EXIT(code) => axtask::exit(code),
                        ...
                    }
                }
            }
        );
    }

In order to support such vcpu scheduling,

For axtask, it needs to

• find a way to get the vcpu running on the current physical core and the VM it belongs to.

  this can be achieve by TaskExt design from arceos monolithic variant.
  there has a simple demo

• support multiple scheduling modes, including pinned on core and core affinity settings, etc.

  this need to modify & refactor current scheduling framework of ArceOS

• support spawning tasks in blocked state, and wait until the VM boot command to set the state of all vcpu tasks to ready

  this can be achieve by semantics like JoinHandler

For axvcpu, it needs to

• have a common and standardized interface to handle operations such as binding physical cores, running, and giving up physical cores of vCPUs from different architectures

  as defined in axvcpu/src/arch_vcpu.rs

      /// A trait for architecture-specific vcpu.
      ///
      /// This trait is an abstraction for virtual CPUs of different architectures.
      pub trait AxArchVCpu: Sized {
          /// The configuration for creating a new [`AxArchVCpu`]. Used by [`AxArchVCpu::new`].
          type CreateConfig;
          /// The configuration for setting up a created [`AxArchVCpu`]. Used by [`AxArchVCpu::setup`].
          type SetupConfig;
  
          /// Create a new `AxArchVCpu`.
          fn new(config: Self::CreateConfig) -> AxResult<Self>;
  
          /// Set the entry point of the vcpu.
          ///
          /// It's guaranteed that this function is called only once, before [`AxArchVCpu::setup`] being called.
          fn set_entry(&mut self, entry: GuestPhysAddr) -> AxResult;
          /// Set the EPT root of the vcpu.
          ///
          /// It's guaranteed that this function is called only once, before [`AxArchVCpu::setup`] being called.
          fn set_ept_root(&mut self, ept_root: HostPhysAddr) -> AxResult;
          /// Setup the vcpu.
          ///
          /// It's guaranteed that this function is called only once, after [`AxArchVCpu::set_entry`] and [`AxArchVCpu::set_ept_root`] being called.
          fn setup(&mut self, config: Self::SetupConfig) -> AxResult;
  
          /// Run the vcpu until a vm-exit occurs.
          fn run(&mut self) -> AxResult<AxArchVCpuExitReason>;
          /// Bind the vcpu to the current physical CPU.
          fn bind(&mut self) -> AxResult;
          /// Unbind the vcpu from the current physical CPU.
          fn unbind(&mut self) -> AxResult;
      }

• converge all interactions with the guest privilege level within the vcpu.run() function, so that a loop block can handle all access from guest VM.

Exception (VM-Exit)

Exception Handling

The vcpu scheduling design mentioned above requires a reasonable exception (VM-Exit) handling framework.

This is the problem:

Due to the different ways in which different architectures support virtualization,
their virtualization privilege level exception handling entries are also different.

We need to design a set of VM-Exit processing frameworks that are as unified as possible for different architectures.

• x86_64

  The handling of VM-Exit under x86_64 has been completed by @aarkegz, see crates/axvm/src/arch/x86_64/vmx
  /vmcs.rs

  For x86_64's virtual machine extensions, the host-state area of the VMCS region can flexibly determine the rip value after a VM-Exit occurs, which is set as the function address of vmx_exit inside VmxVcpu structure.
  Therefore, we can save the context properly in the host sp during vmlaunch/resume, store the host sp pointer, and pop the context from the host sp in the vmx_exit function when a VM-Exit occurs. All of these operations are performed in vmx mod, which elegantly limits the interaction with the guest VM to the vcpu.run() function.

• aarch64

  For aarch64 architecture, VBAR_EL2 register holds the vector base address for any exception that is taken to EL2.
  So, to run arceos in EL2 to support virtualization, we need to make intrusive modifications to arceos's [axhal module] (https://github.com/arceos-org/arceos/tree/main/modules/axhal).

  • For VM-Entry

    • arch_vcpu.run()
    • save callee saved registers (EL2)
    • pass context frame pointer, something like &vcpu.vm_context_frame
    • recore VM context from &vcpu.vm_context_frame, including
      • status register
      • general register
    • eret

  • For VM-Exit

    • exception handler
    • get pointer of current vcpu's context frame pointer
    • store VM context into &current_vcpu.vm_context_frame, including
      • status register
      • general register
    • pop callee saved registers (EL2)
    • back to arch_vcpu.run()

Exception Type (VM-Exit Reason)

• architecture independent
• reference: KVM exit reasion
• current implementation crates/axvm/src/vcpu.rs

Memory Management

This is very similar to the address space management of the arceos-monolithic.

@equation314 suggested that we can use axmm.

However, its PageTable of aspace cannot be changed.
It uses the PageTable type provided by arceos axhal/paging, and the page table entry structure cannot be modified.

    use axhal::paging::{MappingFlags, PageTable};
    /// The virtual memory address space.
    pub struct AddrSpace {
        va_range: VirtAddrRange,
        areas: MemorySet<MappingFlags, PageTable, Backend>,
        pt: PageTable,
    }

We can take advantage of its memory_set crate and register the PageTable as our AxNestPageTable.

Still, we hope to find a way to unify address space management for both monolithic and hypervisor variants of ArceOS.

@hky1999 is working on that, dev branch : https://github.com/arceos-hypervisor/arceos-umhv/tree/vm_cfg

Emulated Device

Emulated Device support for arceos-hypervisor is supported by the implementation of Exception Handling and Memory Management (most of device are accessed by MMIO).

We plan to design an axdevice module for device management of each VM, it provides struct like AxEmulatedDevices, which will be owned and managed by AxVM.

pub struct AxEmulatedDevices {
    mmio_devices: BTreeMap<Range<usize>, dyn EmuDev>,
    #[cfg(target_arch = "x86_64")]
    pio_devices: BTreeMap<Range<usize>, dyn EmuDev>,
}

pub trait EmuDev {
    fn emu_type(&self) -> EmuDeviceType;
    fn address_range(&self) -> Range<usize>;
    fn handler(&self, ctx: &AccessContext) -> AxResult;
}

When a VM-Exit caused by MMIO (or PIO) access occurs, axvcpu will record the current access information, including address, bit width, read/write, etc. The emulated_device_handler function of axdevice needs to find the corresponding emulated device according to the access address, call the corresponding device's processing function and pass in the access information.

Providing emulated device support for guest VMs requires considerable work.
Currently we focus on emulated interrupt controller for different architectures and virtio-devices (mainly Virtio-Blk).

[aarkegz]

A question:

VCpu scheduling upon ArceOS may looks like this:

    for vcpu in vm.vcpu_list() {
        axtask::spawn(|| {
                let curr = axtask::current();
                let vcpu = unsafe { curr.task_ext().vcpu.clone() };
                let vm = unsafe { curr.task_ext().vm.clone() };

                loop {
                    let exit_reason = vcpu.run();
                    match exit_reason {
                        MMIO(emu_ctx) => vm.handle_device(emu_ctx),
                        HVC(emu_ctx) => vm.handle_hvc(emu_ctx),
                        YIELD => axtask::yield_now(),
                        EXIT(code) => axtask::exit(code),
                        ...
                    }
                }
            }
        );
    }

In order to support such vcpu scheduling,

Should we clone vm and vcpu here? Or vcpu and vm here are just smart pointers like Arc or something?

[hky1999]

Y R right, it;s Arc and no need to clone, you can see the demo in PR

[aarkegz]

Got it. That makes perfect sense.

[hky1999]

In fact, it will be a clippy warning

[aarkegz]

oh?

[aarkegz]

To me it's a very good design doc. 👍

[hky1999]

Do not gao me 😭😭😭