Rework breakpoint mechanism

[rageagainsthepc]

Now that we have added the required funtionality to KVMi we can rework the breakpoint API in order to incorporate SLAT switches. This would enable us to use guests with more than one vCPU. It would also increase performance in situations where the memory page that contains a breakpoint is read by the guest.

[phip1611]

This would enable us to use guests with more than one vCPU

Hey @rageagainsthepc,
should we interpret this as smartvmi not supporting multiple vCPUs at all at the moment? Because that is what we keep observing in our test setup.

We always get one of the following two errors:

• what(): setSingleStepCallback: Registering a second callback to the current VCPU is not allowed.
• Exception: storeOriginalValue: Breakpoint originalValue @ xxx is already an INT3 breakpoi

Can you outline what needs to be done to solve this? What is the quick and dirty solution? What is the solid solution? We might be able to work on this and contribute.

Thanks,
Philipp

[rageagainsthepc]

I am not entirely sure whether those errors are linked to a multiple vCPU setup but it's possible (and in the case of the second one very likely). Your assumptions are correct. Right now we only support a single vCPU. The problem is that we modify pages directly with INT3 writes. When multiple vCPUs hit the same breakpoint more or less simultaneously our breakpoint mechanism will inevitably break because multiple callbacks are trying to modify the same memory location. Usually the best way to solve this is by leveraging EPT:

1. copy the physical page
2. write the breakpoint on one of the identical pages
3. create a new EPT view
4. remap the original location so it points to the newly created page
5. modify permissions so that the page with the breakpoint is only X, but not RW
6. modify the page without the breakpoint so it is RW but not X
7. register an event to be notified for every thread context switch
8. switch EPTs accordingly for every thread context switch and also if one of the page generates a memory interrupt

If am not mistaken DRAKVUF does this in a similar way and there might also be a libvmi example if you are looking for an example on how to approach this.

If you are looking for a quick and dirty solution, I recommend using a single vCPU ;)

[lbeierlieb]

Hey @rageagainsthepc,
since November, I'm supporting Cyberus Technology and IFIS with their SmartVMI engineering effort. We managed to understand the reason why SmartVMI exits with a "Registering a second callback to the current vCPU is not allowed” exception, when breakpoints are active in a VM with multiple vCPUs.

A breakpoint hit is supposed to be handled as follows:

• VM executes INT3 instruction → VM exit
• the hypervisor sends a representing event to SmartVMI, where a designated
  software interrupt handler processes it
• SW interrupt handler:
  • all callbacks registered to that breakpoint are executed
  • replace INT3 instruction in guest physical memory with original instruction byte
  • register handler for single-step event
  • put vCPU into single-stepping mode
  • finish with response to hypervisor to continue executing
• VM executes the restored instruction, then traps again due to the enabled single-stepping
• the hypervisor sends a representing event to SmartVMI, where the previously assigned single-step handler processes it
• single-step handler:
  • INT3 instruction is reinserted
  • single-stepping is disabled for this vCPU
  • finish with response to hypervisor to continue executing

This works flawlessly with only one vCPU (nothing else is running, the only vCPU is stopped or single-stepping during the whole process), and it works sometimes with other vCPUs active. However, it goes wrong when a context switch happens on another vCPU during the execution of the SW interupt handler. The context switch handler (triggered by CR3 write) goes through all breakpoints and checks for each if it is currently active and if it is supposed to be active after the context switch (there are process-specific breakpoints). If there is a discrepancy between the two, the breakpoint state is toggled accordingly.

Now, when the SW interrupt handler restores the original instruction, it also sets the breakpoint state to disabled. In our minimal SmartVMI example (SmartVMI without plugins, connected to a Windows guest), there are two global (i.e., not process-specific breakpoints), which should always be active. So, the context switch handler, which runs directly after the SW interrupt handler (and problematically, sometimes, before the single-stepping has been executed), sees that the relevant breakpoint should be active and that is currently is not, and, in consequence, re-enables the breakpoint. This also re-inserts the INT3 instruction. The single-stepping now executes the INT3, resulting in the SW interrupt handler running again, trying to register a single-step handler, causing an exception when the check determines there already is another handler registered.

The other exception "Breakpoint originalValue @ xxx is already an INT3 breakpoint" has nothing to do with the multicore setup, it occurred when restarting SmartVMI after a crash (due to the second callback in our case), because the INT3 instructions were not cleaned up and still present.

We improved upon the existing approach by implementing a "TempDisable" state for the breakpoint and also extended the logic for determining whether a process-specific needs to be active to work with multiple vCPUs. This fixes the existing problems and allows to run with multiple vCPUs. However, it is of course still limited by the approach itself and it is possible that other vCPU fly by a breakpoint when it is currently disabled for single-stepping. Does it make sense to try and upstream this "does not crash but is also not 100% reliable fo multiple vCPUs"-solution or should we rather wait with upstream contributions for an upcoming EPT/altp2m solution?

[rageagainsthepc]

Hi @lbeierlieb,

Thanks for the thorough explanation. At that moment our team is not actively developing this software any further. This might change in the future, but as of right now no work is being done besides the occasional housekeeping. Therefore I would say that it makes perfectly sense to upstream these improvements. Especially since we would probably try to make it possible to switch between breakpoint mechanism because some hypervisors (e.g. KVMi) still have trouble handling multiple sets of EPTs. Thanks in advance for your effort. 🙏

[lbeierlieb]

Thank you for the quick response and the information!
Alright, then we'll happily prepare a pull request for the existing changes and see from there.