TL;DR: The host can force a SEV-SNP guest to skip some debug events such as single-step breakpoints and hardware data breakpoints on affected processors by forcing an intercept to occur at the same time. If Restricted injection or Alternate injection is enabled it may be impossible for the host to correctly pass those events to the guest when a VM exit has occurred at the same time. AMD considers the suppression of debug events in SEV-SNP guests a low-severity vulnerability and assigned it CVE-2023-20573. You can find out more in their security bulletin.

SEV-SNP Threat Model

SEV-SNP is a confidential computing technology available on AMD EPYC server CPUs starting with EPYC Milan. Like other confidential computing technologies (Intel SGX, ARM CCA, RISC-V CoVE) it can be used to securely run isolated workloads even if the host OS is untrusted, compromised, or malicious. In the case of SEV-SNP, these workloads are encrypted virtual machines that are isolated from the host by the CPU. The two main goals are confidentiality and integrity: It should be impossible for the host to read or write the VM’s memory and registers. It should also be impossible for the host to influence the control flow of the guest.

Intercepts

During the execution of a VM, the host can intercept certain events happening in the guest. Among these events are mostly reads and writes to special registers (e.g. CR[0-15], IDTR, GDTR) and execution of certain instructions (e.g. HLT, CPUID, INVLPG). One of these events is the nested page fault which happens when the guest attempts to access guest physical memory that isn’t mapped in the nested page tables. This event gives the host a chance to correct the nested page tables before resuming the execution of the guest.

EXITINTINFO

If an intercept is triggered while delivering an exception or interrupt, information about it is saved in the EXITINTINFO field. This allows the host to reinject the exception/interrupt after handling the intercepts to ensure proper execution in the guest.

Note that in a lot of cases, reinjection isn’t actually needed. For example, if an invalid instruction causes a #UD exception inside the guest and the interrupt descriptor table is not mapped in the nested page tables, this will cause a nested page fault. Once the host corrects the nested page tables, execution will continue at the same instruction, once again causing a #UD exception, which can now be delivered because the host mapped in the guest physical memory backing the interrupt descriptor table. The #UD exception will happen regardless of whether the host chooses to reinject.

This property is important for confidential guests because their threat model assumes that the host is untrusted. The guest can’t rely on the host to reinject exceptions.

Restricted Injection and Alternate Injection

Restricted Injection and Alternate Injection are optional features that can be activated in the guest to protect it from unexpected exceptions/interrupts injected by the host. With Restricted Injection the host can only inject #HV exceptions. With Alternate Injection the host can’t inject any exceptions (injections can only be done by the guest itself). These restrictions also apply to events in EXITINTINFO, they cannot be reinjected.

Exception Types

The AMD64 Architecture Programmer’s Manual lists three different types of exceptions:

  1. Faults happen before execution of an instruction.
  2. Traps happen after execution of an instruction.
  3. Aborts cannot be recovered (the saved instruction pointer is unspecified).

These types primarily affect the saved instruction pointer in the interrupt stack frame and don’t actually reflect the moment the exception is generated. For example, the INTO instruction generates an #OF trap with the saved instruction pointer set to the next instruction, but the exception happens before the instruction completes executing. How do we know this? If we unmap the interrupt descriptor table and let an INTO instruction generate an #OF exception, the unmapped interrupt descriptor table causes a nested page fault intercept with the guest instruction pointer still pointing to the INTO instruction.

Faults will never need to be reinjected because they will happen again after handling an intercept.

There are only two abort-type exceptions: #DB and #MC. #DB (double fault) occurs when handling an exception causes an exception. Therefore #DB happens at the same time as the exception they’ve been caused by. They will need to be reinjected if the exception they’ve been caused by would have needed to be reinjected. It’s unclear to me whether #MC (machine check) exceptions need to be reinjected.

Lastly, traps will need to be reinjected if they happen after the instruction pointer has been updated.

Where It All Goes Wrong

There is one exception that can happen after the instruction pointer was updated: #DB (Debug). There are several ways to trigger a #DB exception:

  1. Instruction execution (fault)
  2. Instruction single-stepping (trap)
  3. Data Read (trap)
  4. Data write (trap)
  5. I/O read (trap)
  6. I/O write (trap)
  7. Task switch (trap)
  8. Debug register access (fault)
  9. int1 instruction (trap but reported before updating the instruction pointer)

The traps (exept int1) are reported after the instruction pointer is increased. As a result, these exceptions can be suppressed by a malicious host if it can cause a VM exit e.g. a nested page fault to occur at the same time. This is a bug, the host shouldn’t be able to change the control flow in a SEV-SNP guest. Additionally, if Restricted Injection or Alternate Injection are enabled it’s impossible for even a well-intentioned to correctly resume processing of these events.

You can find a proof of concept showing the suppression of debug events here.

Impact

The impact of this should be fairly low for most production workloads. I’m not aware of any software that relies on debug events for security. Some features of debuggers such as single-stepping and hardware breakpoints can be rendered unusable.

Discovery

I stumbled across this bug by accident while implementing Linux KVM host patches for Restricted Injection. At first, I didn’t know about EXITINTINFO and was confused about seemingly random crashes. It turned out that KVM was trying to reinject #PF exceptions that occurred at the same time as an interrupt. Initially, I thought that KVM could simply choose not to reinject, but after thinking about it some more I realized that there are situations where this leads to incorrect behavior in the guest.