CVE-2023-31346

AMD SEV-SNP guests can request services from the SEV firmware by exchanging encrypted messages. The content and layout of the request and response messages are defined in the SEV Secure Nested Paging Firmware ABI Specification in chapter 7.

One of these services is CPUID reporting. CPUID data contains important information about the processor, its supported features, its limits, and the positions of special address bits among other things. Unfortunately, a guest can’t get this data itself, so it has to ask the host. A SEV-SNP guest doesn’t want to trust the host though, so it can use the CPUID reporting service to check whether the untrusted, host-supplied CPUID data is correct. The guest writes some values for some CPUID leaves into the request and sends them to the SEV firmware. The SEV firmware reads those values and checks them. If any of the values are not within the bounds defined in the processor’s CPUID Policy, the values are corrected. All entries as well as a success indicator are sent back to the guest in the response message.

What’s interesting about CPUID reporting requests and responses is that they’re not fixed in size. The requests contain a COUNT field specifying how many CPUID leaves the guest has placed in the request so the sizes of the requests and responses vary accordingly. The spec caps COUNT at 64 (COUNT_MAX) and so also places an upper bound on the size of requests and responses. This is convenient for the firmware because it can avoid dynamically sized types and represent a requests and responses as fixed size structs called snp_msg_cpuid_req_t and snp_msg_cpuid_rsp_t. The firmware uses the size of the response struct when zeroing out the response before filling it with data.

While it would be possible to derive the size of the requests and responses from the COUNT field, this is not the case when they’re encrypted. To solve this the encrypted messages are preceded by an unencrypted header that contains the size of the message among other information.

Because the size of the response varies depending on how many CPUID leaves it contains, the firmware needs to calculate its size at runtime. Fortunately, this is very simple: It turns out that for CPUID reporting the layout of requests and responses is exactly the same, so to make things simple the SEV firmware simply uses the size of the encrypted request as the response’s size. While this is mostly fine, there’s one edge case the firmware didn’t account for: It’s possible for a malicious guest to send encrypted CPUID reporting requests that are larger than the snp_msg_cpuid_req_t. On first thought, this doesn’t seem like a problem: Regardless of the size of the message, the COUNT field is still capped at 64 so the firmware will just ignore the extra data. The problem is that the size is also used for the response and so suddenly the firmware reads more data from the response buffer than expected. In more concrete terms any CPUID leaves past COUNT_MAX are not zeroed when the response is initialized and contains uninitialized data in the firmware’s memory that is leaked as part of the response.

Fortunately, the leaked memory is unlikely to contain sensitive information because the SEV firmware always zeroes out sensitive information when it’s no longer needed. AFAICT this bug can only be used to leak non-sensitive data like launch pages of SEV-ES guests. For this reason, I’d classify this as a low-severity vulnerability.

There are two more variations of this bug:

1. It’s also possible to send a smaller request than would be expected for a given COUNT. In that case, the firmware would interpret uninitialized data as CPUID leaves and try to check them. This would leak the same data as the main variant but is likely harder to exploit.
2. The header for encrypted response messages contains some reserved fields. These reserved fields are never initialized and so also leak memory when they are sent back to the guest.

These bugs should be fixed by either properly initializing all response fields and/or checking the size of requests or restricting the size of responses.

Proof of Concept code for this bug is available on GitHub.

CVE-2023-31347

CPUs have internal clocks that can be read out by programs. On x86 the timestamp is stored in the internal TSC register. It can be read out using the rdtsc or rdtscp instructions. If a hypervisor wants to change a guest’s perception of time, it can set some fields in the VM’s control block to apply a scaling factor and offset to the guest’s view of the TSC register.

In SEV-SNP’s threat model, the hypervisor is untrusted, and so ideally it shouldn’t be able to change a guest’s view of time. This is implemented using a SEV-SNP feature called SecureTSC. If SecureTSC is enabled, the hypervisor can set the TSC scaling factor once when the guest is launched, but crucially can never change it again after the guest has been launched. The offset will always be 0.

SEV-SNP guests can be live-migrated to other hosts. This is very cool, but this opens up a problem: What if the old and new hosts have different TSC frequencies? If the guest was just migrated over without any changes, this would change its perception of time because the TSC frequency would suddenly change. To solve this, the SEV-SNP firmware adjusts the scaling factor such that the TSC frequency stays the same. Lastly, the TSC scaling factor is only adjusted if the SecureTSC feature is enabled in the guest.

Enough theory, let’s talk about how things are actually implemented: Both the set of enabled SEV-SNP features and TSC scaling factor are stored in a structure called the VMCB Save Area (VMSA). This region is encrypted and integrity-protected and contains state like registers, so it’s only natural for it to contain the set of features and the TSC scaling factor as well. So to check whether the SEV firmware needs to adjust the TSC scaling factor, it needs to decrypt the VMSA and check the enabled features. If SecureTSC, it can then update the scaling factor, before re-encrypting the changed VMSA. This is likely what the SEV firmware was meant to to do, but it’s not what it’s actually doing: A mistake was made and the set of enabled features is checked before the VMSA is decrypted and so the check can’t possibly be correct. AFAICT the buffer that’s checked for the SecureTSC feature is never initialized when the migration is happening and so the check is done against stale data. The result of all of this is that the scaling factor may not be adjusted like expected when a SEV-SNP guest is migrated to another host.

I’m not aware of anyone actually using SecureTSC (the Linux patches for enabling SecureTSC haven’t been upstreamed yet) and I’m also not aware of an exploit that requires messing with a guest’s perception of time, and so given the lack of potential for abuse, I’d classify as a low-severity vulnerability.

On the Publishing of the SEV Firmware

IMHO publishing the source code of the SEV firmware was a good decision.

As a security researcher, I want to not just have to trust AMD blindly, but actually check for myself that the SEV firmware is working correctly. I found and reported these two bugs shortly after the SEV firmware was published at the end of August 2023, but that’s not the whole story: While writing code for mushroom, I first noticed that some reserved fields are not zero and worked around that even in mushroom’s first public commit. The oldest reference to this workaround I was able to find in my code was from February 19, 2023, while mushroom was still private. The problem is that at the time I didn’t understand what was going on. I saw the non-zero reserved fields, but just assumed that they’d contain undocumented values and didn’t realize that they were leaking uninitialized memory from the SEV firmware. Only after the SEV firmware was published did I notice that this wasn’t intentional and was able to report this.

As a user of the SEV firmware, I hope that the increased visibility of the SEV firmware will lead to more bugs being fixed, and hope to benefit from the increased security. Having access to the source code is also invaluable when debugging code that interacts with the firmware and explaining unexpected behavior.

SEV-SNP Threat Model

SEV-SNP is a confidential computing technology available on AMD EPYC server CPUs starting with EPYC Milan. Like other confidential computing technologies (Intel SGX, ARM CCA, RISC-V CoVE) it can be used to securely run isolated workloads even if the host OS is untrusted, compromised, or malicious. In the case of SEV-SNP, these workloads are encrypted virtual machines that are isolated from the host by the CPU. The two main goals are confidentiality and integrity: It should be impossible for the host to read or write the VM’s memory and registers. It should also be impossible for the host to influence the control flow of the guest.

Intercepts

During the execution of a VM, the host can intercept certain events happening in the guest. Among these events are mostly reads and writes to special registers (e.g. CR[0-15], IDTR, GDTR) and execution of certain instructions (e.g. HLT, CPUID, INVLPG). One of these events is the nested page fault which happens when the guest attempts to access guest physical memory that isn’t mapped in the nested page tables. This event gives the host a chance to correct the nested page tables before resuming the execution of the guest.

EXITINTINFO

If an intercept is triggered while delivering an exception or interrupt, information about it is saved in the EXITINTINFO field. This allows the host to reinject the exception/interrupt after handling the intercepts to ensure proper execution in the guest.

Note that in a lot of cases, reinjection isn’t actually needed. For example, if an invalid instruction causes a #UD exception inside the guest and the interrupt descriptor table is not mapped in the nested page tables, this will cause a nested page fault. Once the host corrects the nested page tables, execution will continue at the same instruction, once again causing a #UD exception, which can now be delivered because the host mapped in the guest physical memory backing the interrupt descriptor table. The #UD exception will happen regardless of whether the host chooses to reinject.

This property is important for confidential guests because their threat model assumes that the host is untrusted. The guest can’t rely on the host to reinject exceptions.

Restricted Injection and Alternate Injection

Restricted Injection and Alternate Injection are optional features that can be activated in the guest to protect it from unexpected exceptions/interrupts injected by the host. With Restricted Injection the host can only inject #HV exceptions. With Alternate Injection the host can’t inject any exceptions (injections can only be done by the guest itself). These restrictions also apply to events in EXITINTINFO, they cannot be reinjected.

Exception Types

The AMD64 Architecture Programmer’s Manual lists three different types of exceptions:

1. Faults happen before execution of an instruction.
2. Traps happen after execution of an instruction.
3. Aborts cannot be recovered (the saved instruction pointer is unspecified).

These types primarily affect the saved instruction pointer in the interrupt stack frame and don’t actually reflect the moment the exception is generated. For example, the INTO instruction generates an #OF trap with the saved instruction pointer set to the next instruction, but the exception happens before the instruction completes executing. How do we know this? If we unmap the interrupt descriptor table and let an INTO instruction generate an #OF exception, the unmapped interrupt descriptor table causes a nested page fault intercept with the guest instruction pointer still pointing to the INTO instruction.

Faults will never need to be reinjected because they will happen again after handling an intercept.

There are only two abort-type exceptions: #DB and #MC. #DB (double fault) occurs when handling an exception causes an exception. Therefore #DB happens at the same time as the exception they’ve been caused by. They will need to be reinjected if the exception they’ve been caused by would have needed to be reinjected. It’s unclear to me whether #MC (machine check) exceptions need to be reinjected.

Lastly, traps will need to be reinjected if they happen after the instruction pointer has been updated.

Where It All Goes Wrong

There is one exception that can happen after the instruction pointer was updated: #DB (Debug). There are several ways to trigger a #DB exception:

1. Instruction execution (fault)
2. Instruction single-stepping (trap)
3. Data Read (trap)
4. Data write (trap)
5. I/O read (trap)
6. I/O write (trap)
7. Task switch (trap)
8. Debug register access (fault)
9. int1 instruction (trap but reported before updating the instruction pointer)

The traps (exept int1) are reported after the instruction pointer is increased. As a result, these exceptions can be suppressed by a malicious host if it can cause a VM exit e.g. a nested page fault to occur at the same time. This is a bug, the host shouldn’t be able to change the control flow in a SEV-SNP guest. Additionally, if Restricted Injection or Alternate Injection are enabled it’s impossible for even a well-intentioned to correctly resume processing of these events.

You can find a proof of concept showing the suppression of debug events here.

Impact

The impact of this should be fairly low for most production workloads. I’m not aware of any software that relies on debug events for security. Some features of debuggers such as single-stepping and hardware breakpoints can be rendered unusable.

Discovery

I stumbled across this bug by accident while implementing Linux KVM host patches for Restricted Injection. At first, I didn’t know about EXITINTINFO and was confused about seemingly random crashes. It turned out that KVM was trying to reinject #PF exceptions that occurred at the same time as an interrupt. Initially, I thought that KVM could simply choose not to reinject, but after thinking about it some more I realized that there are situations where this leads to incorrect behavior in the guest.

Short Intro to AMD SEV

AMD SEV, SEV-ES and SEV-SNP are confidential computing technologies available on AMD EPYC CPUs that make it possible to securely run trusted VMs on an untrusted hypervisor. In very simple terms, SEV just encrypts the guest’s memory, SEV-ES added encryption and integrity protection of the guest’s registers and SEV-SNP added integrity protection of the guest’s memory. Guests communicate with the hypervisor via unencrypted shared memory pages.

VM Exits, AEs and NAEs

Conventional VMs depend on the hypervisor to emulate certain CPU instructions e.g. cpuid (queries information about the CPU itself), inb (writes a byte to an I/O port) and rdmsr (reads a machine specific register). Whenever one of these events occurs the VM exits to the hypervisor which then emulates the event by directly changing the VM’s registers and resumes the VM.

While this works well for conventional VMs, this approach does not work starting with SEV-ES because the guest’s registers are encrypted and the hypervisor has no way to emulate most events without access to the guest’s registers. To fix this a new exception type was added: Whenever an VM exit occurs that would require access to the guest’s registers a #VC exception is triggered in the guest. The guest will look at the VM exit that was supposed to happen, determine which registers are needed to handle it and then informs the hypervisor about the VM exit and associated registers through a structure called the GHCB (Guest-Hypervisor Communication Block). The hypervisor takes the information from the GHCB, handles the VM exit like a regular VM exit and writes the results back to the GHCB. Finally the guest reads back the results from the GHCB and resumes regular control flow.

There are a couple of VM exits that don’t require access to any registers and those events will not trigger a #VC exception. These exits are called automatic exits (AE) while the ones triggering a #VC are called non-automatic exits (NAE).

MMIO

There are two ways for an hypervisor to implement MMIO (memory-mapped I/O) regions:

1. Map some memory into the region that’s shared between the guest and the hypervisor. This is very fast because it’s just a normal memory access for the guest, but the downside is that there’s no way for the hypervisor to be notified about the accesses. This method is appropriate for MMIO regions such as frame buffers.
2. Set an invalid bit in the nested page table entry that maps the guest physical address of the MMIO region to a host physical address. Once the guest tries to access the that guest physical address, the CPU will notice the invalid bit and cause a “nested page fault” VM exit. When a nested page fault occurs the CPU also informs the hypervisor about the instruction bytes that triggered it. The hypervisor is supposed to decode the instruction and emulate the memory access accordingly. This is fairly slow because there’s a context switch on every access, but the upside is that the hypervisor can handle each access individually. This method is approriate for MMIO regions such as registers in PCI devices.

Both of these methods also work with SEV(-ES/-SNP) guests:

1. The guest controls whether a memory page is shared with the hypervisor by setting a special bit, the C bit, in its pagetables. If the guest knows that a region is supposed to be a MMIO region, it can omit the C bit and accesses to the page are no longer encrypted. Both the hypervisor can see the same unencrypted memory.
2. With SEV nested page fault VM exits doesn’t require any intervention from the guest. The hypervisor can decode the faulting instruction and access the registers as it would for a conventional VM. With SEV-ES and SEV-SNP nested page faults are considered non automatic exits and so the guest will have to handle them. Unlike the hypervisor, the guest is not given the instruction bytes of the faulting instruction. Instead it’s supposed to look at the instruction pointer at the time of the exception, decode the instruction from its memory and finally emulate it.

The Bug

It turns out that handling nested page fault exceptions in SEV-ES/SEV-SNP is inherently racy: The guest kernel isn’t provided with the instruction bytes of the faulting instruction, so it has to go and fetch them itself, but by the time it’s doing that the instruction bytes might have already been changed by another thread. As a result the kernel shouldn’t assume that the instruction it has fetched is the one that caused the #VC exception. Usually when the CPU tries to access memory it checks that it’s currently allowed to access that memory. Those permission checks happen before a #VC exception would be triggered. The problem with the way Linux was handling nested page fault exceptions in its #VC handler was that it was incorrectly assuming that the CPU had already checked memory permissions for the instruction Linux decoded. This is vulnerable to the following race condition:

1. Thread 1 executes the instruction mov [MMIO], 1234 at address 0x414141
   • This instruction tries to write the value 1234 to address MMIO.
   • The CPU checks that it’s allowed to access MMIO.
   • MMIO is MMIO region with an invalid bit set in the nested page tables, so a nested page fault #VC exception is triggered.
2. Thread 2 changes the instruction at 0x414141 to movs [KERNEL], [USER]
   • This instruction reads the values at address USER and writes it to address KERNEL.
3. On thread 1, the kernel decodes the instruction at 0x414141 and sees movs [KERNEL], [USER] and emulates it without permission checks.

When neither USER nor KERNEL point to a MMIO memory region, Linux would emulate this as such by simply copying the value from USER to KERNEL ¹. This is a problem because the write to KERNEL shouldn’t be allowed to happen as regular usermode programs shouldn’t have access to kernel memory from usermode. As a result this vulnerability can be used to write arbitrary kernel memory. By swapping the USER and KERNEL arguments this can also be used to read arbitrary kernel memory instead.

Another implementation detail that is very convenient is that Linux will properly forward page fault exceptions while emulating the movs instruction. This means that if KERNEL is not a valid address, the kernel will not crash, but forward the fault to the user which can handle the exception. This can be used to check whether an address is valid.

There is one significant problem with this attack: It relies on the user having access to a MMIO region. This is commonly not possible because usermode programs rarely directly communicate with the devices that expose these MMIO regions. There’s a way to solve this though: Remember how the hypervisor is explicity untrusted in SEV’s threat model? If we assume the hypervisor is malicous, it can change a regular user mapping into a MMIO memory region by simply setting an invalid bit in the nested page table for that address. As a result a malicous hypervisor can help an unprivileged process in the guest achieve privilege escalation.

Exploit

This vulnerability gives us the following exploit primitives:

1. Check whether an address is valid
2. Arbitrary kernel read
3. Arbitrary kernel write

My proof of concept exploit performs the following steps to escalate it’s privileges:

1. Allocate a huge chunk of memory and write to it to force the kernel to populate it.
2. Wait for the untrusted hypervisor to execute the custom attack command in QEMU. Once this command has been executed, the host will set an invalid bit in the nested page table entry for the guest physical address at 0x1f8b0c000 and therefore change it to be a MMIO region. The hope is that this address is belongs the the big chunk of memory allocated in step 1.
3. Continously write to all pages in the allocated chunk of memory using a mov. Once the page that was turned into a MMIO page is hit, the write will trigger a #VC exception. Unlike for the movs instruction, for the mov instruction the kernel will check that the memory region was supposed to be an MMIO memory region. Once the kernel notices that the region wasn’t supposed to be a MMIO region it delivers a SIGBUS signal to usermode. The PoC uses this signal to detect when it hit the page that was turned into a MMIO page.
4. Use primitive 1 to find the base address of the kernel by trying all possible values.
5. Use primitive 2 to read from kernel memory in order to find the init task.
6. Use primitive 2 to iterate over the init tasks children to find the task_struct associated with the PoC’s process.
7. Use primitive 2 to read the credentials of the init task (i.e. root credentials) and use primitive 3 to copy them over to the PoC’s task. This escalates the task’s privileges to root.
8. Spawn a shell.

Here’s a video demonstration of the PoC running on an unpatched system: https://www.youtube.com/watch?v=6D90uO_PgoU.

Patch

This vulnerability was patched by restricting MMIO accesses to the kernel because of the complexity involved in preventing the race condition. It turned out that there isn’t any usermode code interacting with MMIO regions anyways, so this doesn’t break things in practice.

A variant of this bug exists for I/O instructions which suffer from the same race condition between execution of the instruction, permission checks by the CPU and decoding of the instruction in the kernel. This variant was patched by checking the I/O port permissions again in the kernel before emulating the instruction.

Footnotes

To understand why we need to talk about a bit about the motherboard I picked - the Supermicro H12SSL-NT. On this motherboard the fans are not actually controlled by the operating system but a separate chip called the ASPEED AST2500 BMC. It’s not possible to modify the firmware directly, but it exposes a web panel that can be used to figure it. Taking a look at the sensor section quickly revealed the problem:

The dual 10Gb network controller regularly reaches 70°C (158°F) causing the fans to spin up. On the “Optimal” fan mode this results in the fans ramping up to about 30%. This may not seem like much but in an otherwise completely silent room I could easily hear the fans at night. Other than that the absolute lowest duty the BMC will ever set the fans to is 20% which again is unfortunately still enough to be audible.

The H12SSL-NT supports up to 7 fans in 2 zones with 4 different fan modes, but unfortunately it’s not possible to manually adjust any fan curves. Not satisfied with this I came up with the following plan:

1. Get RCE on the BMC.
2. Patch the curves on the running system.
3. Profit

Achieving remote code execution

I downloaded the latest firmware from Supermicro’s website, ran binwalk and threw some of the binaries into Ghidra. Eventually I discovered some code responsible for sending out email notifications that looked vulnerable to command injection. Simplified the code looks like this:

snprintf(buffer, 0x1000,
         "echo \'%s\' | %s --host=%s --port=%d --domain=%s --timeout=1 --auth=off --from=%s %s",
         message,
         "/bin/msmtp",
         host,
         port,
         domain,
         from,
         sender);
run_shellcmd(buffer);

host, port, domain, from and sender can be controlled by an authenticated attacker by simply changing the values in the email notification settings.

This is already useful for me and my server because I’m obviously an admin on my own machine and can change the settings to exploit this vulnerability, but I decided to dig a little deeper to see if the bug can also be triggered by an unauthenticated attacker. There are several event types that will trigger an email to be sent out. They can be grouped into three categories: alerts, resource changes and status changes. These events can also be seen in the Maintenance Event Log.

The login messages quickly piqued my interest because an unauthenticated attacker might be able to influence them and they would likely end up in the message parameter. After a bit of trial and error and discovering that spaces are apparently not allowed in the username I was able to come up with '`ping${IFS}-c${IFS}4${IFS}192.168.13.4` and was able to confirm code execution.

# tcpdump -i enp9s0 icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp9s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:42:07.012482 IP 192.168.13.12 > desktop: ICMP echo request, id 2058, seq 0, length 64
12:42:07.012500 IP desktop > 192.168.13.12: ICMP echo reply, id 2058, seq 0, length 64
12:42:08.004739 IP 192.168.13.12 > desktop: ICMP echo request, id 2058, seq 1, length 64
12:42:08.004757 IP desktop > 192.168.13.12: ICMP echo reply, id 2058, seq 1, length 64
12:42:09.004745 IP 192.168.13.12 > desktop: ICMP echo request, id 2058, seq 2, length 64
12:42:09.004760 IP desktop > 192.168.13.12: ICMP echo reply, id 2058, seq 2, length 64
12:42:10.004776 IP 192.168.13.12 > desktop: ICMP echo request, id 2058, seq 3, length 64
12:42:10.004805 IP desktop > 192.168.13.12: ICMP echo reply, id 2058, seq 3, length 64

The exploit attempt as seen in the logs:

At this point I responsibly disclosed the vulnerability to Supermicro. This vulnerability has been assigned CVE-2023-35861.

Now, in this blog post I’ll just use this to run my own code on the BMC to patch the fan curves, but a malicious attacker could potentially do a lot more damage as they have essentially full control over the server, so the actual impact is higher than it might seem at first.

Patching the fan curves

While exploring the firmware I gained a good understanding of the internal workings of the BMC and had already discovered how the fan curves are implemented. For each fan zone there’s a list of sensors in them. Bundled with each sensor are 4 temperature and duty values for each fan mode. As an example the network controller that caused me troubles has the following values by default:

zones:
  - MB_10G:
      Standard:
        65: 20
        73: 40
        84: 80
        90: 100
      Full:
        65: 100
        73: 100
        84: 100
        90: 100
      Optimal:
        65: 20
        73: 45
        84: 81
        90: 100
      HeavyIO:
        65: 20
        73: 45
        84: 81
        90: 100

(The firmware does not actually represent the values in yaml.)

These values are found in a shared library called libipmi.so and are used by a process called ipmi_sensor. To patch the curves I wrote fan-recurve - a tool can be used to dump and patch the curves in a running system. It uses the ptrace API to attach to the process and locate and modify the fan curves in place.

Results

Now, did all of this actually work? Yes, it helped a lot.

Before the fans looked like this … … and now they look like this.

Note that the BMC marks some of the fans as “Critical” but this just means that they’ve spun all the way down to 0 rpm, so this is actually a good thing and exactly what I wanted in the first place. For the network controller I installed a tiny 40mm fan blowing air onto its heat sink. That fan is the only fan in zone 2 (FANA), so I changed the fan curves in zone 2 to be higher than what they usually are to keep the network controller nice and cool.

In case anyone’s worried that spinning fans to 0 rpm will cause problems, here are the temperatures with my modified fan curves, they’re completely fine.