Windows guest crashes when I load windows boot manager after loading MiniVisor, right after when ExitBootServicesHandler is called

[subvert0r]

I get the following error whenever I try to load windows boot manager after I have loaded the MiniVisor EFI Driver, exactly right after ExitBootServicesHandler is called and windows logo is shown:

A fault has occurred causing a virtual CPU to enter the shutdown state. 
If this fault had occurred outside of a virtual machine, it would have caused the physical machine to restart. 
The shutdown state can be reached by incorrectly configuring the virtual machine, a bug in the guest operating system, or a problem in VMware Workstation.
Click OK to restart the virtual machine or Cancel to power off the virtual machine.

Tried it with Windows 10 and 11.

Seems to be related to
#12

but this also happens when I set the number of virtual cores for the guest to 1.

secure boot is off, VBS is off, and nested virtualization (Virtualize VT-X) is turned on in VMware workstation config for the VM.
This stops happening when I change the EFI code and make it not turn on hypervisor on CPU cores (via commenting EnableHypervisorOnAllProcessors).

[tandasat]

Thanks for reporting the issue. I have not been able to repro this with either 1 or 2 cores. Here is what I do:

• Try with the same EDK2 (although I doubt this is an issue)

git clone -b edk2-stable202111 --recurse-submodules https://github.com/tianocore/edk2

• Enable the full logging on VMware workstation and try.

With that, send me

• vmware.log generated in the same dir as .vmx file on triple fault (this should contain line(s) with TRIPLE FAULT IIRC)
• either serial or console logs showing the base address of the hypervisor (eg, log line starting with PrintLoadedImageInformation), and
• .efi and .pdb files generated in edk2\Build\MiniVisor\NOOPT_VS2019\X64\MiniVisorPkg\Builds\Platform\EFI\MiniVisorDxe\DEBUG

Finally, what's your processor on the host? Mine is i7-1265U.

[subvert0r]

Thanks for reporting the issue. I have not been able to repro this with either 1 or 2 cores. Here is what I do:

• Try with the same EDK2 (although I doubt this is an issue)

git clone -b edk2-stable202111 --recurse-submodules https://github.com/tianocore/edk2

• Enable the full logging on VMware workstation and try.

With that, send me

• vmware.log generated in the same dir as .vmx file on triple fault (this should contain line(s) with TRIPLE FAULT IIRC)
• either serial or console logs showing the base address of the hypervisor (eg, log line starting with PrintLoadedImageInformation), and
• .efi and .pdb files generated in edk2\Build\MiniVisor\NOOPT_VS2019\X64\MiniVisorPkg\Builds\Platform\EFI\MiniVisorDxe\DEBUG

Finally, what's your processor on the host? Mine is i7-1265U.

I found the root cause of it. It was because I loaded the minivisor efi module by adding a new boot entry (by using https://github.com/Metabolix/HackBGRT project), and booting from that and then adding the code for loading the next boot entry which is Windows.

Turns out it seems like the first EFI module that gets loaded by the firmware cannot be a Runtime driver (but idk), and needs to be a EFI application (loader), so I wrote a loader for it and it fixed the issue.

Although I'm not sure if this limitation even exists or not, but looking at other projects most of them write a loader EFI application to load the Runtime EFI driver for some reason?

[tandasat]

Thank you for reporting back to me with the root cause. Glad to hear it resolved.

It is an interesting discovery about the boot entry. I did not look into the spec but given that it starts a UEFI program, it makes sense that UEFI drivers do not work.

My guess on the reason of many projects using loader apps is flexibility. To load a runtime driver, you need to enter the UEFI shell, which requires manual operations and disabling secure boot. If you build a loader as an UEFI app, you could drop it off and make it work as you intended. If you sign it and edit secure boot db, then it could become secure boot compatible.

[subvert0r]

Thank you for reporting back to me with the root cause. Glad to hear it resolved.

It is an interesting discovery about the boot entry. I did not look into the spec but given that it starts a UEFI program, it makes sense that UEFI drivers do not work.

My guess on the reason of many projects using loader apps is flexibility. To load a runtime driver, you need to enter the UEFI shell, which requires manual operations and disabling secure boot. If you build a loader as an UEFI app, you could drop it off and make it work as you intended. If you sign it and edit secure boot db, then it could become secure boot compatible.

Unfortunately the problem happened again, this time in a KVM windows 11 VM. It is exactly similar to:

#12

Right after ExitBootServicesHandler was called is printed, machine freezes and nothing happens.

Would you mind giving it a try with KVM and see if it happens for you too or not?

[tandasat]

Unfortunately, I will not going to have a time to test it out with KVM. I do not have much experience with KVM, but perhaps you can debug the guest and see what guest RIP looks like.