RFC: enable openhcl to run in VTL0, use nested virtualization #281

jstarks · 2024-11-08T20:50:26Z

This change enables OpenHCL to be launched in a non-VSM capable partition (i.e., in VTL0) and to use nested virtualization to run the target guest OS.

To achieve this, we build the existing virt_kvm backend into the paravisor openvmm build. So this has all the same limitations as virt_kvm has on the host.

With this, it becomes possible to do some basic OpenHCL testing on a Linux platform, with no Windows dependencies. Of course, since we are using nested virtualization, there are some limitations:

The host is not aware of the L2 guest, so it cannot offer vmbus or assigned devices directly to it. All devices must be emulated in the paravisor. There is no support for an interactive graphics console or anything else we haven't built a relay for yet.
Performance is pretty bad in my testing. Of course, I tested this with 3x nested virtualization, so that is not too surprising. I expect this to be much better on a native Linux host.

Currently, we still depend on the GET and other Hyper-V HCL devices, so this can really only be used in conjunction with openvmm on the host. Future changes can support alternate configuration mechanisms so that this would be able to run in qemu or another VMM.

Also, virt_whp and virt_hvf do not yet support nested virtualization (even though WHP and HVF do), so even with openvmm, this only works on Linux.

Finally, I only built and tested this on x64, because I do not have a KVM ARM64 environment that supports nested virtualization.

This change enables OpenHCL to be launched in a non-VSM capable partition (i.e., in VTL0) and to use nested virtualization to run the target guest OS. To achieve this, we build the existing `virt_kvm` backend into the paravisor openvmm build. So this has all the same limitations as `virt_kvm` has on the host. With this, it becomes possible to do some basic OpenHCL testing on a Linux platform, with no Windows dependencies. Of course, since we are using nested virtualization, there are some limitations: * The host is not aware of the L2 guest, so it cannot offer vmbus or assigned devices directly to it. All devices must be emulated in the paravisor. There is no support for an interactive graphics console or anything else we haven't built a relay for yet. * Performance is pretty bad in my testing. Of course, I tested this with 3x nested virtualization, so that is not too surprising. I expect this to be much better on a native Linux host. Currently, we still depend on the GET and other Hyper-V HCL devices, so this can really only be used in conjunction with openvmm on the host. Future changes can support alternate configuration mechanisms so that this would be able to run in qemu or another VMM. Also, virt_whp and virt_hvf do not yet support nested virtualization (even though WHP and HVF do), so even with openvmm, this only works on Linux. Finally, I only built and tested this on x64, because I do not have a KVM ARM64 environment that supports nested virtualization.

daprilik · 2024-11-08T22:04:08Z

flowey/flowey_lib_hvlite/src/build_openhcl_igvm_from_recipe.rs

@@ -54,11 +54,20 @@ pub enum IgvmManifestPath {
    LocalOnlyCustom(PathBuf),
 }

+#[derive(Serialize, Deserialize, Clone, Debug)]
+pub enum InitrdRootfsPath {


nit: here and elsewhere, I would suggest using the slightly more verbose name InitrdRootfsConfigPath, to make it marginally more clear we're talking about the rootfs.config files here.

it took me a sec to realize this wasn't related to the pre-packaged initrd layers themselves

daprilik · 2024-11-08T22:11:56Z

openhcl/rootfs.config

-dir /lib/modules/000    0755 0 0
-dir /lib/modules/001    0755 0 0
-dir /lib/modules/999    0755 0 0
+dir /lib/modules/auto        0755 0 0


what's the "auto" here for?

Anything in auto will be automatically loaded. Modules elsewhere need explicit code to load.

I'll add a comment.

daprilik · 2024-11-08T22:18:34Z

openhcl/underhill_core/src/partition.rs

+//! it at some point. Right now, there are enough differences in requirements
+//! that this is not practical.


what sort of differences are we talking about here?

daprilik · 2024-11-08T22:24:35Z

openhcl/underhill_init/src/lib.rs

+            } else if vendor.is_amd_compatible() {
+                Some("amd")
+            } else {
+                None


shouldn't this have a louder error path?

mattkur · 2024-11-09T00:04:25Z

You got an example of how one would run this?

smmalis37 · 2024-11-09T00:10:09Z

Can we add some sort of basic vmm test for this? I can work with you on petri & macro support for whatever's needed (haven't looked too in depth yet).

jstarks · 2024-11-09T02:05:23Z

Can we add some sort of basic vmm test for this? I can work with you on petri & macro support for whatever's needed (haven't looked too in depth yet).

Yeah. Shouldn't be too bad. Linux x64 host only for now, until we add nested support to virt_whp.

jstarks · 2024-11-09T02:06:41Z

You got an example of how one would run this?

I'll add one, once the kernel updates are in. But basically it's similar to how you run openhcl within openvmm today, except you don't pass --vtl2.

smalis-msft · 2024-11-11T15:43:30Z

openhcl/underhill_core/src/partition.rs

+    /// exit flowing through the paravisor.
+    ///
+    /// This is best effort. Exits for this range may still flow through the paravisor.
+    fn register_host_io_port_fast_path(&self, range: RangeInclusive<u16>) -> Box<dyn Send>;


What is this return value useful for?

smalis-msft · 2024-11-11T15:43:41Z

openhcl/underhill_core/Cargo.toml

@@ -20,6 +20,11 @@ openssl-vendored = ["underhill_attestation/openssl-vendored"]
 # Enable VPCI device support
 vpci = []

+# Enable support for running the guest OS via nested virtualization with KVM.
+# (Note that the virt_kvm crate is always a dependency just to avoid build


Is this a binary size concern?

smalis-msft · 2024-11-11T15:45:53Z

openhcl/underhill_core/src/partition.rs

+    }
+}
+
+impl OpenhclPartition for virt_kvm::KvmPartition {


Maybe worth mentioning in a comment that this is not a shipping configuration, hence all the stubs?

smalis-msft · 2024-11-11T15:48:56Z

openhcl/underhill_core/src/worker.rs

@@ -292,6 +291,8 @@ pub struct UnderhillEnvCfg {
    pub no_sidecar_hotplug: bool,
    /// Enables the GDB stub for debugging the guest.
    pub gdbstub: bool,
+    /// Use KVM instead of mshv_vtl.
+    pub kvm: bool,


enum Backend? Presumably we may have more than just mshv_vtl and kvm in the future?

That could also allow carrying the ProtoPartition on an enum, instead of having to do all the if lets on it. Like it feels a little off to me that every branch where we don't have a proto partition assumes kvm, an enum could make the link clearer.

jstarks requested review from a team as code owners November 8, 2024 20:50

regen

8ebfe55

jstarks requested a review from a team as a code owner November 8, 2024 20:51

daprilik reviewed Nov 8, 2024

View reviewed changes

smalis-msft reviewed Nov 11, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: enable openhcl to run in VTL0, use nested virtualization #281

RFC: enable openhcl to run in VTL0, use nested virtualization #281

jstarks commented Nov 8, 2024

daprilik Nov 8, 2024

daprilik Nov 8, 2024

jstarks Nov 9, 2024

daprilik Nov 8, 2024

daprilik Nov 8, 2024

mattkur commented Nov 9, 2024

smmalis37 commented Nov 9, 2024

jstarks commented Nov 9, 2024

jstarks commented Nov 9, 2024

smalis-msft Nov 11, 2024

smalis-msft Nov 11, 2024

smalis-msft Nov 11, 2024

smalis-msft Nov 11, 2024

smalis-msft Nov 11, 2024

		//! it at some point. Right now, there are enough differences in requirements
		//! that this is not practical.

RFC: enable openhcl to run in VTL0, use nested virtualization #281

Are you sure you want to change the base?

RFC: enable openhcl to run in VTL0, use nested virtualization #281

Conversation

jstarks commented Nov 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattkur commented Nov 9, 2024

smmalis37 commented Nov 9, 2024

jstarks commented Nov 9, 2024

jstarks commented Nov 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment