Understanding the exec command & Resource usage

[ccrvlh]

After deploying the example, I press "s" within k9s to shell into the pod, and I'm not sure where I go to:

• uname -r shows my hosts kernel
• free -m shows the host memory
• pstree only shows the tini+dnsmasq with entrypoint.sh and the cloudhypervisor tree

That's somewhat unexpected, how would the pod know about the host?
-> It seems that the "container" is being read as a cloud-hypervisor host container itself, instead of having the vm as the unit within the pod, so pod > "host" container > vm

--

The other thing I couldn't quite understand is the resource consumption.
Metrics server (as shown through k9s) shows ~200MB of RAM for the POD and 1m CPU, when I do htop on the host, I can indeed find the VM process with cloud-hypervisor's socket process running, and that also shows 193MB.

When I SSH into the image (like the instructions in the README), I can see the VM kernel (different from the host), so I'm indeed within the VM. Doing free -m shows 66MB of used memory and 1024MB total as expected. I'm not entirely sure, but
~130MB seems like a fairly high overhead for cloud hypervisor?

The last thing I noticed is that the kubectl get vm command doesn't work when you also have kubevirt running in the cluster, which is somewhat expected given that the resources do have conflicting names.

[fengye87]

When you press "s", you shell into the VM pod, not the VM. You cannot shell into the VM directly, that's not how Virtink works.

We don't have the exact numbers for CH's overhead, but 130MB doesn't sound too surprise to me. The CH folks may have a better answer here.

And yes, since Virtink and kubevirt are both using the vm resource alias, kubectl (and k9s) would get confused of which resource you're requesting. You may use the fully qualified name to disambiguate them.

[ccrvlh]

The reason the size caught my attention is that this is almost twice the size of VM's deployed using Kata Containers (driven by CH), and about 1,5x more than VMs deployed with Kubevirt (QEMU). I'll run some tests with different images/kernels to see if I can find any relevant different in size. I was running a couple of tests on memory earlier and free ended up not being so accurate (when looking at the process as a whole), Firecracker has really low overhead at around ~6MB, so don't really expect this to be CH, so this may be just the image being used...

Thinking about the "exec" now looks pretty obvious, indeed. So to make sure I'm following, what's actually happening is: VirTink is deploying a regular pod with a base container image that carries the CH binary, and then the CRD is passing the kernel/rootfs image to the pre runner that is actually starting the CH socket and starting the machine. For that to work mounting /dev/kvm is needed (and maybe some other host devices) and that's why I can see the host information while exec'ing into the pod. So from what I understand this is a more "shallow" virtualization strategy (in the sense that it happens in the last layer of a K8S resources, and not that the virtualization is shallow per se), whereas Kata is actually changing the runtime, so instead of building the cgroup/chroot etc etc for the container it's actually starting a VM.

Does that make sense? Curious what strategy Kubevirt uses though.

[fengye87]

Yes, your understanding is right. And KubeVirt is using a similar strategy.

[ccrvlh]

@fengye87 thanks. One last question if you don't mind: how was the design process for layered pod > container > VM > container? By looking at Kata, which has pod > vm > agent > container, I wonder if it would be possible to reduce the number of layers, and just have a direct pod > vm connection. The vm would have to behave just like a container for it's manager (eg containerd), so without additional tools, there would be restrictions, but would allow for a more direct way of managing micro VM without having to go through the extra container layers. Wondering if that would be possible and if you guys considered that approach before going with the current implementation. Thanks!

[fengye87]

That all depends on which interface you're integrating with. For kata, it decided to integrate on the CRI interface, so it has to act like a container runtime out of a virtualization core. For Virtink, we decided to not be invasive to K8s and tried to be just an add-on that can be installed to any canonical K8s clusters. It's true that we've considered some other implementations. We've considered to implement our own virtualization specialized kubelet to deal with VM directly, but still keep K8s original scheduler. If we went that way, we could reduce the number of layers as much as possible, but would also become far less K8s-ish. The way we see K8s is it's a cluster OS, something comparable to Linux, but on the cluster level. We believe K8s should become a common layer for any distributed systems. That's why we decided not to fight with K8s.