fix timeout when stopping KVM machine with CRI-O container runtime

[prezha]

we noticed in our integration tests that kvm machine with cri-o container runtime will sometime not gracefully shutdown waiting a given time of two minutes (eg: https://storage.googleapis.com/minikube-builds/logs/19736/36442/KVM_Linux_crio.html#fail_TestStartStop%2fgroup%2fembed-certs%2fserial%2fStop)

there is a known upstream issue that is tracking that

as we had similar issues in the past (and increased the timeout value several times), this pr aims to solve that problem by:

• waiting initial 90 seconds for the kvm machine to gracefully stop, and if that fails, instead of terminating with error:
• requesting a forceful shutdown (ie, "destroy" in libvirt) and waiting for another 30 seconds

in total, this should: self-resolve the issue in many cases, while reducing the total wait period

bonus:

• ensure any error from trying to close connection/domain is caught and logged but does not overwrite the underlying error
• more consistent error handling
• more consistent logging

[prezha]

/ok-to-test

[spowelljr]

Hi @prezha, following up on this according to cri-o/cri-o#8609 (comment), this was fixed in crun 1.17. We've updated to crun 1.17 in #19640, is this PR still needed?

[prezha]

hey @spowelljr, i think we do as we've occasionally seen these timeouts even before (for reasons related to other/different issues), so it would also reduce the flakiness of our integration tests

[spowelljr]

The before message is a better user facing error, cannot create vs creating, although we should have validation for this way before

[prezha]

agree, reworded it

[minikube-pr-bot]

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 19758) |
+----------------+----------+---------------------+
| minikube start | 52.5s    | 51.2s               |
| enable ingress | 18.4s    | 18.0s               |
+----------------+----------+---------------------+

Times for minikube ingress: 16.6s 19.1s 19.6s 19.5s 17.1s
Times for minikube (PR 19758) ingress: 18.6s 16.6s 19.1s 19.1s 16.6s

Times for minikube start: 52.2s 53.5s 49.5s 52.9s 54.3s
Times for minikube (PR 19758) start: 51.5s 54.4s 49.8s 48.3s 51.9s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 19758) |
+----------------+----------+---------------------+
| minikube start | 23.6s    | 24.0s               |
| enable ingress | 13.0s    | 12.8s               |
+----------------+----------+---------------------+

Times for minikube start: 24.7s 24.4s 21.6s 22.4s 25.0s
Times for minikube (PR 19758) start: 24.7s 22.3s 25.2s 22.4s 25.3s

Times for minikube ingress: 13.8s 12.3s 12.9s 13.3s 12.8s
Times for minikube (PR 19758) ingress: 13.3s 12.4s 13.3s 12.3s 12.9s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 19758) |
+----------------+----------+---------------------+
| minikube start | 23.1s    | 21.3s               |
| enable ingress | 28.8s    | 33.2s               |
+----------------+----------+---------------------+

Times for minikube start: 23.0s 21.2s 23.9s 22.8s 24.6s
Times for minikube (PR 19758) start: 20.6s 20.8s 20.9s 21.0s 23.2s

Times for minikube ingress: 22.8s 31.8s 27.8s 22.8s 38.8s
Times for minikube (PR 19758) ingress: 28.9s 30.8s 38.8s 38.8s 28.8s

[spowelljr]

Found a typo, but the tests look good!

[spowelljr]

Suggested change

	return s, fmt.Errorf("timed out waitig for domain %s, current state is %q", method, s)
	return s, fmt.Errorf("timed out waiting for domain %s, current state is %q", method, s)

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: prezha, spowelljr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [prezha,spowelljr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[minikube-pr-bot]

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment	Test Name	Flake Rate
Docker_Linux_containerd_arm64 (1 failed)	TestFunctional/parallel/MountCmd/specific-port(gopogh)	0.00% (chart)
KVM_Linux (1 failed)	TestFunctional/parallel/MountCmd/specific-port(gopogh)	0.00% (chart)

Besides the following environments also have failed tests:

• QEMU_macOS: 156 failed (gopogh)

• Docker_Linux_crio: 2 failed (gopogh)

• Hyperkit_macOS: 34 failed (gopogh)

• Docker_Cloud_Shell: 3 failed (gopogh)

• Docker_Linux_crio_arm64: 2 failed (gopogh)

• KVM_Linux_crio: 10 failed (gopogh)

• Hyper-V_Windows: 13 failed (gopogh)

To see the flake rates of all tests by environment, click here.