Failed instances should be allowed to stop and restart

[gjcolombo]

Currently Nexus accepts no attempts to change the state of a Failed instance:

omicron/nexus/src/app/instance.rs

Lines 395 to 416 in ee0aac0

	fn check_runtime_change_allowed(
	&self,
	runtime: &nexus::InstanceRuntimeState,
	) -> Result<(), Error> {
	// Users are allowed to request a start or stop even if the instance is
	// already in the desired state (or moving to it), and we will issue a
	// request to the SA to make the state change in these cases in case the
	// runtime state we saw here was stale. However, users are not allowed
	// to change the state of an instance that's migrating, failed or
	// destroyed.
	let allowed = match runtime.run_state {
	InstanceState::Creating => true,
	InstanceState::Starting => true,
	InstanceState::Running => true,
	InstanceState::Stopping => true,
	InstanceState::Stopped => true,
	InstanceState::Rebooting => true,
	InstanceState::Migrating => false,
	InstanceState::Repairing => false,
	InstanceState::Failed => false,
	InstanceState::Destroyed => false,
	};

There are plenty of reasons an instance could move to the Failed state (e.g. a failure to start the VM in Propolis, a heartbeat failure like those discussed in #2727, etc.). A VM user needs to be able to stop and attempt to restart a failed instance.

(Note that, on the Propolis end, once an instance has failed, it can't be restarted--the Propolis zone needs to be destroyed and recreated.)

[askfongjojo]

One of the things I think will be helpful is to avoid automatically setting instance state to Failed when it doesn't fall into the normal lifecycle state transition. This is especially true when a VM instance is wiped during an unintended propolis zone clean-up (e.g. because of a sled-agent crash and restart, or an orderly shutdown of a sled for maintenance before instance migration can take place). This class of scenarios fit with the condition of a stopped instance because the propolis zone is already gone, unlike failed instances that are subject to the condition mentioned above, i.e.:

on the Propolis end, once an instance has failed, it can't be restarted--the Propolis zone needs to be destroyed and recreated

[hawkw]

#6455 allowed failed instances to be restarted. I'm currently working on allowing them to be stopped as well.