More crutest and dsc updates to support Volume layer activities. (#1472)

This adds support for crutest to use a provided dsc endpoint to
construct a Volume object.

These changes should be a fix for both issues:
#1451 and
#1457

Moved the existing volume creation steps to a new function, and added
another option on how we can create a Volume. The two previous ways of
creating a volume are not changed (though I changed a log message and
added some warnings). The new code is in taking the dsc provided
endpoint and using that to construct a volume.

Additional dsc changes were made to help provide Volume info.
Renamed things in dsc to better reflect what information they hold.
Specifically, update a bunch of region set comments, as dsc just
controls crucible-downstairs processes, and does not know which ones are
part of what region set.

**New dsc commands**
* get_ds_uuid: Returns the UUID for the given client ID
* all_running: Returns true if all downstairs that dsc knows about are
currently in Running state.
* get_region_count: Returns the total number of regions that dsc knows
about.

**New dsc behavior**
* dsc will now wait on all downstairs starting before taking any
commands. The ability for dsc to answer a request can be used by a test
to confirm that all downstairs had started.
* Add the ability to supply a dsc endpoint to crutest-cli (fix
#1459)

**Other changes**
`tools/test_replay.sh` transitioned to using the new --dsc option, as
that test already required a dsc endpoint and was using a hard coded
default value for it.

`tools/test_restart_repair.sh` was updated to wait for dsc to report
that all downstairs are online after a restart. This avoids a race where
we told dsc to start, and then start crutest, but the downstairs are not
yet online.

All the tests that use dsc will eventually transition to using it to
construct a Volume, but I'm pushing that work to another PR.

---------

Co-authored-by: Alan Hanson <[email protected]>

crutest/src/cli.rs

@@ -25,8 +25,9 @@ pub struct CliAction {
 #[allow(clippy::derive_partial_eq_without_eq)]
 #[derive(Debug, Parser, PartialEq)]
 pub enum DscCommand {
- /// Connect to the default DSC server (http://127.0.0.1:9998)
- Connect,
+ /// IP:Port for a dsc server
+ /// #[clap(long, global = true, default_value = "127.0.0.1:9998", action)]
+ Connect { server: SocketAddr },
  /// Disable random stopping of downstairs
  DisableRandomStop,
  /// Disable auto restart on the given downstairs client ID
@@ -380,7 +381,7 @@ async fn handle_dsc(
 ) {
  if let Some(dsc_client) = dsc_client {
  match dsc_cmd {
- DscCommand::Connect => {
+ DscCommand::Connect { .. } => {
  println!("Already connected");
  }
  DscCommand::DisableRandomStop => {
@@ -444,9 +445,9 @@ async fn handle_dsc(
  println!("Got res: {:?}", res);
  }
  }
- } else if dsc_cmd == DscCommand::Connect {
- let url = "http://127.0.0.1:9998".to_string();
- println!("Connect to {:?}", url);
+ } else if let DscCommand::Connect { server } = dsc_cmd {
+ let url = format!("http://{}", server).to_string();
+ println!("Connecting to {:?}", url);
  let rs = Client::new(&url);
  *dsc_client = Some(rs);
  } else {