experiment.run just hangs if commissioning fails

[Bridenbecker]

From David Kafatan at NYU

• experiment.run just hangs if commissioning fails
  • in this case, sbatch was not part of the path. That caused an error that somehow didn’t get caught by experiment.run

[ZDu-IDM]

Seems like we need to add try/except for subprocess.run call:

FILE: idmtools_platform_slurm/idmtools_platform_slurm/slurm_operations/local_operations.py

    def submit_job(self, item: Union[Experiment, Simulation], **kwargs) -> None:
        """
        Submit a Slurm job.
        Args:
            item: idmtools Experiment or Simulation
            kwargs: keyword arguments used to expand functionality
        Returns:
            None
        """
        if isinstance(item, Experiment):
            working_directory = self.get_directory(item)
            subprocess.run(['bash', 'batch.sh'], stdout=subprocess.PIPE, cwd=str(working_directory))
        elif isinstance(item, Simulation):
            pass
        else:
            raise NotImplementedError(f"Submit job is not implemented on SlurmPlatform.")