Handle process externally in ModelicaSystem.simulate and ModelicaSystem.linearized

[kiroorooki]

Hello!
I've noticed that we could run simulations in parallel if we handled the process outside the simulate() method.

It is useful to me because I need to run a log of simulations, and running them in parallel saves me a lot of time.
In order to keep the same functionality, I added a boolean handleProcessOutside, and if set to true, I don't wait() and terminate() the process, I simply return it, so the user can handle the process themselves. That way people can use the library the same way, or get extended functionality if needed with just a boolean.

In my case, I run all my simulations, and I make sure all of them are done later when I save my results (opening .mat file, converting things to csv...).

I just realized the library changed quite a bit, but I believe the same functionality can still be implemented in _run_cmd and propagated to the methods using it (simulate and linearized).

Thanks for your hard work, I hope my suggestion makes sense!

This is how I currently implemented it in my version for reference (quite outdated):
`

def simulate(self, resultfile=None, simflags=None, verbose=True, handleProcessOutside=False):  # 11
    """
    This method simulates model according to the simulation options.
    usage
    >>> simulate()
    >>> simulate(resultfile="a.mat")
    >>> simulate(simflags="-noEventEmit -noRestart -override=e=0.3,g=10) set runtime simulation flags
    """
    if(resultfile is None):
        r=""
        self.resultfile = os.path.join(self.tempdir, self.modelName + "_res.mat").replace("\\", "/")
    else:
        r=" -r=" + resultfile
        self.resultfile = resultfile

    # allow runtime simulation flags from user input
    if(simflags is None):
        simflags=""
    else:
        simflags=" " + simflags

    overrideFile = os.path.join(self.tempdir, '{}.{}'.format(self.modelName + "_override", "txt")).replace("\\", "/")
    if (self.overridevariables or self.simoptionsoverride):
        tmpdict=self.overridevariables.copy()
        tmpdict.update(self.simoptionsoverride)
        # write to override file
        file = open(overrideFile, "w")
        for (key, value) in tmpdict.items():
            name = key + "=" + value + "\n"
            file.write(name)
        file.close()
        override =" -overrideFile=" + overrideFile
    else:
        override =""

    if (self.inputFlag):  # if model has input quantities
        for i in self.inputlist:
            val=self.inputlist[i]
            if(val==None):
                val=[(float(self.simulateOptions["startTime"]), 0.0), (float(self.simulateOptions["stopTime"]), 0.0)]
                self.inputlist[i]=[(float(self.simulateOptions["startTime"]), 0.0), (float(self.simulateOptions["stopTime"]), 0.0)]
            if float(self.simulateOptions["startTime"]) != val[0][0]:
                print("!!! startTime not matched for Input ",i)
                return
            if float(self.simulateOptions["stopTime"]) != val[-1][0]:
                print("!!! stopTime not matched for Input ",i)
                return
            if val[0][0] < float(self.simulateOptions["startTime"]):
                print('Input time value is less than simulation startTime for inputs', i)
                return
        self.createCSVData()  # create csv file
        csvinput=" -csvInput=" + self.csvFile
    else:
        csvinput=""

    if (platform.system() == "Windows"):
        getExeFile = os.path.join(self.tempdir, '{}.{}'.format(self.modelName, "exe")).replace("\\", "/")
    else:
        getExeFile = os.path.join(self.tempdir, self.modelName).replace("\\", "/")
    currentDir = os.getcwd()
    if (os.path.exists(getExeFile)):
        cmd = getExeFile + override + csvinput + r + simflags
        #print(cmd)
        os.chdir(self.tempdir)
        if (platform.system() == "Windows"):
            omhome = os.path.join(os.environ.get("OPENMODELICAHOME"))
            dllPath = os.path.join(omhome, "bin").replace("\\", "/") + os.pathsep + os.path.join(omhome, "lib/omc").replace("\\", "/") + os.pathsep + os.path.join(omhome, "lib/omc/cpp").replace("\\", "/") +  os.pathsep + os.path.join(omhome, "lib/omc/omsicpp").replace("\\", "/")
            my_env = os.environ.copy()
            my_env["PATH"] = dllPath + os.pathsep + my_env["PATH"]
            if not verbose:
                p = subprocess.Popen(cmd, env=my_env, stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
            else:
                p = subprocess.Popen(cmd, env=my_env)
            if not handleProcessOutside:
                p.wait()
                p.terminate()
        else:
            if not verbose:
                p = subprocess.Popen(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
            else:
                p = subprocess.Popen(cmd)
        os.chdir(currentDir)
        self.simulationFlag = True
        if handleProcessOutside:
            return p
    else:
        raise Exception("Error: Application file path not found: " +  getExeFile)

`

[casella]

@arun3688, @adrpo what do you think?

[arun3688]

@kiroorooki The suggestion looks fine, can you update your OMPython to the latest master and make a PR with your changes

[adrpo]

I guess we could also add some facility to OMPython similar to what we have for Library Testing in parallel. Basically we first build the model, then run it in parallel with the Python Parallel jobs feature.

[adrpo]

The only problem with the suggestion might be conflicting generated files and result file names. If those are different for each parallel simulate command then it should work fine.

[arun3688]

ModelicaSystem() creates a separate working directory for each session, so there should not be a problem in conflicting files with same names. We can use python's Multiprocessing library to achieve parallel simulation

[casella]

This would be very interesting, e.g., to run massively parallel black-box optimization on the cloud.