Input files for scenario creation and simulation initialization

[krosenfeld-IDM]

From Open Design Questions section on input files

Jonathan has been decoupling scenario creation from simulation by having the first step create the modeled population (and births and deaths data) as CSV or .npy files. Christopher has been initializing the population de novo on each run. At some point it comes down to whether we choose to think of model initialization as a separate pre-processing step or as part of running the model itself. See here.
(@jonathanhhb original author?)

[krosenfeld-IDM]

@KevinMcCarthyAtIDM response:

I argue for runtime generation of the parameters of a simulation as the "framework/reference design" standard. When running sims representing real spatial networks, there will be times where of course the simplest way to bring in information will be to have data files that are read rather than executed (population, lat, long etc. by node for a real setting, ...). But I don't think the core model framework should expect a certain set of inputs, formatted a certain way; to the extent that reading data files is necessary to set up a complicated simulation, it's the model user's job to write the "exoskeleton" code that pulls that input into the model. This also, I think, philosophically follows from the above on dynamic properties - extending from agents to nodes, if I don't need certain information or certain processes, I'd rather remove them than have to set parameters to zero... and I think requiring input files at all can easily start down the path towards hard requirements on what those input files contain.
Big caveat - in running large suites of simulations, functionality to record what differentiates one sim from another is crucial! So we'll need ways to record that info. Though not requiring input files per sim enables us to construct a more minimal record, with one instance of 'parameters' shared across simulations, and only per-sim records of the things that change from sim to sim.

[jonathanhhb]

I agree that we want to keep things nicely generalized and flexible. I think the part I'm stuck on is exemplified by looking deeply at a non-trivial workflow.

In what I've been doing with EULAs, I start with some parameters (population size, number of nodes, and EULA age cutoff), and then add some algorithms or data for relative node populations and age structure and expected lifespans. From there I create a dataframe with the entire population using columns like node (id), age, and expected_lifespan. Then I split the population into "modeled pop" and "eula pop". The latter gets transformed and ultimately pre-simulated until I get my fits.npy which represents the EULA population over time for each node.

Then I take the modeled population and add all the epi columns. And then initialize some of those perhaps.

It's possible to do that all programmatically each time, but that unnecessarily adds time to each sim. It feels funny to me to not cache the modeled population dataframe as a compressed file and start the sim from that.

(If/when one isn't EULA-gizing, one could probably get the speed so that the model dataframe initialization is similar in perf to loading a saved population dataframe.)

In reality I think we'll support both.

[krosenfeld-IDM]

Yeah - I agree that there are advantages to both 👍 👍 . Thanks for the summary of what you've been doing with the EULAs @jonathanhhb - it's helpful for me to follow the thread.