[WIP] Agnostic

.github/workflows/testagnostic.yml

@@ -0,0 +1,44 @@
+# agnostic (pyomo released)
+
+name: agnostic tests
+
+on:
+ push:
+ branches: [ main ]
+ pull_request:
+ branches: [ main ]
+
+defaults:
+ run:
+ shell: bash -l {0}
+
+jobs:
+ build:
+
+ runs-on: ubuntu-latest
+
+ steps:
+ - uses: actions/checkout@v3
+ - uses: conda-incubator/setup-miniconda@v2
+ with:
+ activate-environment: test_env
+ python-version: 3.9
+ auto-activate-base: false
+ - name: Install dependencies
+ run: |
+ conda install mpi4py pandas setuptools
+ pip install pyomo xpress
+ pip install numpy
+ python -m pip install amplpy --upgrade
+ python -m amplpy.modules install highs cbc gurobi
+ # license?
+
+ - name: setup the program
+ run: |
+ pip install -e .
+
+ - name: run tests
+ timeout-minutes: 100
+ run: |
+ cd mpisppy/tests
+ python test_agnostic.py 

agnostic.txt

@@ -0,0 +1,138 @@
+newest notes on top
+----------------------------------------
+
+host means mpisppy
+guest means whatever the guest language is (which can even be Pyomo, of course)
+
+
+Jan 3 2024: We are going to require that the guest takes care of
+ bundling and presents the host with "proper" bundles
+ that look to the host like regular scenarios.
+ We have AMPL and Pyomo working as guests and GAMS seems to work OK,
+ except that there are file contention issues that can probably be solved
+ easily (I think they have an example with parallel execution)
+
+-------------------------
+
+
+Aug 27
+
+
+The example is now farmer_agnostic.py and agnostic_cylinders.py (ag.bash)
+in the farmer example directory.
+
+HEY::: who updates W? Let's do it right before the solve so we don't have to care...
+ this is a little dangerous at present because we are being silent when
+ they are not there because of xhatters
+
+
+== nonant communication needs to be on the host side, of course, but
+ then callouts need to get values copied to/from the guest
+
+== we need to enforce the assumption that the nonants match in every way possible
+ between host and guest at all times (e.g. _restore_nonants needs to callout,
+ but _save_nonants does not.
+
+== bundling is left hanging in dangerous ways: e.g. Eobjective in two places
+
+Aug 26
+
+Work proceeds on three fronts:
+
+- Addiing callouts to mpisppy
+- creating the example guest language functions (in Pyomo)
+- creating examples of the guest language funtions in other languages
+
+= note: the callouts are almost always inside the local scenario loop in mpisspy
+
+= note: after the solve we are going update the host model x values so
+it can update the xbars
+
+= The host will update the w's then a callout will send the new values to
+ the guest
+
+= I am not even thinking about extensions...
+
+= For bundling, we need to be able to solve EFs (to state the obvious)
+
+circa Aug 25
+
+- no changes needed in spbase so long as agnostic scenario creator is passed in
+
+- EF will be basically a rewrite because we use blocks
+
+- ? for the phbase constructor: pass cfg that contains Ag or pass Ag?; use the options dict for now
+
+- working on an example, which is farmer_agnostic.py run from the __main__ of agnostic.py
+ (farmer_agnostic is presently run from the utils directory and I have a copy of farmer.py there as well)
+
+
+===============================================================
+Thoughts about AML-agnostic extensions to mpi-sppy
+(started by DLW 18 Dec 2022)
+Just thinking about support for straight PH for now. Bundles are probably the first thing to add.
+
+-1. BTW: both GAMS and AMPL have python APIs.
+
+0. On the Python side each scenario is still a Pyomo "model" (that perhaps has only the nonant Vars)
+with _mpisppy_data and _mpisppy_model attached.
+ - it might result in a little speed-up to cut Pyomo out some day, but we should start with Pyomo, I think
+
+1. Some functions in spbase, most functions in phbase, and maybe some functions in spopt will call this function:
+
+ def callout_agnostic(cfg, name, **kwargs):
+ """ callout for AML-agnostic support
+ Args:
+ cfg (Config): the field "AML_agnostic" might contain a module with callouts
+ name (str): the function name to call
+ kwargs (dict): the keywords args for the callout function
+ Calls:
+ a callout function that presumably has side-effects
+ Returns:
+ True if the callout was done and False if not
+ """
+
+ if cfg.get(AML_agnostic, ifmissing=None) is not None:
+ if not hasattr(cfg.AML_agnostic, name):
+ raise RuntimeError(f"AML-agnostic module is missing function {name}")
+ fct = getattr(cfg.AML_agnostic, name)
+ fct(**kwargs)
+ return True
+ else:
+ return False
+
+ The function that is called by fct(**kwargs) will do the work of
+ interacting with the AML and updating mpi-sppy structures that are
+ passed in as kwargs. Note that cfg is going to need to be attached
+ some some objects that don't presently have it (e.g. SPOpt). Some
+ functions in mpi-sppy will want to return immediately if
+ callout_agnostic returns True (i.e., have the callout function do
+ all the work).
+
+
+2 in spbase:
+ - don't support variable_prob
+ - _set_scense needs a callout
+
+3 The scenario_creator function will be in Python and needs to call an AML scenario creator function
+
+4. In solve_one it might be easiest to just do everything in the callout function. Maybe that will be the
+case for many callouts. But from a maintenance perspective, it would best to have mpi-sppy code
+do as much as possible and the callout Python do as little as possible.
+
+5. Think about Compute_Xbar and Update_W. They probably need to do all their processing then do
+a callout at the end so the AML model can be updated.
+
+=======================================
+Notes about callouts
+
+- There are going to be a lot of them!
+
+- Maybe it would be better to drop the name argument and use
+inspect.stack()[1][3] in callout_agnostic so the name of the cfg field
+is the name of the calling function in mpi-sppy.
+ This would
+ o standardize names,
+ o eliminate some cut-and-paste errors,
+ o make it a little wierd if there were ever two callouts from one function,
+ but not that wierd (just handled with kwarg flag) and two callouts should be rare.

doc/src/agnostic.rst

@@ -0,0 +1,161 @@
+AML Agnosticism
+===============
+
+The mpi-sppy package provides callouts so that algebraic modeling languages
+(AMLs) other than Pyomo can be used. A growing number of AMLs are supported
+as `guest` languages (we refer to mpi-sppy as the `host`).
+
+From the end-user's perspective
+-------------------------------
+
+When mpi-sppy is used for a model developed in an AML for which support
+has been added, the end-user runs the ``mpisppy.agnostic.agnostic_cylinders.py``
+program which serves as a driver that takes command line arguments and
+launches the requested cylinders. The file
+``mpisppy.agnostic.go.bash`` provides examples of a few command lines.
+
+
+From the modeler's perspective
+------------------------------
+
+Assuming support has been added for the desired AML, the modeler supplies
+two files:
+
+- a model file with the model written in the guest AML (AMPL example: ``mpisppy.agnostic.examples.farmer.mod``)
+- a thin model wrapper for the model file written in Python (AMPL example: ``mpisppy.agnostic.examples.farmer_ampl_model.py``). This thin python wrapper is model specific.
+
+There can be a little confusion if there are error messages because
+both files are sometimes refered to as the `model file.`
+
+Most modelers will probably want to import the deterministic guest model into their
+python wrapper for the model and the scenario_creator function in the wrapper
+modifies the stochastic paramaters to have values that depend on the scenario
+name argument to the scenario_creator function.
+
+(An exception is when the guest is in Pyomo, then the wrapper
+file might as well contain the model specification as well so
+there typically is only one file.)
+
+
+From the developers perspective
+-------------------------------
+
+If support has not yet been added for an AML, it is almost easier to
+add support than to write a guest interface for a particular model. To
+add support for a language, you need to write a general guest
+interface in Python for it (see, e.g., ampl_guest.py or
+pyomo_guest.py) and you need to add/edit a few lines in
+``mpisppy.agnostic.agnostic_cylinders.py`` to allow end-users to
+access it.
+
+
+Special Note for developers
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The general-purpose guest interfaces might not be the fastest possible
+for many guest languages because they don't use indexes from the
+original model when updating the objective function. If this is an issue,
+you might want to write a problem-specific module to replace the guest
+interface and the model wrapper with a single module. For an example, see
+``examples.farmer.farmer_xxxx_agnostic``, where xxxx is replaced,
+e.g., by ampl. 
+
+Architecture
+^^^^^^^^^^^^
+The following picture presents the architecture of the files. 
+
+.. image:: images/agnostic_architecture.png
+ :alt: Architecture of the agnostic files
+ :width: 700px
+ :align: center
+
+We note "xxxx" the specific problem, for instance farmer. We note "yyyy" the guest language, for instance "ampl".
+Two methods are presented. Either a method specific to the problem, or a generic method.
+Regardless of the method, the file ``agnostic.py`` and ``xxxx.yyyy`` need to be used. 
+``agnostic.py`` is already implemented and must not be modified as all the files presented above the line "developer".
+``xxxx.yyyy`` is the model in the guest language and must be given by the modeler such as all the files under the line "modeler".
+
+The files ``agnostic_yyyy_cylinders.py`` and ``agnostic_cylinders.py`` are equivalent.
+The file ``xxxx_yyyy_agnostic.py`` for the specific case is split into ``yyyy_guest.py`` and ``xxxx_yyyy_model.py`` for the generic case.
+
+
+It is worth noting that the scenario creator is defined in 3 files.
+It is first defined in the file specific to the problem and the guest language ``xxxx_yyyy_model.py``. At this point it may not return a scenario.
+It is then wrapped in a file only specific to the language ``yyyy_guest.py``. At chich point it returns the dictionary ``gd`` which indludes the scenario.
+Finally the tree structure is attached in ``agnostic.py``.
+
+
+Bundles
+-------
+
+The use of scenario bundles can dramatically improve the performance
+of scenario decomposition algorithms such as PH and APH. Although mpi-sppy
+has facitilites for forming bundles, the mpi-sppy
+``agnostic`` package assumes that bundles will be completely handled
+by the guest. Bundles will be returned by the scenario creator function
+as if they are a scenario. Although it seems sort of like a trick, it is
+really the way bundles are intended to operate so we sometimes refer to 
+`true` bundles, which are used in non-agnostic way as briefly
+described in section :ref:`Pickled-Bundles`.
+
+Overview of Recommended Bundle Practices
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Modify the scenario creator function so that if the scenario name
+starts with the string "scen" it returns a single scenario, but if the
+name starts with "bundle" it returns the full extensive formulation for
+a group of scenarios (i.e. a bundle). We typically number scenarios
+and the scenario or bundle number is at the end of the first
+positional argument for the scenario creator function (i.e. at
+the end of the scenario name).
+
+If the name starts with bundle, the scenario creator function can call
+itself with the proper list of scenario names to get the scenarios
+to form the EF that will be returned. We recommend names for
+bundles such as "bundle_xxx_yyy" where xxx and yyy give the
+first and last scenario number in the bundle.
+You could also pass in a dictionary that maps bundle numbers to lists of
+scenario numbers as a keyword argument to the scenario_creator function
+and then append the bundle number to "bundle" and pass it as the positional
+scenario name argument to the scenario creator function.
+
+Some notes
+^^^^^^^^^^
+
+- The helper function called ``scenario_names_creator`` needs to be co-opted
+to instead create bundle names and the code in the scenario_creator function
+then needs to create its own scenario names for bundles. At the time
+of this writing this results in a major hack being needed in order to
+get bundle information to the names creator in the Pyomo example described
+below. You need to supply a function called ``bundle_hack`` in your python model file that
+does whatever needs to be done to alert the names creator that there
+bundles. The function takes the config object as an argument.
+See ``mpisppy.agnostic.farmer4agnostic.py``
+- There is a heavy bias toward uniform probabilities in the examples and in
+ the mpi-sppy utilities. Scenario probabilities are attached to the scenario
+ as ``_mpisppy_probability`` so if your probabilities are not uniform, you will
+ need to calculate them for each bundle (your EF maker code can do that for you). Note that even if probabilities are uniform for the scenarios, they won't
+ be uniform for the bundles unless you require that the bundle size divides
+ the number of scenarios.
+- There is a similar bias toward two stage problems, which is
+ extreme for the agnostic package. If you have a multi-stage
+ problem, you can make things a lot easier for yourself if you require
+ that the bundles contain all scenarios emanating from each second stage node
+ (e.g., on bundle per some integer number of second stage nodes). This
+ is what is done in (non-agnostic) :ref:`Pickled-Bundles`. The result of this
+ is that your multi-stage problem will look like a two-stage problem to
+ mpi-sppy.
+
+Example
+^^^^^^^
+
+The example ``mpisppy.agnostic.farmer4agnostic.py`` contains example code.
+
+.. Note::
+ In order to get information from the command line about bundles into the
+ ``scenario_names_creator`` the ``bundle_hack`` function is called
+ called by the cylinders driver program very early. For this example,
+ function sets global variables called ``bunsize`` and ``numbuns``.
+
+The script ``mpisppy.agnostic.examples.go.bash`` runs the example (and maybe some
+other examples).

doc/src/index.rst

@@ -30,6 +30,7 @@ MPI is used.
  pickledbundles.rst
  grad_rho.rst
  w_rho.rst
+ agnostic.rst
  admmWrapper.rst
  stoch_admmWrapper.rst
  helper_functions.rst

doc/src/pickledbundles.rst

@@ -1,3 +1,5 @@
+.. _Pickled-Bundles:
+
 Pickled Bundles
 ===============
 
@@ -6,7 +8,10 @@ the bleeding edge. The idea is that bundles are formed and then saved
 as dill pickle files for rapid retrieval. The file
 ``aircond_cylinders.py`` in the aircond example directory provides an
 example. The latter part of the ``allways.bash`` script demonstrates
-how to run it.
+how to run it. (The directory is mpi-sppy.examples.aircond assuming you installed
+into the directory mpi-sppy, but the important file aircondB.py is
+mpisppy.tests.examples.aircondB.py, which contains the scenario creator
+function.)
 
 In the future, we plan to support this concept with higher levels of abstraction.
 

examples/farmer/AMPL/AMPLpy_examples/AAAreadme.txt

@@ -0,0 +1 @@
+Examples from AMPL