Preparing test script for tests before master merging

README.md

@@ -1,5 +1,7 @@
 # Balmorel
 
+A scenario that's possible to run with an academic trial license of GAMS can be found in the balmorel-demo branch.
+
 ## What is Balmorel?
 
 Balmorel is a partial equilibrium model for analysing the electricity and combined heat and power sectors in an international perspective. It is highly versatile and may be applied for long range planning as well as shorter time operational analysis. Balmorel is implemented as a mainly linear programming optimisation problem.
@@ -18,4 +20,30 @@ Balmorel is a modelling tool that can be used by energy system experts, energy c
 
 ## How is Balmorel supported and further developed?
 
-The model is developed and distributed under open source ideals. The source code has been provided on its homepage since 2001 and was assigned the [ISC license](https://opensource.org/licenses/ISC) in 2017. Ample documentation is available in the folder [within this repository](base/documentation). Application examples and contact information can be found on the [Balmorel homepage](https://balmorel.com). Presently the model development is mainly project driven, with a users' network around it, supporting the open source development idea.
+The model is developed and distributed under open source ideals. The source code has been provided on its homepage since 2001 and was assigned the [ISC license](https://opensource.org/licenses/ISC) in 2017. Ample documentation is available in the folder [within this repository](base/documentation). We also refer to information on [openmod](https://wiki.openmod-initiative.org/wiki/Balmorel). Presently the model development is mainly project driven, with a users' network around it, supporting the open source development idea.
+
+### Procedure for Merging Changes to Master
+
+The DTU Balmorel team has decided on the following procedure for merging new changes to the master:
+1. A decision to merge should be agreed upon in a Balmorel meeting with the admins
+2. Create a new branch from master, which will be used as a temporary branch to do all the changes
+3. Make the desired changes in a new scenario:
+
+    a. Add a new scenario called 'changes', which result in the following folder structure:
+
+        Balmorel
+        ├── base
+        ├── changes
+        │   └── model
+        │   │   ├── cplex.op4
+        │   │   └── Balmorel.gms
+        ├── simex
+        ├── README.md
+        ├── .gitignore
+        └── .gitattributes
+
+    b. Keep the base scenario identical to the current master and apply all changes to the 'changes' scenario
+
+4. Run the [test script](base/auxils/master_merge_tests/merge_tests.ipynb) and make sure that the model stays feasible and that you verify the changes
+5. Make a pull request and report the changes made with the KPI outputs from the tests in 4. in ... (todo: check where it makes sense to put these descriptions, as simple as possible)
+

base/auxils/master_merge_tests/merge_tests.ipynb

@@ -0,0 +1,72 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Test Runs for Merging Changes\n",
+    "\n",
+    "This notebook is used for testing changes made to the master branch, by running the models in base/model (which should be identical to the master branch) and change/model (which should be the model run with your changes) \n",
+    "\n",
+    "Please follow the procedure described in the [readme](../../../README.md)!\n",
+    "\n",
+    "The [pybalmorel](https://pypi.org/project/pybalmorel/) package is required to run this notebook. In a terminal, create a new python environment, activate it and install version 0.2.2 by running the following command:\n",
+    "```\n",
+    "pip install pybalmorel==0.2.2\n",
+    "```\n",
+    "Documentation on creating and activating virtual environments for standalone python can be found [here](https://docs.python.org/3/library/venv.html) and for a conda installation [here](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pybalmorel import MainResults\n",
+    "from gams import GamsWorkspace\n",
+    "import os\n",
+    "\n",
+    "### 1.1 Running Balmorel\n",
+    "scenarios = ['base', 'changes']\n",
+    "for scenario in scenarios:\n",
+    "    ws = GamsWorkspace(working_directory='../../../%s/model'%scenario)\n",
+    "\n",
+    "    job = ws.add_job_from_file(os.path.join(os.path.abspath('../../../%s/model'%scenario), 'Balmorel'))\n",
+    "    out = job.run()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "### 1.2 Load Results\n",
+    "mr = MainResults(files=['MainResults.gdx']*2,\n",
+    "                 paths=['../../../%s/model'%scenario for scenario in scenarios])"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python (.BAF-Env)",
+   "language": "python",
+   "name": ".baf-env"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

changes/model/Balmorel.gms

@@ -0,0 +1,210 @@
+* File Balmorel.gms
+$TITLE Balmorel version 4.03 (March 2022; latest 20220318)
+
+SCALAR IBALVERSN 'This version of Balmorel' /403.20220318/;
+* Efforts have been made to make a good model.
+* However, most probably the model is incomplete and subject to errors.
+* It is distributed with the idea that it will be usefull anyway,
+* and with the purpose of getting the essential feedback,
+* which in turn will permit the development of improved versions
+* to the benefit of other user.
+* Hopefully it will be applied in that spirit.
+
+* All GAMS code of the Balmorel model is distributed under ICS license,
+* see the license file in the base/model folder.
+
+
+*-------------------------------------------------------------------------------
+*-------------------------------------------------------------------------------
+*-------------------------------------------------------------------------------
+
+
+* This is a preliminary version of Balmorel 3.03.
+* It is intended for review and commenting.
+* See a short list of changes since previous version 3.02 in base/documentation/Balmorel303.txt.
+* It is scheduled that a final version 3.03 will be available early 2017.
+
+*-------------------------------------------------------------------------------
+*-------------------------------------------------------------------------------
+*-------------------------------------------------------------------------------
+
+
+* GAMS options are $included from file balgams.opt.
+* In order to make them apply globally, the option $ONGLOBAL will first be set here:
+$ONGLOBAL
+
+$ifi %system.filesys%==UNIX
+execute 'chmod  -R ug+rw "../.."';
+
+* The balgams file holds control settings for GAMS.
+* Use local file if it exists, otherwise use the one in folder  ../../base/model/.
+$ifi     exist 'balgams.opt'  $include  'balgams.opt';
+$ifi not exist 'balgams.opt'  $include  '../../base/model/balgams.opt';
+
+* The balopt file holds option (control) settings for the Balmorel model.
+* Use local file if it exists, otherwise use the one in folder  ../../base/model/.
+
+
+$ifi     exist 'balopt.opt'  $include                  'balopt.opt';
+$ifi not exist 'balopt.opt'  $include '../../base/model/balopt.opt';
+
+$ifi %system.filesys%==UNIX
+execute 'chmod  -R ug+rw "../.."';
+
+* If merging of savepoint files is to be performed,
+* make sure that there are no gdx files initially in applied folders:
+$ifi %system.filesys%==UNIX
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute "rm *.gdx";
+$ifi %system.filesys%==MSNT
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute "del *.gdx";
+$ifi %system.filesys%==UNIX
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute "rm ../output/temp/*.gdx";
+$ifi %system.filesys%==MSNT
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute "del ..\output\temp\*.gdx";
+
+*-------------------------------------------------------------------------------
+*-------------------------------------------------------------------------------
+*---- Some generally applicable stuff: -----------------------------------------
+*-------------------------------------------------------------------------------
+
+
+* The following may be used in display, put or execute_unload statements (the text, not the value, holds relevant information)
+SCALAR SystemDateTime "%system.date%,  %system.time%" /NA/;
+
+* Displaying only a text string will not be accessible through the 'table of content' on the .lst file, therefore this
+SCALAR INFODISPLAY  "See information next:" /NA/;
+DISPLAY INFODISPLAY, "Balmorel run stared at", SystemDateTime;
+
+* Convenient Factors, typically relating Output and Input:
+SCALAR IOF100     'Multiplier 100'        /100/;
+SCALAR IOF1000    'Multiplier 1000'       /1000/;
+SCALAR IOF1000000 'Multiplier 1000000'    /1000000/;
+SCALAR IOF0001    'Multiplier 0.001'      /0.001/;
+SCALAR IOF0000001 'Multiplier 0.000001'   /0.000001/;
+SCALAR IOF3P6     'Multiplier 3.6'        /3.6/;
+SCALAR IOF24      'Multiplier 24'         /24/;
+SCALAR IOF8760    'Multiplier 8760'       /8760/;
+SCALAR IOF8784    'Multiplier 8784'       /8784/;
+SCALAR IOF365     'Multiplier 365'        /365/;
+SCALAR IOF05      'Multiplier 0.5'        /0.5/;
+SCALAR IOF1_      'Multiplier 1 (used with QOBJ and derivation of marginal values)'   /1/;!! special, possibly to disappear in future versions
+* Scalars for occational use, their meaning will be context dependent:
+SCALAR ISCALAR1   '(Context dependent)';
+SCALAR ISCALAR2   '(Context dependent)';
+SCALAR ISCALAR3   '(Context dependent)';
+SCALAR ISCALAR4   '(Context dependent)';
+SCALAR ISCALAR5   '(Context dependent)';
+
+* Initialisations of printing of log and error files and messages:
+$INCLUDE '../../base/logerror/logerinc/error1.inc';
+
+* Ensuring existence of needed output folders:
+$ifi not dexist "../../simex"            execute 'mkdir -p "../../simex"';
+$ifi not dexist "../logerror/logerinc"   execute 'mkdir -p "../logerror/logerinc"';
+$ifi not dexist "../output/economic"     execute 'mkdir -p "../output/economic"';
+$ifi not dexist "../output/inputout"     execute 'mkdir -p "../output/inputout"';
+$ifi not dexist "../output/printout"     execute 'mkdir -p "../output/printout"';
+$ifi not dexist "../output/temp"         execute 'mkdir -p "../output/temp"';
+
+
+*-------------------------------------------------------------------------------
+*---- End: Some generally applicable stuff -------------------------------------
+*-------------------------------------------------------------------------------
+
+
+
+*-------------------------------------------------------------------------------
+*-------------------------------------------------------------------------------
+* In case of Model Balbase4 the following included file substitutes the remaining part of the present file Balmorel.gms:
+* Use local file if it exists, otherwise use the one in folder  ../../base/model/.
+$ifi %BB4%==yes $ifi     exist 'Balmorelbb4.inc'  $include  'Balmorelbb4.inc';
+$ifi %BB4%==yes $ifi not exist 'Balmorelbb4.inc'  $include  '../../base/model/Balmorelbb4.inc';
+$ifi %BB4%==yes $goto ENDOFMODEL
+*-------------------------------------------------------------------------------
+*-------------------------------------------------------------------------------
+
+*----- End of model:------------------------------------------------------------
+$label ENDOFMODEL
+$ifi %all_endofmodelgdx%==yes execute_unload "all_endofmodel.gdx";
+*----- End of model ------------------------------------------------------------
+
+
+*--- Results collection for this case ------------------------------------------
+
+$ifi not %system.filesys%==MSNT $goto endofMSNToutput
+*The following section until $label endofMSNToutput is related to Windows output only
+*Please use only backslash \ instead of forward slash / in this section until the label
+
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'gdxmerge "%relpathoutput%temp\*.gdx"';
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'move merged.gdx "%relpathoutput%%CASEID%.gdx"';
+
+$ifi %MERGECASE%==NONE
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'gdxmerge "..\output\%CASEID%.gdx"';
+$ifi %MERGECASE%==NONE
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'move merged.gdx "%relpathoutput%%CASEID%-results.gdx"'
+
+$ifi %MERGECASE%==NONE
+$ifi %MERGEDSAVEPOINTRESULTS2MDB%==yes execute '=gdx2access "%relpathoutput%%CASEID%-results.gdx"';
+$ifi %MERGECASE%==NONE
+$ifi %MERGEDSAVEPOINTRESULTS2SQLITE%==yes execute '=gdx2sqlite -i "%relpathoutput%%CASEID%-results.gdx" -o "%relpathoutput%%CASEID%-results.db"';
+
+*--- Results collection and comparison for differents cases --------------------
+
+$ifi not %MERGECASE%==NONE
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'gdxmerge "%relpathoutput%%CASEID%.gdx" "%relpathModel%..\..\%MERGEWITH%/output\%MERGEWITH%.gdx"';
+$ifi not %MERGECASE%==NONE
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'move merged.gdx "%relpathoutput%%CASEID%-resmerged.gdx"';
+
+$ifi not %MERGECASE%==NONE
+$ifi %MERGEDSAVEPOINTRESULTS2MDB%==yes execute '=gdx2access "%relpathoutput%%CASEID%-resmerged.gdx"';
+$ifi not %MERGECASE%==NONE
+$ifi %MERGEDSAVEPOINTRESULTS2SQLITE%==yes execute '=gdx2sqlite -i "%relpathoutput%%CASEID%-resmerged.gdx" -o "%relpathoutput%%CASEID%-resmerged.db"';
+
+$ifi not %DIFFCASE%==NONE
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'gdxdiff "%relpathoutput%%CASEID%-results.gdx" "%relpathModel%..\..\%DIFFWITH%/output/%DIFFWITH%-results.gdx"';
+$ifi not %DIFFCASE%==NONE
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'move diffile.gdx "%relpathoutput%%CASEID%-diff.gdx"';
+
+$label endofMSNToutput
+
+$ifi not %system.filesys%==UNIX $goto endofUNIXoutput
+*The following section until $label endofUNIXoutput is related to UNIX output only
+*Please use only forward slash / instead of backslash \ in this section until the label
+
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'gdxmerge "../output/temp/*.gdx"';
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'mv ./merged.gdx ./"%relpathoutput%%CASEID%.gdx"';
+
+$ifi %MERGECASE%==NONE
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'gdxmerge "../output/%CASEID%.gdx"';
+$ifi %MERGECASE%==NONE
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'mv ./merged.gdx ./"%relpathoutput%%CASEID%-results.gdx"'
+
+$ifi %MERGECASE%==NONE
+$ifi %MERGEDSAVEPOINTRESULTS2MDB%==yes execute '=gdx2access ./"%relpathoutput%%CASEID%-results.gdx"';
+$ifi %MERGECASE%==NONE
+$ifi %MERGEDSAVEPOINTRESULTS2SQLITE%==yes execute '=gdx2sqlite -i ./"%relpathoutput%%CASEID%-results.gdx" -o ./"%relpathoutput%%CASEID%-results.db"';
+
+*--- Results collection and comparison for differents cases --------------------
+
+$ifi not %MERGECASE%==NONE
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'gdxmerge ./"%relpathoutput%%CASEID%.gdx" ./"%relpathModel%../../%MERGEWITH%/output/%MERGEWITH%.gdx"';
+$ifi not %MERGECASE%==NONE
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'mv ./merged.gdx ./"%relpathoutput%%CASEID%-resmerged.gdx"';
+
+$ifi not %MERGECASE%==NONE
+$ifi %MERGEDSAVEPOINTRESULTS2MDB%==yes execute '=gdx2access ./"%relpathoutput%%CASEID%-resmerged.gdx"';
+$ifi not %MERGECASE%==NONE
+$ifi %MERGEDSAVEPOINTRESULTS2SQLITE%==yes execute '=gdx2sqlite -i ./"%relpathoutput%%CASEID%-resmerged.gdx" -o ./"%relpathoutput%%CASEID%-resmerged.db"';
+
+$ifi not %DIFFCASE%==NONE
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'gdxdiff ./"%relpathoutput%%CASEID%-results.gdx" ./"%relpathModel%../../%DIFFWITH%/output/%DIFFWITH%-results.gdx"';
+$ifi not %DIFFCASE%==NONE
+$ifi %MERGESAVEPOINTRESULTS%==yes  execute 'mv ./diffile.gdx ./"%relpathoutput%%CASEID%-diff.gdx"';
+
+
+$label endofUNIXoutput
+
+*----- End of file:------------------------------------------------------------
+$label endoffile
+
+

changes/model/cplex.op4

@@ -0,0 +1,14 @@
+* Option file 4 for CPLEX solver. For possible options, see the GAMS Solver Manual.
+* To use with CPLEX,  set  '$Setglobal USEOPTIONFILE 4' in balgams.opt.
+* Do not change this file, it is predefined.
+* To use other options with CPLEX, make and/or use some other file, see USEOPTIONFILE in balgams.opt.
+* "LPmethod  4": Barrier method
+* "Threads  -1": Use all available threads except 1.
+* If memory tight, set the Threads parameter to 1.
+* advind is set to 0 by default since for large problems a better performance has been experienced. If 0, the optimization will not use the previous solution to try to find the next run faster.
+
+LPmethod  4
+Threads  -1
+advind 0
+
+