v0.2.0: Merge pull request #377 from PyPSA/develop

v0.2.0

.github/ISSUE_TEMPLATE/bug_report.yaml

@@ -9,23 +9,13 @@ body:
 - type: checkboxes
   attributes:
     label: Checklist
-    description: Environment checks
+    description: Code and environment checks
     options:
+    - label: I am using the current [`master`](https://github.com/PyPSA/pypsa-usa/tree/master) branch
+      required: true
     - label: I am running on an up-to-date [`pypsa-usa` environment](https://github.com/PyPSA/pypsa-usa/blob/master/workflow/envs/environment.yaml). Update via `conda env update -f envs/environment.yaml`
       required: true
 
-- type: dropdown
-  attributes:
-    label: Issue exisits on what branch
-    description: Code checks
-    multiple: true
-    options:
-    - Master
-    - Develop
-    - Other
-  validations:
-    required: true
-
 - type: textarea
   attributes:
     label: The Issue

docs/requirements.txt

@@ -3,7 +3,6 @@
 # under `install_requires` in `setup.cfg` is also listed here!
 
 pypsa
-# powerplantmatching>=0.5.5
 atlite>=0.2.9
 dask[distributed]
 matplotlib

docs/source/config-configuration.md

@@ -28,7 +28,7 @@ The `run` section is used for running and storing scenarios with different confi
 
 The `scenario` section is used for setting the wildcards and defining planning horizon settings. All configurations within this section are described in [wildcards](#wildcards) with the exception of planning_horizons and foresight.
 
-Planning horizons determines which year of future demand forecast to use for your planning model. If you leave `planning_horizons:` empty, historical demand will be set according to `snapshots`.
+Planning horizons determines which year(s) of future demand forecast to use for your planning model. To build a multi-investment period model set multiple `planning_horizons:` years. The `foresight:` option specifies whether perfect foresight or myopoic foresight optimization model is developed. In perfect foresight, a monolithic model is developed where all `planning_horizons` specified are optimized at once, e.g. future horizon values of costs and demand are incorporated into decisions made in earlier planning horizons. Myopic optimization solves each planning horizon sequentially, and passes the results forward. Currently only `perfect` foresight is implemented.
 
 ```{eval-rst}
 .. literalinclude:: ../../workflow/repo_data/config/config.default.yaml

docs/source/config-wildcards.md

@@ -76,8 +76,13 @@ The wildcard, in general, consists of two parts:
 The `{opts}` wildcard is used for electricity-only studies. It triggers
 optional constraints, which are activated in either :mod:`prepare_network` or
 the :mod:`solve_network` step. It may hold multiple triggers separated by `-`,
-i.e. `Co2L-3H` contains the `Co2L` trigger and the `3H` switch. There are
-currently:
+i.e. `Co2L-3H` contains the `Co2L` trigger and the `3H` switch.
+
+The REM, SAFER, RPS can be defined using either the reeds zone name 'p##"
+the state code (eg, TX, CA, MT), pypsa-usa interconnect name (western, eastern, texas, usa),
+or nerc region name.
+
+There are currently:
 
 ```{eval-rst}
 .. csv-table::

docs/source/configtables/costs.csv

@@ -1,5 +1,5 @@
 ,Unit,Values,Description
-year,--,YYYY; e.g. ``2030``,"Year for which to retrieve cost assumptions of `resources/costs.csv`."
+social_discount_rate,--,float; e.g. ``0.02``,"Discount rate used for weighting multi-investment objective function values."
 version,--,vX.X.X; e.g. ``v0.5.0``,"Version of `technology-data` repository to use."
 rooftop_share,--,float,"Share of rooftop PV when calculating capital cost of solar (joint rooftop and utility-scale PV)."
 fill_values,--,float,"Default values if not specified for a technology in `resources/costs.csv`."

docs/source/configtables/lines.csv

@@ -5,3 +5,4 @@ s_nom_max,MW,float,Global upper limit for the maximum capacity of each extendabl
 max_extension,MW,float,Upper limit for the extended capacity of each extendable line.
 length_factor,--,float,Correction factor to account for the fact that buses are *not* connected by lines through air-line distance.
 interface_transmission_limits, --, true or false, Activate the Interface Transmission Limits (ITL) zones limits.
+transport_model,--,true or false,Converts model from DC PF to Transport Model by replacing lines with Links. Link p_nom value uses NARIS network. Must be used with n_clusters == minimum # of Clusters.

docs/source/configtables/opts.csv

@@ -2,7 +2,7 @@ Trigger, Description, Definition, Status
 ``nH``; i.e. ``2H``-``6H``, Resample the time-resolution by averaging over every ``n`` snapshots, ``prepare_network``: `average_every_nhours() <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L110>`_ and its `caller <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L146>`__), In active use
 ``nSEG``; e.g. ``4380SEG``, Apply time series segmentation with `tsam <https://tsam.readthedocs.io/en/latest/index.html>`_ package to ``n`` adjacent snapshots of varying lengths based on capacity factors of varying renewables  hydro inflow and load., ``prepare_network``: apply_time_segmentation(),In active use
 ``Co2L``, Add an overall absolute carbon-dioxide emissions limit configured in ``electricity: co2limit``. If a float is appended an overall emission limit relative to the emission level given in ``electricity: co2base`` is added (e.g. ``Co2L0.05`` limits emissisions to 5% of what is given in ``electricity: co2base``), ``prepare_network``: `add_co2limit() <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L19>`_ and its `caller <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L154>`__, In active use
-``RCo2L``, Add regional carbon-dioxide emissions limits configured in ``electricity: regional_Co2_limits``. These can be specified in the file linked at ``electricity: regional_Co2_limits`` in the configuration. File defaults to ``repo_data/regional_co2_limits.csv``., ``prepare_network``: `add_co2limit() <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L19>`_ and its `caller <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L154>`__, In active use
+``REM``, Add regional carbon-dioxide emissions limits configured in ``electricity: regional_Co2_limits``. These can be specified in the file linked at ``electricity: regional_Co2_limits`` in the configuration. File defaults to ``repo_data/regional_co2_limits.csv``., ``prepare_network``: `add_co2limit() <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L19>`_ and its `caller <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L154>`__, In active use
 ``RPS``,Add Renewable Portfolio Standard (RPS) constraints to shares of generator production (MWh) per carrier for individual countries. Standards can be set for single carriers or groups of carriers. Each constraint can be designated for a specified planning horizon in multi-period models. Opts and path for portfolio_standards.csv must be defined, ``solve_network``, In active use
 ``SAFE``, Add a capacity reserve margin of a certain fraction above the peak demand to which renewable generators and storage do *not* contribute. Ignores network., ``solve_network`` `add_opts_constraints() <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/solve_network.py#L73>`__,In active use
 ``SAFER``,Adds Regional Capacity Reserve Margin (SAFE) Constraints for defined region- set to a percentage above peak demand level for which renewables and storage do not contribute to. , ``solve_network``,In active use

docs/source/configtables/scenario.csv

@@ -1,3 +1,3 @@
 ,Unit,Values,Description
-planning_horizons,int,"(2018-2023, 2030, 2040, 2050)","Specifies the year of demand data to use. Historical values will use EIA930 data, Future years will use NREL EFS data."
-foresight,bool,"{true, false}",Not implemented (placeholder)
+planning_horizons,int,"(2018-2023, 2030, 2040, 2050)","Specifies the year of demand data to use. Historical values will use EIA930 data, Future years will use NREL EFS data. Specify multiple planning horizons to build a multi-horizon model."
+foresight,str,"perfect", "Specifies foresight option for multi-horizon optimization. Currently only "perfect" foresight is supported. Myopic foresight will be added in the future."

docs/source/contributing.md

@@ -34,41 +34,8 @@ New issue reports should include information about your programming environment
 (e.g., operating system, Python version) and steps to reproduce the problem.
 Please try also to simplify the reproduction steps to a very minimal example
 that still illustrates the problem you are facing. By removing other factors,
-you help us to identify the root cause of the issue. There are four templates
-set up to help with this process:
+you help us to identify the root cause of the issue.
 
-## Documentation Improvements
-
-You can help improve PyPSA-USA docs by making them more readable and
-coherent, or by adding missing information and correcting mistakes.
-
-PyPSA-USA's documentation uses
-[Sphinx](https://www.sphinx-doc.org/en/master/) as its main documentation
-compiler. This means that the docs are kept in the same repository as the
-project code, and that any documentation update is done in the same way was a
-code contribution. We use Markdown language with
-[MyST](https://myst-parser.readthedocs.io/en/latest/syntax/syntax.html)
-extensions
-
-When working on documentation changes, first install the required dependencies
-in your environment
-
-```bash
-~/pypsa-usa $ pip install sphinx sphinx-book-theme myst-parser
-```
-
-Then compile the documentation on your local machine
-
-```bash
-~/pypsa-usa $ cd docs && make html && cd ..
-```
-
-And use Python's built-in web server for a preview in your web browser
-(`http://localhost:8000`)
-
-```bash
-~/pypsa-usa $ python3 -m http.server --directory 'docs/build/html'
-```
 
 ## Code Contributions
 
@@ -103,7 +70,7 @@ copy of the code under your account on the repository service.
     ~/repositories/pypsa-usa $
     ```
 
-### 4. Implement your changes
+### 3. Implement your changes
 
 - Create a new branch with a name in the form of `issue-###` where `###` is
 the auto assigned issue number from GitHub.
@@ -128,15 +95,48 @@ to new functions, modules and classes.
 
    to record your changes in [git](https://git-scm.com).
 
-### 5. Submit your contribution
+### 4. Submit a Pull Request to the **`develop`** branch
 
-- If everything works fine, push your local branch to
-[GitHub](https://github.com/PyPSA/pypsa-usa) with:
+- If everything works fine, push your local branch to your fork with:
 
     ```bash
     git push -u origin my-feature
     ```
 
 - Go to the web page of your fork and click
-[Create a Pull Request](https://github.com/PyPSA/pypsa-usa/pulls) to
-send your changes for review.
+[Create a Pull Request](https://github.com/PyPSA/pypsa-usa/pulls). Then make sure you are submitting the pull request from your_git/issue-xxx branch to the PyPSA/PyPSA-USA:develop branch.
+
+- Communicate on the github Pull Request page to reconcile any changes to be made!
+
+## Documentation Improvements
+
+You can help improve PyPSA-USA docs by making them more readable and
+coherent, or by adding missing information and correcting mistakes.
+
+PyPSA-USA's documentation uses
+[Sphinx](https://www.sphinx-doc.org/en/master/) as its main documentation
+compiler. This means that the docs are kept in the same repository as the
+project code, and that any documentation update is done in the same way was a
+code contribution. We use Markdown language with
+[MyST](https://myst-parser.readthedocs.io/en/latest/syntax/syntax.html)
+extensions
+
+When working on documentation changes, first install the required dependencies
+in your environment
+
+```bash
+~/pypsa-usa $ pip install sphinx sphinx-book-theme myst-parser
+```
+
+Then compile the documentation on your local machine
+
+```bash
+~/pypsa-usa $ cd docs && make html && cd ..
+```
+
+And use Python's built-in web server for a preview in your web browser
+(`http://localhost:8000`)
+
+```bash
+~/pypsa-usa $ python3 -m http.server --directory 'docs/build/html'
+```

workflow/Snakefile

@@ -10,6 +10,7 @@ HTTP = HTTPRemoteProvider()
 from os.path import normpath
 from itertools import chain
 from pathlib import Path
+import pandas as pd
 
 FIGURES_MAPS = [
     "capacity_map_base.pdf",
@@ -25,22 +26,16 @@ FIGURES_EMISSIONS = [
     "emissions_area.pdf",
     "emissions_accumulated.pdf",
     "emissions_accumulated_tech.pdf",
-    "emissions_area.html",
-    # "emissions_node.html",
-    "emissions_region.html",
-    "emissions_accumulated_tech.html",
     "bar_regional_emissions.pdf",
 ]
 
 FIGURES_PRODUCTION = [
     "production_area.pdf",
-    "production_area.html",
 ]
 
 FIGURES_SYSTEM = [
     "production_bar.pdf",
     "capacity_additions_bar.pdf",
-    "costs_bar.pdf",
     "bar_regional_capacity_additions.pdf",
     "global_constraint_shadow_prices.pdf",
     "generator_data_panel.pdf",
@@ -50,19 +45,30 @@ FIGURES_SYSTEM = [
     "region_lmps.pdf",
 ]
 
-FIGURES_NATURAL_GAS = ["natural_gas.html"]
+FIGURES_NATURAL_GAS = [
+    "natural_gas_demand.html",
+    "natural_gas_processing.html",
+    "natural_gas_linepack.html",
+    "natural_gas_storage.html",
+    "natural_gas_domestic_trade.html",
+    "natural_gas_international_trade.html",
+]
 
 FIGURES_VALIDATE = [
-    "seasonal_stacked_plot.pdf",
+    "daily_stacked_comparison.pdf",
     "carrier_production_bar.pdf",
     "val_bar_regional_emissions.pdf",
     "val_bar_state_emissions.pdf",
     "val_generator_data_panel.pdf",
-    "val_heatmap_capacity_factor.pdf",
+    # "val_heatmap_capacity_factor.pdf",
     "val_box_region_lmps.pdf",
     "val_map_load_shedding.pdf",
     "val_generator_stack.pdf",
     "val_map_line_loading.pdf",
+    "val_state_generation_deviation.pdf",
+    "val_heatmap_state_generation_carrier.pdf",
+    "val_cap_state_generation.pdf",
+    "val_fuel_costs.pdf",
 ]
 
 
@@ -86,8 +92,7 @@ wildcard_constraints:
 
 
 # Merge subworkflow configs and main config
-configfile: "config/config.default.yaml"
-# configfile: "config/tests/config.validation.yaml"
+configfile: "config/config.validation.yaml"
 configfile: "config/config.cluster.yaml"
 configfile: "config/config.common.yaml"
 configfile: "config/config.plotting.yaml"

workflow/envs/environment.yaml

@@ -9,7 +9,7 @@ dependencies:
 
 - pypsa>=0.26.1
 - atlite>=0.2.9
-- linopy
+- linopy==0.3.8
 
 - dask==2023.7.0
 - dask-core==2023.7.0
@@ -20,7 +20,7 @@ dependencies:
 - xlrd==2.0.1
 - openpyxl==3.1.2
 - pycountry==22.3.5
-- seaborn==0.12.2
+- seaborn==0.13.2
 - snakemake-minimal==7.32.4
 - memory_profiler==0.61.0
 - yaml==0.2.5
@@ -35,7 +35,7 @@ dependencies:
 - matplotlib==3.8.0
 - plotly==5.17.0
 - graphviz
-- powerplantmatching
+- duckdb==1.0.0
 
 # Keep in conda environment when calling ipython
 - ipython==8.16.1

workflow/notebooks/Tutorial/tutorial.ipynb

@@ -52,18 +52,18 @@
     "network_c_ec = pypsa.Network(\n",
     "    os.path.join(resources_folder, f\"{interconnect}/elec_s_{num_clusters}_ec.nc\")\n",
     ")\n",
-    "network_presolve = pypsa.Network(\n",
-    "    os.path.join(\n",
-    "        resources_folder,\n",
-    "        f\"{interconnect}/elec_s_{num_clusters}_ec_l{ll}_{opts}_{sector}.nc\",\n",
-    "    )\n",
-    ")\n",
-    "network_solved = pypsa.Network(\n",
-    "    os.path.join(\n",
-    "        results_folder,\n",
-    "        f\"{interconnect}/networks/elec_s_{num_clusters}_ec_l{ll}_{opts}_{sector}.nc\",\n",
-    "    )\n",
-    ")"
+    "# network_presolve = pypsa.Network(\n",
+    "#     os.path.join(\n",
+    "#         resources_folder,\n",
+    "#         f\"{interconnect}/elec_s_{num_clusters}_ec_l{ll}_{opts}_{sector}.nc\",\n",
+    "#     )\n",
+    "# )\n",
+    "# network_solved = pypsa.Network(\n",
+    "#     os.path.join(\n",
+    "#         results_folder,\n",
+    "#         f\"{interconnect}/networks/elec_s_{num_clusters}_ec_l{ll}_{opts}_{sector}.nc\",\n",
+    "#     )\n",
+    "# )"
    ]
   },
   {
@@ -80,7 +80,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "n_base_network"
+    "n_base_network.investment_periods"
    ]
   },
   {
@@ -97,7 +97,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "network_elec"
+    "network_elec.investment_periods"
    ]
   },
   {
@@ -123,7 +123,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "network_s"
+    "network_s.loads_t.p_set"
    ]
   },
   {
@@ -172,6 +172,15 @@
     "network_c_ec"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "network_c_ec.investment_periods"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -257,7 +266,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.9"
   }
  },
  "nbformat": 4,