Merge pull request #209 from ktehranchi/master

Add Regional Policy Constraints, OASIS Natural Gas Data, Fix Validation Plots

.gitignore

@@ -268,3 +268,4 @@ connect.sh
 config/config.cluster.yaml
 /workflow/repo_data/dag.png
 !.pre-commit-config.yaml
+

docs/source/config-configuration.md

@@ -19,9 +19,11 @@ Most users will leave this as `pypsa-usa`.
 | Renewable Time-Series | Atlite | WECC-ADS |
 | Hydro Time-Series | Breakthrough (temp) | WECC-ADS |
 | Demand | EIA930 | WECC-ADS |
-| Years Supported | 2019 (soon 2017-2023) | 2032 |
-| Interconnections Supported | WECC (soon US) | WECC |
-| Cost Projections | NREL-ATB | NREL-ATB |
+| Historical Demand | 2019-2023 | - |
+| Future Demand | NREL EFS | WECC ADS 2032 |
+| Interconnections Supported | WECC + ERCOT | WECC |
+| Capital Costs | NREL-ATB | NREL-ATB |
+| Fuel Costs | CAISO/EIA | CAISO/EIA |
 | Purpose[^+] | CEM, PCS | PCS |
 
 [^+]: CEM = Capacity Expansion Model, PCS = Production Cost Simulation

docs/source/config-wildcards.md

@@ -1,10 +1,6 @@
 (wildcards)=
 # Wildcards
 
-For more detailed definitions of wildcards, please reference [pypsa-eur](https://pypsa-eur.readthedocs.io/en/latest/wildcards.html).
-Not all wildcards implemented in [pypsa-eur](https://pypsa-eur.readthedocs.io/en/latest/wildcards.html) are available
-yet in pypsa-usa.
-
 It is easy to run PyPSA-USA for multiple scenarios using the wildcards feature of `snakemake`.
 Wildcards generalise a rule to produce all files that follow a regular expression pattern
 which e.g. defines one particular scenario. One can think of a wildcard as a parameter that shows
@@ -47,13 +43,6 @@ during the workflow. If other years are needed, the user will need to prepaer th
 cutout themself.
 ```
 
-<!-- (technology)=
-## The ``{technology}`` wildcard -->
-
-<!-- The ``{technology}`` wildcard specifies for which renewable energy technology to produce availability time
-series and potentials using the rule :mod:`build_renewable_profiles`.
-It can take the values ``onwind``, ``offwind-ac``, ``offwind-dc``, and ``solar`` but **not** ``hydro``
-(since hydroelectric plant profiles are created by a different rule). -->
 
 <!-- (simpl)=
 ## The ``{simpl}`` wildcard -->

docs/source/configtables/opts.csv

@@ -2,11 +2,13 @@ Trigger, Description, Definition, Status,
 ``nH``; i.e. ``2H``-``6H``, Resample the time-resolution by averaging over every ``n`` snapshots, ``prepare_network``: `average_every_nhours() <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L110>`_ and its `caller <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L146>`__), In active use,
 ``nSEG``; e.g. ``4380SEG``," ""Apply time series segmentation with `tsam <https://tsam.readthedocs.io/en/latest/index.html>`_ package to ``n`` adjacent snapshots of varying lengths based on capacity factors of varying renewables"," hydro inflow and load.""", ``prepare_network``: apply_time_segmentation(), In active use
 ``Co2L``, Add an overall absolute carbon-dioxide emissions limit configured in ``electricity: co2limit``. If a float is appended an overall emission limit relative to the emission level given in ``electricity: co2base`` is added (e.g. ``Co2L0.05`` limits emissisions to 5% of what is given in ``electricity: co2base``), ``prepare_network``: `add_co2limit() <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L19>`_ and its `caller <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L154>`__, In active use,
+``RCo2L``, Add regional carbon-dioxide emissions limits configured in ``electricity: regional_Co2_limits``. These can be specified in the file linked at ``electricity: regional_Co2_limits`` in the configuration. File defaults to ``repo_data/regional_co2_limits.csv``., ``prepare_network``: `add_co2limit() <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L19>`_ and its `caller <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L154>`__, In active use,
+``RPS``,"Add Renewable Portfolio Standard (RPS) constraints to shares of generator production (MWh) per carrier for individual countries. Standards can be set for single carriers or groups of carriers. Each constraint can be designated for a specified planning horizon in multi-period models. Opts and path for portfolio_standards.csv must be defined", ``solve_network``, In active use,
 ``Ep``, Add cost for a carbon-dioxide price configured in ``costs: emission_prices: co2`` to ``marginal_cost`` of generators (other emission types listed in ``network.carriers`` possible as well), ``prepare_network``: `add_emission_prices() <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L24>`_ and its `caller <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/prepare_network.py#L158>`__, In active use,
-``CCL``, Add minimum and maximum levels of generator nominal capacity per carrier for individual regions. These can be specified in the file linked at ``electricity: agg_p_nom_limits`` in the configuration. File defaults to ``data/agg_p_nom_minmax.csv``., ``solve_network``, In active use,
-``EQ``," ""Require each region or node to on average produce a minimal share of its total consumption itself. Example: ``EQ0.5c`` demands each region to produce on average at least 50% of its consumption; ``EQ0.5`` demands each node to produce on average at least 50% of its consumption.""", ``solve_network``, In active use,
-``BAU``, Add a per-``carrier`` minimal overall capacity; i.e. at least ``40GW`` of ``OCGT`` in a given Interconnect; configured in ``electricity: BAU_mincapacities``, ``solve_network``: `add_opts_constraints() <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/solve_network.py#L66>`__, Untested,
-``SAFE``, Add a capacity reserve margin (a.k.a Planning Reserve Margin) of a certain fraction above the peak demand to which renewable generators and storage do *not* contribute. Ignores network constraints., ``solve_network`` `add_opts_constraints() <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/solve_network.py#L73>`__, Untested,
+``CCL``,"Add minimum or maximum levels of generator nominal capacity per carrier for individual countries. Each constraint can be designated for a specified planning horizon in multi-period models. Opts and path for agg_p_nom_minmax.csv must be defined", ``solve_network``, In active use,
+``EQ``," ""Require each country or node to on average produce a minimal share of its total consumption itself. Example: ``EQ0.5c`` demands each country to produce on average at least 50% of its consumption; ``EQ0.5`` demands each node to produce on average at least 50% of its consumption.""", ``solve_network``, In active use,
+``ATK``," ""Require each node to be autarkic. Example: ``ATK`` removes all lines and links. ``ATKc`` removes all cross-border lines and links.""", ``prepare_network``, In active use,
+``BAU``, Add a per-``carrier`` minimal overall capacity; i.e. at least ``40GW`` of ``OCGT`` in Europe; configured in ``electricity: BAU_mincapacities``, ``solve_network``: `add_opts_constraints() <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/solve_network.py#L66>`__, Untested,
+``SAFE``, Add a capacity reserve margin of a certain fraction above the peak demand to which renewable generators and storage do *not* contribute. Ignores network., ``solve_network`` `add_opts_constraints() <https://github.com/PyPSA/pypsa-eur/blob/6b964540ed39d44079cdabddee8333f486d0cd63/scripts/solve_network.py#L73>`__, Untested,
 ``carrier+{c|p|m}factor``,"Alter the capital cost (``c``), installable potential (``p``) or marginal costs (``m``) of a carrier by a factor. Example: ``solar+c0.5`` reduces the capital cost of solar to 50\% of original values.", ``prepare_network``, In active use,
 ``CH4L``,"Add an overall absolute gas limit. If configured in ``electricity: gaslimit`` it is given in MWh thermal, if a float is appended, the overall gaslimit is assumed to be given in TWh thermal (e.g. ``CH4L200`` limits gas dispatch to 200 TWh termal)", ``prepare_network``: ``add_gaslimit()``, In active use,
-``ATK``," ""Require each node to be autarkic. Example: ``ATK`` removes all lines and links. ``ATKc`` removes all cross-border lines and links.""", ``prepare_network``, In active use,

docs/source/rules-build-network.md

@@ -1,24 +1,36 @@
 (build-rules)=
 # Build Network
 
-(shapes)=
-## Rule `build_shapes`
-```{eval-rst}
-.. automodule:: build_shapes
-```
-
 (base)=
 ## Rule `build_base_network`
 ```{eval-rst}
 .. automodule:: build_base_network
 ```
 
+(shapes)=
+## Rule `build_shapes`
+```{eval-rst}
+.. automodule:: build_shapes
+```
+
 (busregions)=
 ## Rule `build_bus_regions`
 ```{eval-rst}
 .. automodule:: build_bus_regions
 ```
 
+(demand)=
+## Rule `build_demand`
+```{eval-rst}
+.. automodule:: build_demand
+```
+
+(fuel_prices)=
+## Rule `build_fuel_prices`
+```{eval-rst}
+.. automodule:: build_fuel_prices
+```
+
 (cutout)=
 ## Rule `build_cutout`
 ```{eval-rst}

docs/source/rules-retrieving-data.md

@@ -75,8 +75,9 @@ Natural Gas infrastructure includes:
 
 (retrieve-eia)=
 ## Rule `retrieve_eia_data`
-
-Historical electrical load data from 2015 till the last present month are retrieved from the [US Energy Information Agency](https://www.eia.gov/) (EIA). Data is downloaded at hourly temporal resolution and at a spatial resolution of balancing authority region.
+```{eval-rst}
+.. automodule:: retrieve_eia_data
+```
 
 (retrieve-wecc)=
 ## Rule `retrieve_WECC_forcast_data`
@@ -151,3 +152,10 @@ Documentation of the configuration file ``config/config.yaml`` at
 **Outputs**
 
 - ``resources/costs.csv``
+
+
+(retrieve-caiso-data)=
+## Rule `retrieve_caiso_data`
+```{eval-rst}
+.. automodule:: retrieve_caiso_data
+```

docs/source/rules-solve.md

@@ -3,8 +3,6 @@
 
 (prepare)=
 ## Rule `prepare_network`
-**Directly uses PyPSA-Eur Implementation**
-
 ```{eval-rst}
 Prepare PyPSA network for solving according to :ref:`opts` and :ref:`ll`, such
 as.

workflow/Snakefile

@@ -90,7 +90,7 @@ include: "rules/build_electricity.smk"
 include: "rules/build_sector.smk"
 include: "rules/solve_electricity.smk"
 include: "rules/postprocess.smk"
-
+include: "rules/validate.smk"
 
 if "E" not in config["scenario"]["sector"]:
     if not config["scenario"]["sector"]:

workflow/config/config.default.yaml

@@ -11,8 +11,8 @@ run:
 # docs :
 scenario:
   interconnect: western #"usa|texas|western|eastern"
-  clusters: [30]
-  opts: [Co2L0.75-4H]
+  clusters: [100]
+  opts: [RCo2L-SAFER-CCL]
   ll: [v1.15]
   scope: "total" # "urban", "rural", or "total"
   sector: "" # G
@@ -73,7 +73,11 @@ electricity:
   co2base: 226.86e+6 #base_from_2020 Locations of the 250 MMmt of CO2 emissions from the WECC 2021.
   gaslimit: false # global gas usage limit of X MWh_th
   retirement: economic # "economic" or "technical"
-  SAFE_reservemargin: 0.15
+  SAFE_reservemargin: 0.14
+  regional_Co2_limits: 'config/policy_constraints/regional_Co2_limits.csv'
+  agg_p_nom_limits: 'config/policy_constraints/agg_p_nom_minmax.csv'  
+  portfolio_standards: 'config/policy_constraints/portfolio_standards.csv'
+  SAFE_regional_reservemargins: 'config/policy_constraints/SAFE_regional_prm.csv'
 
   operational_reserve:
     activate: false
@@ -219,7 +223,8 @@ load:
 costs:  # based on the potentials, assuming  (0.1 kW/m2 and 10 m2/person)
   year: 2030
   version: v0.6.0
-  rooftop_share: 0.14
+  rooftop_share: 0.0
+  ng_fuel_year: 2019 # year of the natural gas price from CAISO [2019- 2023]
   fill_values:
     FOM: 0
     VOM: 0

workflow/config/policy_constraints/SAFE_regional_prm.csv

@@ -0,0 +1,12 @@
+region, prm 
+California, 0.14 
+Oregon, 0.14 
+Washington, 0.14 
+Colorado, 0.10 
+New Mexico, 0.14 
+Arizona, 0.14 
+Utah, 0.19 
+Wyoming, 0.14 
+Idaho, 0.14 
+Texas, 0.14 
+Montana, 0.14 

workflow/config/policy_constraints/agg_p_nom_minmax.csv

@@ -0,0 +1,2 @@
+planning_horizon,region,carrier,min,max
+2030,California,onwind,0,

workflow/config/policy_constraints/portfolio_standards.csv

@@ -0,0 +1,15 @@
+planning_horizon,region,carrier,pct
+2030,California,"onwind, solar, offwind_floating, geothermal, offwind, hydro",0.6
+2030,Arizona,"onwind, solar, offwind_floating, geothermal, offwind, hydro",0.15
+2030,Colorado,"onwind, solar, offwind_floating, geothermal, offwind, hydro",0.2395
+2030,New Mexico,"onwind, solar, offwind_floating, geothermal, offwind, hydro",0.5
+2030,Nevada,"onwind, solar, offwind_floating, geothermal, offwind, hydro",0.5
+2030,Oregon,"onwind, solar, offwind_floating, geothermal, offwind, hydro",0.86
+2030,Washington,"onwind, solar, offwind_floating, geothermal, offwind, hydro",0.8
+2040,Arizona,"onwind, solar, offwind_floating, geothermal, offwind, hydro",0.15
+2040,California,"onwind, solar, offwind_floating, geothermal, offwind, hydro",0.95
+2040,Colorado,"onwind, solar, offwind_floating, geothermal, offwind, hydro",0.84
+2040,New Mexico,"onwind, solar, offwind_floating, geothermal, offwind, hydro",0.5
+2040,Nevada,"onwind, solar, offwind_floating, geothermal, offwind, hydro",0.5
+2040,Oregon,"onwind, solar, offwind_floating, geothermal, offwind, hydro",0.93
+2040,Washington,"onwind, solar, offwind_floating, geothermal, offwind, hydro",1

workflow/config/policy_constraints/regional_Co2_limits.csv

@@ -0,0 +1,3 @@
+region,planning_horizon,limit,import_emissions_factor
+California,2030,30000000,0.428
+California,2040,18000000,0.428

workflow/config/tests/config.test.yaml

@@ -76,7 +76,12 @@ electricity:
   co2base: 226.86e+6 #base_from_2020 Locations of the 250 MMmt of CO2 emissions from the WECC 2021.
   gaslimit: false # global gas usage limit of X MWh_th
   retirement: economic # "economic" or "technical"
-  SAFE_reservemargin: 0.15
+  SAFE_reservemargin: 0.14
+  regional_Co2_limits: 'config/policy_constraints/regional_Co2_limits.csv'
+  agg_p_nom_limits: 'config/policy_constraints/agg_p_nom_minmax.csv'  
+  portfolio_standards: 'config/policy_constraints/portfolio_standards.csv'
+  SAFE_regional_reservemargins: 'config/policy_constraints/SAFE_regional_prm.csv'
+
 
   operational_reserve:
     activate: false
@@ -222,7 +227,8 @@ load:
 costs:  # based on the potentials, assuming  (0.1 kW/m2 and 10 m2/person)
   year: 2030
   version: v0.6.0
-  rooftop_share: 0.14
+  rooftop_share: 0.0
+  ng_fuel_year: 2019 # year of the natural gas price from CAISO [2019- 2023]
   fill_values:
     FOM: 0
     VOM: 0

workflow/config/tests/config.test_simple.yaml

@@ -12,7 +12,7 @@ run:
 scenario:
   interconnect: [western] #"usa|texas|western|eastern"
   clusters: [40]
-  opts: [Co2L0.30-3H-Ep-SAFE]
+  opts: [Co2L1.0-6H-RCo2L-SAFER-RPS]
   ll: [v1.0]
   scope: "total" # "urban", "rural", or "total"
   sector: "" # G
@@ -29,7 +29,7 @@ countries: [US]
 
 snapshots:
   start: "2019-01-01"
-  end: "2019-02-01"
+  end: "2019-01-14"
   inclusive: 'left'
 
 # docs :
@@ -72,7 +72,11 @@ electricity:
   co2base: 225.0e+6 #base_from_2020 Locations of the 250 MMmt of CO2 emissions from the WECC 2021.
   gaslimit: false # global gas usage limit of X MWh_th
   retirement: economic # "economic" or "technical"
-  SAFE_reservemargin: 0.15
+  SAFE_reservemargin: 0.14
+  regional_Co2_limits: 'config/policy_constraints/regional_Co2_limits.csv'
+  agg_p_nom_limits: 'config/policy_constraints/agg_p_nom_minmax.csv'  
+  portfolio_standards: 'config/policy_constraints/portfolio_standards.csv'
+  SAFE_regional_reservemargins: 'config/policy_constraints/SAFE_regional_prm.csv'
 
   operational_reserve:
     activate: false
@@ -218,7 +222,8 @@ load:
 costs:  # based on the potentials, assuming  (0.1 kW/m2 and 10 m2/person)
   year: 2030
   version: v0.6.0
-  rooftop_share: 0.14
+  rooftop_share: 0.0
+  ng_fuel_year: 2019 # year of the natural gas price from CAISO [2019- 2023]
   fill_values:
     FOM: 0
     VOM: 0

workflow/config/tests/config.validation.yaml

@@ -10,7 +10,7 @@ run:
 
 # docs :
 scenario:
-  interconnect: [western, texas] #"usa|texas|western|eastern"
+  interconnect: [western] #"usa|texas|western|eastern"
   clusters: [40]
   opts: [Co2L2.0]
   ll: [v1.0]
@@ -73,6 +73,11 @@ electricity:
   co2base: 226.86e+6 #base_from_2020 Locations of the 250 MMmt of CO2 emissions from the WECC 2021.
   gaslimit: false # global gas usage limit of X MWh_th
   retirement: economic # "economic" or "technical"
+  SAFE_reservemargin: 0.14
+  regional_Co2_limits: 'config/policy_constraints/regional_Co2_limits.csv'
+  agg_p_nom_limits: 'config/policy_constraints/agg_p_nom_minmax.csv'  
+  portfolio_standards: '../config/policy_constraints/portfolio_standards.csv'
+  SAFE_regional_reservemargins: 'config/policy_constraints/SAFE_regional_prm.csv'
 
   operational_reserve:
     activate: false
@@ -85,10 +90,10 @@ electricity:
     H2: 168
 
   extendable_carriers:
-    Generator: [] #offwind, offwind_floating,
-    StorageUnit: [] # battery, H2
-    Store: [] #[battery, H2]
-    Link: [] #[H2 pipeline]
+    Generator: []
+    StorageUnit: [] 
+    Store: [] 
+    Link: []
 
   demand: #EFS used for given planning_horizons year
     EFS_case: reference # reference, medium, high
@@ -218,7 +223,8 @@ load:
 costs:  # based on the potentials, assuming  (0.1 kW/m2 and 10 m2/person)
   year: 2030
   version: v0.6.0
-  rooftop_share: 0.14
+  rooftop_share: 0.0
+  ng_fuel_year: 2019 # year of the natural gas price from CAISO [2019- 2023]
   fill_values:
     FOM: 0
     VOM: 0