Land Model Post-Processing Workflow Summary

[juliabruneau]

The Complete Data Harvesting and Post-Processing Workflow with Examples

From issues: #261 and #280
Example land segment: A51019 , scenario: hsp2_2022, and landuse: for.

• 1. Data Harvesting

• Purpose: exports a data table from a land segment .h5 file as csv
• Uses Generic Script to Export hdf5 data to CSV #538
  • Example: Rscript HARParchive/HARP-2022-Summer/AutomatedScripts/export_hsp_h5.R /opt/model/p53/p532c-sova/output/hspf/land/out/for/hsp2_2022/forA51019.h5 /opt/model/p53/p532c-sova/output/hspf/land/out/for/hsp2_2022/eos/forA51019_pwater.csv /RESULTS/PERLND_P001/PWATER/table
  • Output: forA51019_pwater.csv in /opt/model/p53/p532c-sova/output/hspf/land/out/for/hsp2_2022/eos

---

• 2. Initial Data Summary

• Purpose: data from the csv is read into a data frame, and then 90-day low flow values and 2 groundwater graphs (active groundwater storage and elements of total outflow from the land segment) are exported to VAHydro
• File name: hsp_pwater.R
• Arguments:
  path to hsp_pwater.R
  land_segment
  scenario_name
  landuse
  output_file_path
  image_file_path

Use:

• Rscript HARParchive/HARP-2022-Summer/AutomatedScripts/hsp_pwater.R A51019 hsp2_2022 for /opt/model/p53/p532c-sova/output/hspf/land/out/for/hsp2_2022/eos/forA51019_pwater.csv /media/model/p532/out/land/hsp2_2022/images

• Output in VAHydro:
  • fig.AGWS
  • fig.totalFlowOut
  • l90_Runit
  • l90_agwo_Runit

---

• 3. Further Analysis

• Purpose: Further trend analysis is performed on active groundwater storage and 3 figures and several regression variables are exported to VAHydro
• File name: hsp_pwater_stats.R
• Arguments:
  path to hsp_pwater_stats.R
  land_segment
  scenario_name
  landuse
  output_file_path
  image_file_path

Use:

• Rscript HARParchive/HARP-2022-Summer/AutomatedScripts/hsp_pwater_stats.R A51019 hsp2_2022 for /opt/model/p53/p532csova/output/hspf/land/out/for/hsp2_2022/eos/forA51019_pwater.csv /media/model/p532/out/land/hsp2_2022/images

Output in VAHydro:

• AGWS25perc - container
  • fig.AGWS25perc
  • rsquared_25th
  • slope_25th
  • p_25th
• AGWSmedian - container
  • fig.AGWSmedian
  • rsquared_med
  • slope_med
  • p_med
• fig.AGWSdecomp

[juliabruneau]

Update on data harvesting:

• A new script: detect_data_source.R is added to the workflow
• It detects if the landuse is impervious or pervious, and it can also be used to detect river segments with a HYDR table
  • Impervious land uses: afo, cfo, rid, nid & cid (IWATER)

Use:

• Rscript ~/HARParchive/HARP-2022-Summer/AutomatedScripts/detect_data_source.R /opt/model/p53/p532c-sova/output/hspf/land/out/for/hsp2_2022/forA51019.h5 /opt/model/p53/p532c-sova/output/hspf/land/out/for/hsp2_2022/eos/forA51019_pwater.csv /RESULTS/PERLND_P001/PWATER
  • Resulting file name and data source options:
    • _pwater.csv and /RESULTS/PERLND_P001/PWATER
    • _iwater.csv and /RESULTS/IMPLND_I001/IWATER
    • _hydr.csv and /RESULTS/RCHRES_R001/HYDR

---

Batch scripts that run the Rscripts on an entire basin:

Example basin: OR1_7700_7980, and scenario: hsp2_2022.

Arguments for both batch scripts:

• scenario_name
• basin

1. Land Segment Data Harvesting

• Exports data table from all land segments in the .h5 as CSVs as either pervious or impervious
• File name: batch_harvest_land.bat

Use:

• bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_harvest_land.bat hsp2_2022 OR1_7700_7980
• in directory: /opt/model/p53/p532c-sova
  • Rscripts used: export_hsp_h5.R and detect_data_source.R
  • Output: $landuse$landseg_iwater.csv or $landuse$landseg_pwater.csv in their respectable file paths

---

2. Pervious Land Segment Data Summary and Analysis

• Analysis is performed from all pervious land uses in all land segments: 90-day low flow, 2 groundwater graphs, trend analysis graphs, and summary statistics, which are all exported into VAHydro
• File name: batch_analysis_pwater.bat

Use:

• bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_analysis_pwater.bat hsp2_2022 OR1_7700_7980
• in directory: /opt/model/p53/p532c-sova
  • Rscripts used: hsp_pwater.R and hsp_pwater_stats.R
  • Output: figures, metrics, ‘containers’, and stats in VAHydro (only pwater)

---

Batch scripts for data harvesting and analyzing river segments can be found in issue: #312 Script to Export river h5s

• batch_harvest_river.bat
• batch_analysis_hydr.bat

[rburghol]

Hey @juliabruneau -- the hard drive was finally filling up, so I finally responded and made the change to where the h5 files get stored after a model run. So, that means that the data harvesting scripts need to look in a slightly different file path:

• New location: /media/model/p532/out/land/h5/[land use]/[scenario]/
• Ex: /media/model/p532/out/land/h5/hyo/hsp2_2022/

[juliabruneau]

Hey @juliabruneau -- the hard drive was finally filling up, so I finally responded and made the change to where the h5 files get stored after a model run. So, that means that the data harvesting scripts need to look in a slightly different file path:

• New location: /media/model/p532/out/land/h5/[land use]/[scenario]/
• Ex: /media/model/p532/out/land/h5/hyo/hsp2_2022/

I used the already working batch_harvest.bat script to check that the new path works & received this error:

Error in H5Fopen(h5_file_path) :
  HDF5. File accessibility. Unable to open file.
Execution halted
pwater csv created

Permission issue?

[rburghol]

That looks like a path issue, I don't think you ned to be able to write to the h5 in order to read it, but I will check that for sure. Can you give me the full command with paths, etc?

[juliabruneau]

This is what I used:

juliasb@deq2:/opt/model/p53/p532c-sova$ bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_harvest.bat hsp2_2022 OR1_7700_7980

[rburghol]

I've tried to reset the permiossions uin case they weren't already done (even tho I did them yesterday). No luck.

Here are some observations:

• I added a print to show the contents of h5_file_path in batch_harvest.bat, and that path seems OK, but I also saw this error coming from somewhere:
  • Error: no such file ":HJ5 file path: /media/model/p532/out/land/h5/cex/hsp2_2022/cexA51011.h5"
• So something in our scripts has a hard-wired path, because it looks to me like you've gotten everything in batch_harvest.bat to use the new location info.

[rburghol]

@juliabruneau - I just verified that I can run the following commands manually from your tree (i.e. using your permissions), and they both work.

• Rscript ~/HARParchive/HARP-2022-Summer/AutomatedScripts/detect_data_source.R /media/model/p532/out/land/h5/for/hsp2_2022/forA51011.h5 $data_source_per
  • Returns: 1
• Rscript ~/HARParchive/HARP-2022-Summer/AutomatedScripts/export_hsp_h5.R /media/model/p532/out/land/h5/for/hsp2_2022/forA51011.h5 ~/test_pwater.csv $data_source_per"/table"
  • You can see all the data if you nano ~/test_pwater.csv

So, there must be:

• something wrong in the batch scripts parameters that it is sending to one of the scripts,
• OR, maybe there might just be some landuse/segments that didn't get run properly because of the hard drive filling yesterday? As in, many or most of the landseg/landuse combinations run OK, but a handful are failing. Check to see if the CSV files are created for some and not for others.
• Or, there is something else I don't get...

FYI I added: suppressPackageStartupMessages() around all the library calls so now the output is waaaaay more sane to try to debug. Now, it should be apparent if there are some that are bad, but some that complete OK.

[juliabruneau]

Batch Script to Harvest Land Segment CSVs

This is the updated batch script: batch_harvest_land.bat, which generates iwater and pwater files and inputs them in their respective directories:

• /media/model/p532/out/land/hsp2_2022/iwater
• /media/model/p532/out/land/hsp2_2022/pwater

Use: bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_harvest_land.bat hsp2_2022 OR1_7700_7980 in directory: /opt/model/p53/p532c-sova

The iwater.csv's in their folder:

juliasb@deq2:/media/model/p532/out/land/hsp2_2022/iwater$ ls -l
total 220220
-rw-rw-r-- 1 juliasb allmodelers 22031839 Jul 29 10:33 afoA51011_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22650664 Jul 29 10:41 afoA51037_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22031839 Jul 29 10:34 cfoA51011_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22650664 Jul 29 10:42 cfoA51037_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22513393 Jul 29 10:34 cidA51011_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22861417 Jul 29 10:42 cidA51037_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22513393 Jul 29 10:38 nidA51011_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22861417 Jul 29 10:45 nidA51037_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22513393 Jul 29 10:40 ridA51011_iwater.csv
-rw-rw-r-- 1 juliasb allmodelers 22861417 Jul 29 10:47 ridA51037_iwater.csv

The new batch script to harvest river segment files is in issue #312 .

[glenncampagna]

Summary of Use for batch harvesting and pwater analysis scripts

batch_harvest_land.bat
From directory: /opt/model/p53/p532c-sova
Use: batch_harvest_land.bat [scenario] [river seg]
Example: bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_harvest_land.bat hsp2_2022 OR1_7700_7980
Calls: detect_data_source.R , export_hsp_h5.R
Output: pwater and iwater csv files located at /media/model/p532/out/land/hsp2_2022/(pwater or iwater)

batch_harvest_river.bat
From directory: /opt/model/p53/p532c-sova
Use: batch_harvest_river.bat [scenario] [river seg]
Example: bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_harvest_river.bat hsp2_2022 OR1_7700_7980
Calls: export_hsp_h5.R
Output: hydr csv file located at /media/model/p532/out/river/hsp2_2022/hydr

batch_analysis_pwater.bat
From directory: /opt/model/p53/p532c-sova
Use: batch_analysis_pwater.bat [scenario] [river seg]
Example: bash ~/HARParchive/HARP-2022-Summer/AutomatedScripts/batch_analysis_pwater.bat hsp2_2022 OR1_7700_7980
Calls: hsp_pwater.R , hsp_pwater_stats.R
Output: groundwater figures and 90-day low flow metrics exported to VAhydro (1 of 2 land sets for example river seg:)
http://deq1.bse.vt.edu/d.dh/om-model-info/6828216

[rburghol]

Analyze the data for forest land use, median AGWC levels

select f.name, model.pid as model_pid, forest.pid as fpid, scen.propname as scenario, 
  fora_slope.propvalue as slope, fora_p.propvalue as pvalue
from dh_properties as fora_med 
left outer join dh_properties as fora_slope
on (
  fora_med.pid = fora_slope.featureid
  and fora_slope.propname = 'slope_med'
  and fora_slope.entity_type = 'dh_properties'
)
left outer join dh_properties as fora_p
on (
  fora_med.pid = fora_p.featureid
  and fora_p.propname = 'p_med'
  and fora_p.entity_type = 'dh_properties'
)
left outer join dh_properties as forest
on (
  forest.pid = fora_med.featureid
  and forest.propname = 'for'
  and forest.entity_type = 'dh_properties'
)
left outer join dh_properties as scen
on (
  scen.pid = forest.featureid
  and scen.entity_type = 'dh_properties'
)
left outer join dh_properties as model
on (
  model.pid = scen.featureid
)
left outer join dh_feature as f
on (
  f.hydroid = model.featureid
)
where fora_med.propname = 'AGWSmedian'
and forest.pid is not null
;