Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[1pt] PR: Create inputs for selected HUC(s) #1178

Open
wants to merge 39 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
3abd9a6
Copy inputs and test_cases data for a single HUC
mluck May 17, 2024
06f3c63
Add recurrence flow files
mluck May 20, 2024
bd75364
Add OSM bridges
mluck May 20, 2024
f44d737
Add all recurrence flow files
mluck May 20, 2024
c916d43
Merge branch 'dev' into dev-sample-data
mluck May 21, 2024
841ad04
Add bathymetry adjustment data
mluck May 21, 2024
d36de6c
Add WBD domain
mluck May 21, 2024
e20af02
Merge branch 'dev' into dev-sample-data
mluck May 21, 2024
b65e892
Relocate files
mluck May 21, 2024
fe09fca
Update Dockerfile
mluck May 21, 2024
cf8d3a6
Adjust recurrence intervals
mluck May 22, 2024
28396a0
Make other test_case dirs
mluck May 23, 2024
ef7429d
Add Jupyter file
mluck May 23, 2024
2332b72
Update CHANGELOG.md
mluck May 23, 2024
a74fd65
Minor edits
mluck May 24, 2024
8ea45b4
Add functionality to pull from S3
mluck May 24, 2024
64372ce
Remove nwm_recur_flows
mluck May 28, 2024
3a6bf4e
Upgrade whitebox to 2.3.4
mluck May 29, 2024
1d5f637
Update CHANGELOG.md
mluck May 29, 2024
df33b26
Revert NWM
mluck Jun 13, 2024
91a49cc
Revert bash_variables
mluck Jun 13, 2024
20a5266
Merge branch 'dev-whitebox-2.3.4' into dev-sample-data
Jun 13, 2024
63c782e
Create README.md
mluck Jun 13, 2024
9635839
Update CHANGELOG.md
mluck Jun 13, 2024
db78492
Update README.md
mluck Jun 14, 2024
45f6299
Address leading s3://
mluck Jun 14, 2024
ff80dfa
Merge branch 'dev-sample-data' of https://github.com/NOAA-OWP/inundat…
mluck Jun 14, 2024
7af409d
Merge branch 'dev' into dev-sample-data
mluck Nov 1, 2024
de4a91a
Merge branch 'dev' into dev-sample-data
mluck Nov 6, 2024
11083b7
use paths in environment variables
mluck Nov 7, 2024
91d0aed
Fix IFC and WBD layer issues
mluck Nov 10, 2024
4e6ae2e
Clean up code
mluck Nov 10, 2024
facca2e
Archive FIM in a Box
mluck Nov 10, 2024
3f6d2cd
Update Changelog and minor edits
mluck Nov 12, 2024
700cd98
Copy WBD national
mluck Nov 13, 2024
941525b
Fix S3 downloads
mluck Nov 19, 2024
3c1f2d5
Use latest bridges data
mluck Nov 20, 2024
c320879
Update README.md with instructions on running get_sample_data.py
mluck Nov 22, 2024
0787cd3
Update README.md
mluck Nov 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 19 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,6 @@ The available inputs, test cases, and versioned FIM outputs can be found by runn
aws s3 ls s3://noaa-nws-owp-fim/hand_fim/ --profile esip
```

Download a directory of sample outputs for a single HUC8:
```
aws s3 sync s3://noaa-nws-owp-fim/hand_fim/outputs/hand_4_5_2_11/12090301 \
/your_local_folder_name/12090301 --profile esip
```
By adjusting pathing, you can also download entire directories such as the `hand_4_5_2_11` folder. An entire output HAND set is approximately 1.7 TB.

**Note**: There may be newer editions than `hand_4_5_11_1`, and it is recommended to adjust the command above for the latest version.
Expand Down Expand Up @@ -138,6 +133,25 @@ docker run --rm -it --name Robs_container \
fim_4:dev_20230620
```

### Subsetting input data
A subset of the data required to run and evaluate FIM can be obtained with the use of ESIP AWS keys. In order to generate these data:
1. Start a Docker container as in the previous step
2. Run `/foss_fim/data/get_sample_data.py` replacing `<aws_access_key_id>` and `<aws_secret_access_key>` with your AWS access keys. To generate data for HUC 03100204, for example:
```
python /foss_fim/data/get_sample_data.py -u 03100204 -i s3://noaa-nws-owp-fim/hand_fim/ -o /outputs/sample-data -r hand_fim -s3 -ak <aws_access_key_id> -sk <aws_secret_access_key>
```
3. Exit the Docker container by typing `exit`. Alternatively, you can leave this container running and run the next command in a new terminal tab or window.
4. Start a new Docker container with the `/data` volume mount pointing at the local output location (`-o`) used in `get_sample_data.py` (step 2). For example:
```bash
docker run --rm -it --name Robs_data_container \
-v /home/projects/fim/code/inundation-mapping/:/foss_fim \
-v /home/projects/fim/data/outputs/:/outputs \
-v /home/projects/fim/data/outputs_temp/:/fim_temp \
-v /home/projects/fim/data/outputs/sample-data:/data \
fim_4:dev_20230620
```
5. Now you can run the following commands with the sample data.

### Produce HAND Hydrofabric
```
fim_pipeline.sh -u <huc8> -n <name_your_run> -o
Expand All @@ -161,15 +175,6 @@ The three sections are:

Running the `fim_pipeline.sh` is a quicker process than running all three steps independently, but you can run some sections more than once if you like.

### Testing in Other HUCs
To test in HUCs other than the provided HUCs, the following processes can be followed to acquire and preprocess additional NHDPlus rasters and vectors. After these steps are run, the "Produce HAND Hydrofabric" step can be run for the new HUCs.

```
/foss_fim/src/acquire_and_preprocess_inputs.py -u <huc8s_to_process>
```
Sorry, this tool is deprecated, replacement scripts are coming soon.


### Evaluating Inundation Map Performance
After `fim_pipeline.sh` completes, or combinations of the three major steps described above, you can evaluate the model's skill. The evaluation benchmark datasets are available through ESIP in the `test_cases` directory.

Expand Down
Loading
Loading