Skip to content

Commit

Permalink
feat: add a workflowTemplate to create national DEM TDE-1166 (#890)
Browse files Browse the repository at this point in the history 
#### Motivation

Having a Nationally combined set of DEMs is useful for any data consumer
wanted to access to the entire country DEM. This dataset is aimed to be
updated on a regular basis to include all new single region/area dataset
that is newly published or updated.


![image](https://github.com/user-attachments/assets/e9d363f5-c586-4ced-8385-24d626ace894)


#### Modification

Add a WorkflowTemplate that create a nationally combined set of DEM
based off a configuration file that list the datasets to include in
order of priority. This WF template is able to be used to run a cron
workflow.

#### Checklist

- [ ] Tests updated
- [x] Docs updated
- [x] Issue linked in Title
  • Loading branch information
@paulfouquet
paulfouquet authored Dec 6, 2024
1 parent 61aa899 commit 731b0cc
Show file tree
Hide file tree
Showing 2 changed files with 441 additions and 4 deletions.
28 changes: 24 additions & 4 deletions workflows/raster/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
- [Standardising](#Standardising)
- [copy](#copy)
- [publish-odr](#Publish-odr)
- [National DEM](#national-dem)
- [tests](#Tests)

# Standardising
Expand All @@ -16,7 +17,7 @@ Publishing to the AWS Registry of Open Data is an optional step [publish-odr](#P
## Workflow Input Parameters

| Parameter | Type | Default | Description |
|------------------------| ----- | ------------------------------------- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ---------------------- | ----- | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| user_group | enum | none | Group of users running the workflow |
| ticket | str | | Ticket ID e.g. 'AIP-55' |
| region | enum | | Region of the dataset |
Expand Down Expand Up @@ -53,7 +54,7 @@ Publishing to the AWS Registry of Open Data is an optional step [publish-odr](#P
### Example Input Parameters

| Parameter | Value |
|------------------------| --------------------------------------------------------------------------------- |
| ---------------------- | --------------------------------------------------------------------------------- |
| ticket | AIP-55 |
| region | bay-of-plenty |
| source | s3://linz-imagery-upload/PRJ39741_BOPLASS_Imagery_2021-22/PRJ39741_03/01_GeoTiff/ |
Expand All @@ -65,7 +66,7 @@ Publishing to the AWS Registry of Open Data is an optional step [publish-odr](#P
| compression | webp |
| create_capture_area | true |
| cutline | s3://linz-imagery-staging/cutline/bay-of-plenty_2021-2022.fgb |
| odr_url | s3://nz-imagery/taranaki/new-plymouth_2017_0.1m/rgb/2193/ |
| odr_url | s3://nz-imagery/taranaki/new-plymouth_2017_0.1m/rgb/2193/ |
| category | rural-aerial-photos |
| gsd | 0.3 |
| producer | Aerial Surveys |
Expand Down Expand Up @@ -133,6 +134,7 @@ graph TD;

if `odr_url` is provided, gets existing `linz:slug` and `collection-id` STAC metadata fields (e.g. for dataset resupply),
If no `odr_url` is provided:

- a ULID is generated for the collection ID
- the input parameters are used to generate the LINZ slug

Expand Down Expand Up @@ -212,7 +214,7 @@ Access permissions are controlled by the [Bucket Sharing Config](https://github.
## Workflow Input Parameters

| Parameter | Type | Default | Description |
| -------------------- | ----- |---------------------------------------------------------------------------------------------------------------------------------------------------------------| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| -------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| user_group | enum | none | Group of users running the workflow |
| ticket | str | | Ticket ID e.g. 'AIP-55' |
| region | enum | | Region of the dataset |
Expand Down Expand Up @@ -284,6 +286,24 @@ graph TD;

See the [copy template](#copy) for more information.

# national-dem

This workflow combines a set of DEMs datasets in order to create a single national dataset composed of 1:50k tiles.

Upon completion all standardised TIFF and STAC files will be located with the ./flat/ directory of the workflow in the artifacts scratch bucket. In addition, a Basemaps link is produced enabling visual QA.

Publishing to the AWS Registry of Open Data is an optional step [publish-odr](#Publish-odr) that can be run automatically after standardisation.

## Workflow Input Parameters

| Parameter | Type | Default | Description |
| -------------- | ---- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| config_file | str | https://raw.githubusercontent.com/linz/basemaps-config/master/config/tileset/elevation.json | Location of the configuration file listing the source datasets to merge. |
| odr_url | str | | (Optional) If an existing dataset add the S3 path to the dataset here to load existing metadata e.g. "s3://nz-elevation/new-zealand/new-zealand/dem_1m/2193/" |
| group | 2 | | How many output tiles to process in each standardising task "pod". Change if you have resource or performance issues when standardising a dataset. |
| publish_to_odr | str | false | Run [publish-odr](#Publish-odr) after standardising has completed successfully |
| copy_option | enum | --force-no-clobber | Used only if `publish_to_odr` is true.<dl><dt>`--no-clobber` </dt><dd> Skip overwriting existing files.</dd><dt> `--force` </dt><dd> Overwrite all files. </dd><dt> `--force-no-clobber` </dt><dd> Overwrite only changed files, skip unchanged files. </dd></dl> |

# Tests

## How To Use the Test Workflow
Expand Down
Loading

0 comments on commit 731b0cc

Please sign in to comment.