Skip to content

Commit

Permalink
fix: copy regex (#889)
Browse files Browse the repository at this point in the history 
#### Motivation

Argo Workflows do not like the previous regex format.

#### Modification

Revert to previous format, retaining `capture-dates.geojson`

#### Checklist

- [ ] Tests updated
- [X] Docs updated
- [ ] Issue linked in Title
  • Loading branch information
@amfage
amfage authored Dec 3, 2024
1 parent fe3fd53 commit 7590309
Show file tree
Hide file tree
Showing 2 changed files with 15 additions and 15 deletions.
28 changes: 14 additions & 14 deletions workflows/raster/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -211,20 +211,20 @@ Access permissions are controlled by the [Bucket Sharing Config](https://github.

## Workflow Input Parameters

| Parameter | Type | Default | Description |
| -------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| user_group | enum | none | Group of users running the workflow |
| ticket | str | | Ticket ID e.g. 'AIP-55' |
| region | enum | | Region of the dataset |
| source | str | s3://linz-imagery-staging/test/sample/ | The URIs (paths) to the s3 source location. Separate multiple source paths with `;` |
| target | str | s3://linz-imagery-staging/test/sample_target/ | The URIs (paths) to the s3 target location |
| include | regex | \\.tiff?\$\|\\.json\$\|\\.tfw\$\|\\Q\/capture-area.geojson\\E\|\\Q\/capture-dates.geojson\\E | A regular expression to match object path(s) or name(s) from within the source path to include in the copy. |
| exclude | regex | | A regular expression to match object path(s) or name(s) from within the source path to exclude from the copy. |
| copy_option | enum | --no-clobber | <dl><dt>`--no-clobber` </dt><dd> Skip overwriting existing files.</dd><dt> `--force` </dt><dd> Overwrite all files. </dd><dt> `--force-no-clobber` </dt><dd> Overwrite only changed files, skip unchanged files. </dd></dl> |
| flatten | enum | false | Flatten the files in the target location (useful for multiple source locations) |
| group | int | 1000 | The maximum number of files for each pod to copy (will use the value of `group` or `group_size` that is reached first). |
| group_size | str | 100Gi | The maximum group size of files for each pod to copy (will use the value of `group` or `group_size` that is reached first). |
| transform | str | `f` | String to be transformed from source to target to renamed filenames, e.g. `f.replace("text to replace", "new_text_to_use")`. Leave as `f` for no transformation. |
| Parameter | Type | Default | Description |
| -------------------- | ----- |---------------------------------------------------------------------------------------------------------------------------------------------------------------| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| user_group | enum | none | Group of users running the workflow |
| ticket | str | | Ticket ID e.g. 'AIP-55' |
| region | enum | | Region of the dataset |
| source | str | s3://linz-imagery-staging/test/sample/ | The URIs (paths) to the s3 source location. Separate multiple source paths with `;` |
| target | str | s3://linz-imagery-staging/test/sample_target/ | The URIs (paths) to the s3 target location |
| include | regex | \\.tiff?\$\|\\.json\$\|\\.tfw\$\|/capture-area\\.geojson\$\|/capture-area\\.geojson\$ | A regular expression to match object path(s) or name(s) from within the source path to include in the copy. |
| exclude | regex | | A regular expression to match object path(s) or name(s) from within the source path to exclude from the copy. |
| copy_option | enum | --no-clobber | <dl><dt>`--no-clobber` </dt><dd> Skip overwriting existing files.</dd><dt> `--force` </dt><dd> Overwrite all files. </dd><dt> `--force-no-clobber` </dt><dd> Overwrite only changed files, skip unchanged files. </dd></dl> |
| flatten | enum | false | Flatten the files in the target location (useful for multiple source locations) |
| group | int | 1000 | The maximum number of files for each pod to copy (will use the value of `group` or `group_size` that is reached first). |
| group_size | str | 100Gi | The maximum group size of files for each pod to copy (will use the value of `group` or `group_size` that is reached first). |
| transform | str | `f` | String to be transformed from source to target to renamed filenames, e.g. `f.replace("text to replace", "new_text_to_use")`. Leave as `f` for no transformation. |
| aws_role_config_path | str | `s3://linz-bucket-config/config-write.elevation.json,s3://linz-bucket-config/config-write.imagery.json,s3://linz-bucket-config/config-write.topographic.json` | s3 URL or comma-separated list of s3 URLs allowing the workflow to write to a target(s). |

## Examples
Expand Down
2 changes: 1 addition & 1 deletion workflows/raster/copy.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ spec:
- name: target
value: 's3://linz-imagery-staging/test/sample_target/'
- name: include
value: '\.tiff?$|\.json$|\.tfw$|\Q/capture-area.geojson\E|\Q/capture-dates.geojson\E'
value: '\.tiff?$|\.json$|\.tfw$|/capture-area\.geojson$|/capture-dates\.geojson$'
- name: exclude
value: ''
- name: copy_option
Expand Down

0 comments on commit 7590309

Please sign in to comment.