Skip to content

Commit

Permalink
feat(basemaps): Update the ETL to support the new version with indivi…
Browse files Browse the repository at this point in the history
…dual layer. BM-977 (#493)

#### Motivation

The new ETL version contains a filename which can used for individual
layer processing, and default to topographic.

#### Modification

- Update the ETL workflow to v1 which is the current latest version
- Add the filename parameter to workflow
- Rename the `cron-vector-etl` workflow to `cron-vector-etl-topographic`
workflow
- Update the ETL entry command to `node /app/index.js` which been
removed from the ETL container.


#### Checklist
- [ ] Tests updated - No test for workflow, already running the workflow
in argo for testing.
- [ ] Docs updated - No doc for vector etl to update
- [x] Issue linked in Title
  • Loading branch information
Wentao-Kuang authored Mar 25, 2024
1 parent 4b8f599 commit d3db2c3
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 4 deletions.
11 changes: 8 additions & 3 deletions workflows/basemaps/vector-etl.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ spec:

- name: version_basemaps_etl
description: Version of the Basemaps ETL eks container to use
value: 'v1.0.0'
value: 'v1'

- name: target
description: S3 Bucket to use for storing the output
Expand All @@ -27,6 +27,10 @@ spec:
- 'linz-basemaps'
- 'linz-basemaps-staging'

- name: filename
description: output filename in s3
value: 'topographic'

- name: create_pull_request
description: Should a pull request be created in linz/basemaps-config
value: 'true'
Expand Down Expand Up @@ -71,13 +75,14 @@ spec:
requests:
memory: 7.8Gi
cpu: 15000m
command: [node, index.cjs]
command: [node, /app/index.js]
env:
- name: AWS_ROLE_CONFIG_PATH
value: s3://linz-bucket-config/config.basemaps.json
args:
- '--all'
- "{{= workflow.parameters.filename == 'topographic'? '--all' : '--layer=' + workflow.parameters.filename }}"
- '--target={{ workflow.parameters.target }}'
- '--filename={{ workflow.parameters.filename }}'
- '--output=/tmp/'
- '--commit'
outputs:
Expand Down
4 changes: 3 additions & 1 deletion workflows/cron/cron-vector-etl.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
name: cron-vector-etl
name: cron-vector-etl-topographic
namespace: argo
labels:
linz.govt.nz/category: basemaps
Expand All @@ -20,5 +20,7 @@ spec:
name: basemaps-vector-etl
arguments:
parameters:
- filename:
value: 'topographic'
- name: 'retry'
value: '2'

0 comments on commit d3db2c3

Please sign in to comment.