Skip to content

Commit

Permalink
Make s3_dst config to top level config
Browse files Browse the repository at this point in the history
Nested configs for s3 URLs are challenging to override by the Snakemake
`--config` option, as discussed in the following comment:

nextstrain/dengue#13 (comment)

This moves the url to the top level, and propagate those changes to the relevant Snakefiles.
  • Loading branch information
j23414 committed Oct 27, 2023
1 parent c310037 commit d9721e1
Show file tree
Hide file tree
Showing 3 changed files with 4 additions and 4 deletions.
2 changes: 1 addition & 1 deletion ingest/Snakefile
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ def _get_all_targets(wildcards):
print(
f"Skipping file upload for {target!r} because there are duplicate remote file names."
)
elif not params.get("dst"):
elif not params.get("s3_dst"):
print(
f"Skipping file upload for {target!r} because the destintion was not defined."
)
Expand Down
4 changes: 2 additions & 2 deletions ingest/config/optional.yaml
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Optional configs used by Nextstrain team
# Params for uploads
# AWS S3 Bucket with prefix
s3_dst: 's3://nextstrain-data/files/workflows/monkeypox'
upload:
# Upload params for AWS S3
s3:
# AWS S3 Bucket with prefix
dst: 's3://nextstrain-data/files/workflows/monkeypox'
# Files to upload to S3 that are in the `data` directory
files_to_upload: [
'genbank.ndjson',
Expand Down
2 changes: 1 addition & 1 deletion ingest/workflow/snakemake_rules/upload.smk
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ rule upload_to_s3:
"data/upload/s3/{file_to_upload}-to-{remote_file_name}.done",
params:
quiet="" if send_notifications else "--quiet",
s3_dst=config["upload"].get("s3", {}).get("dst", ""),
s3_dst=config.get("s3_dst", ""),
cloudfront_domain=config["upload"].get("s3", {}).get("cloudfront_domain", ""),
shell:
"""
Expand Down

0 comments on commit d9721e1

Please sign in to comment.