Auto deploy public builds #114
Conversation
Uses the Nextstrain CLI to upload all inputs for `all_public` in rename.smk to the provided URL.
Use the new `all_deploy` rule to automatically deploy all builds as the last step of the public builds pipeline. The public builds should no longer have to be manually inspected since outliers are now automatically removed.¹ ¹ #110
|
Hmm, the test run started through the GH Action workflow did not work as expected... From the logs However, I was able to run the build on AWS Batch manually with: And see the build is running as expected when I re-attach to the job: |
|
Oh, the upload would have failed anyways because I didn't update the AWS IAM user permissions for |
|
Still have no idea why the GH Action triggered run didn't work as intended, but the AWS Batch run that I triggered locally completed successfully and the live builds at https://nextstrain.org/flu/seasonal have been updated. |
|
Huh okay, so I get the same "Nothing to be done" message if I run the build command a brand new clone of the repo. So something in my local repo is influencing the DAG 🤦♀️ |
We were running into the Snakemake issue where the log directive in the default rule causes it to skip tasks.¹ This issue has been fixed since Snakemake 7.25.1, but we are still pinned to 7.24.1 in nextstrain/docker-base.² Just removing the log directive here is the simplest fix for this particular issue. We'll still be able to see the upload logs in the AWS Batch logs. ¹ snakemake/snakemake#2142 ² https://github.com/nextstrain/docker-base/blob/95cea4f44a803ce27992706c87a7237492dbb083/Dockerfile#L277
|
Confirmed that 087558c fixed the issue for me in a newly cloned repo. |
|
@j23414 With this merged, the public builds should automatically deploy in tomorrow's run. You won't have to download the build from AWS Batch anymore, but I would definitely check on Friday that the builds at nextstrain.org/flu/seasonal have been updated. |
Description of proposed changes
Deploy the public builds to s3://nextstrain-data as part of the automated public workflow.
The public builds should no longer have to be manually inspected since outliers are now automatically removed as of #110.
Related issue(s)
Follow up to #106
Testing