Upload sequences for all segments to S3 #183
Conversation
Adds all remaining gene to the list of segments whose sequences should be uploaded to S3.
Since Yam is likely extinct and we've already uploaded its most recent data to S3 once, we can remove it from the config now. If it returns, we can add it back.
|
The test upload ran without issues for all four lineages and all eight segments. The runtime was 2 hours instead of 45-50 minutes, though. We could reduce the runtime by dropping Yam from the lineages to consider. I'll push a commit next that makes this change. |
There was a problem hiding this comment.
I was slightly worried that the additional segments might overload RethinkDB. However, we are using the Docker runtime within the upload workflow, so we are still limited to just 4 concurrent connections to Fauna.
If we are worried about the additional runtime, we could consider running the additional segments to a separate GH Action workflow that doesn't affect the automated builds.
That's a good idea. Maybe we can try it this Thursday and see if it is annoyingly slow even without Yam? This runtime highlights again how incredibly slow the RethinkDB setup is. It shouldn't take hours to download this much data... |
Description of proposed changes
Adds all remaining gene to the list of segments whose sequences should be uploaded to S3.
Related issue(s)
Checklist