Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accelerate SnpeffPostUpdateUploader #121

Open
erikyao opened this issue Apr 29, 2021 · 0 comments
Open

Accelerate SnpeffPostUpdateUploader #121

erikyao opened this issue Apr 29, 2021 · 0 comments
Assignees

Comments

@erikyao
Copy link
Contributor

erikyao commented Apr 29, 2021

The ideal implementation would involve a message queue:

  1. Any uploader extending SnpeffPostUpdateUploader will simply put the new IDs into the message queue in the post_update_data step, then return.
  2. SnpeffBaseUploader then takes over the responsibility to upload the SnpEff documents by looking for the new IDs in the message queue.

If the above implementation can not be done in time, at least use concurrency and/or parallelism in SnpeffPostUpdateUploader to upload the SnpEff documents. The slowness of a full release may be attributed to this SnpeffPostUpdateUploader. E.g. the wellderly uploading only takes 15min, while its post_update_data step takes 160min.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants