Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add file checksums to BCO format #4

Open
bentsherman opened this issue Sep 28, 2023 · 0 comments
Open

Add file checksums to BCO format #4

bentsherman opened this issue Sep 28, 2023 · 0 comments

Comments

@bentsherman
Copy link
Member

The BCO format supports SHA1 checksums for URI objects. These checksums are particularly useful for intermediate output files since they are usually deleted once the run completes.

There is a Nextflow PR nextflow-io/nextflow#3802 to save the MD5 checksum of each output file to the task cache to facilitate better provenance reporting. The MD5 checksum is advantageous because the big three cloud providers automatically compute the MD5 checksum of an object when it is uploaded, so Nextflow can re-use these checksums and save some time.

Unfortunate that BCO only allows for SHA1. We would have to compute SHA1 in addition to the MD5 checksums already being computed. Need to think on this more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant