WIP: write sigs to tmpdir first to improve restart capacity

[bluegenes]

To allow restart, write sigs to a tmpdir instead of directly to a zipfile. Then copy all sigs to the zipfile at the very end.

If we get interrupted, first read all the existing sigs that we can, build a manifest, and use this to intersect with sig templates to avoid rebuilding. I think the way to do this is to store a Hashmap of filename: Records for faster lookup+intersection as we loop through files.

To do:

• figure out how to properly do this intersection

thoughts:

• do we want to write mini manifests as we go, so we can just read those in upon restart? Currently set to read all files in the tmpdir.
• we currently have to build a new record for each signature b/c the filename + location needs to change. Is there a way to minimize this overhead / modify the existing record?

[bluegenes]

Intersecting existing sig Record info with sig templates does not seem straightforward atm, see sourmash-bio/sourmash#3322

[bluegenes]

chat with titus:

• another idea: write *sig.zip files, batched e.g. 1000 sigs per zipfile (make tunable)

[bluegenes]

closing in favor of #101 and #102