Update links for GISAID and GenBank accessions

[joverlee521]

Description of proposed changes

Adds a rule to fetch GISAID and GenBank accession links from UCSC
and uses the output in the transform_genbank and transform_gisaid
rules.

The UCSC file is modified to keep the headers and format exactly the
same as the current source-data/accessions.tsv.gz file so that it can
be directly replaced.

Related issue(s)

Resolves #484

Checklist

• Checks pass
• GISAID trial run
• GenBank trial run
• 2nd GISAID trial
• 2nd GenBank trial

[joverlee521]

Brief look at the trial run output metadata

1. I subset to the gisaid_epi_isl and genbank_accession columns, only including rows where both are not "?".

zstdcat open-accession-metadata.tsv.zst \
  | csvtk -t cut -f gisaid_epi_isl,genbank_accession \
  | csvtk -t filter2 -f '$genbank_accession != ?' 
  | csvtk -t sort -k genbank_accession > open-accessions.tsv

2. I used daff to compare the trial data with the prod data to get a summary of changes:

GISAID

@@	count
+++	2141069
---	81421
->	755

GenBank

@@	count
+++	1248778
---	7964
->	2424

The number of removals (---) and changes (->) makes me think that we might want to keep the original source-data/accessions.tsv.gz and just append the UCSC file.

[joverlee521]

Looking at the new trial metadata

GISAID

@@	count
+++	2141428
->	613

GenBank

@@	count
+++	1249453
->	2204

I think it's reasonable to default to the latest accession links in the downloaded UCSC file. If there are accession links that should be manually corrected, we can do so with annotations.

[AngieHinrichs]

Glad you're using the latest now and sorry I fell behind with the manual updates! Feel free to send me examples of any changes/omissions that look concerning. When the same sequence has been submitted multiple times to GISAID or GenBank (and especially when it has been submitted multiple times to both), there can be some instability from one run to the next regarding which of multiple possible IDs get paired up, which would explain some changes. Also, over time, some sequences have updates submitted, or are withdrawn from one repo or the other (or both).

[joverlee521]

Thank you @AngieHinrichs, both the manual and automatic accession links are extremely helpful ❤️

When the same sequence has been submitted multiple times to GISAID or GenBank (and especially when it has been submitted multiple times to both), there can be some instability from one run to the next regarding which of multiple possible IDs get paired up, which would explain some changes. Also, over time, some sequences have updates submitted, or are withdrawn from one repo or the other (or both).

This makes sense to me, but @trvrb please let me know if you'd like the order to be switched here so that we can have stable accession links, i.e. append the source-data/accessions.tsv.gz file to the downloaded file so that the hard-coded links in repo have precedence over the automated ones.

[trvrb]

I think it makes sense as you say to have the linkage be the best current attempt at linkage and it's okay if this is not fully stable.

[joverlee521]

If there's no other feedback, I'll merge this tomorrow morning before the automated run.