feat(ingest): Format ingested country: division into geoLocCountry, geoLocAdmin1, geoLocAdmin2

[anna-parker]

resolves #

preview URL: https://format-country.loculus.org/

Summary

We have mentioned in the past it would be good to break down NCBI's geoloc field. This is of the format: country: division, where division can normally be further split into geoLocAdmin1 and geoLocAdmin2.

Sadly the format of the division field is inconsistent. For example for the US some samples have state, city others city, state, yet others do not separate state with a , but or use the abbreviation. This means to extract more specific details we will need to do string matching. It also became clear to me that NIH does not clean their sequences based on the common occurrence of Conneticut. Astrahan is another example of this. I have now added a list of common misspellings, potentially in future we could use fuzzy matching for this.

I got chatgpt to write me up some nice yaml lists with all states/regions/provinces for countries where we have a lot of samples with geoLocAdmin1 information.

By default for unmapped countries I keep the mapping as is: division is mapped to geoLocAdmin1.

For mapped countries I put the state(region/province) in geoLocAdmin1 if the division matches the state (division must include the state name). I put the entire division string into geoLocAdmin2 as well - unless it is an exact match for the state, then I do not additionally add it to geoLocAdmin2 (to avoid duplication). If no state matches division I put the entire division string in geoLocAdmin2.

Potential Alternatives

• augur curate: https://docs.nextstrain.org/projects/augur/en/stable/usage/cli/curate/apply-geolocation-rules.html, this appears to rely on having a division and location field, this can be generated with https://docs.nextstrain.org/projects/augur/en/stable/usage/cli/curate/parse-genbank-location.html - remains to see how well this will work when division is not in a standard format
• https://pypi.org/project/pycountry/ has all countries and region based on ISO standards, also has fuzzy matching - will have to self-define thresholds

Screenshot

Previous list of geoLocAdmin1 fields:


New formatting:

PR Checklist

• All necessary documentation has been adapted.
• The implemented feature is covered by an appropriate test.

[anna-parker]

1. Download https://raw.githubusercontent.com/nextstrain/ncov-ingest/refs/heads/master/source-data/gisaid_geoLocationRules.tsv
2. Install augur (need to downgrade python to Python 3.11.10)
3. Run augur curate, relies on metadata being in tsv in format specified by dataformat tsv - either use command or modify format_ncbi_dataset_report some more.

dataformat tsv virus-genome --package results/ncbi_dataset.zip > results/metadata.tsv 
augur curate parse-genbank-location --metadata results/metadata.tsv --location-field='Geographic Location' --output-metadata results/curated.tsv
augur curate apply-geolocation-rules --metadata results/curated.tsv --output-metadata results/curated_rules.tsv  --geolocation-rules=gisaid_geoLocationRules.tsv --region-field='Geographic Region'

Still need to map the output of division and location into geoLocAdmin1 and geoLocAdmin2 - this should be done in a separate rule and then joined with other metadata fields, maybe in prepare data.

[anna-parker]

Closing as I improved this in #3026 by using pycountry for official ISO standard lists of geoLocAdmin1 divisions for every country and additionally used fuzzywuzy for fuzzy-matching