Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingest): Use augur curate to curate country metadata #3015

Draft
wants to merge 36 commits into
base: main
Choose a base branch
from
Draft

feat(ingest): Use augur curate to curate country metadata #3015

wants to merge 36 commits into from

Commits on Oct 10, 2024

  1. format authors in prepro

    @anna-parker
    anna-parker committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    b44dc19 View commit details
    Browse the repository at this point in the history

  2. Fix ingest

    @anna-parker
    anna-parker committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    8c8912d View commit details
    Browse the repository at this point in the history

  3. try to fix pattern

    @anna-parker
    anna-parker committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    c6ba486 View commit details
    Browse the repository at this point in the history

  4. simplify regex

    @anna-parker
    anna-parker committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    4aba133 View commit details
    Browse the repository at this point in the history

  5. fix check

    @anna-parker
    anna-parker committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    77828b6 View commit details
    Browse the repository at this point in the history

  6. Add tests

    @anna-parker
    anna-parker committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    77a5d02 View commit details
    Browse the repository at this point in the history

  7. Add to ena submission

    @anna-parker
    anna-parker committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    410a645 View commit details
    Browse the repository at this point in the history

  8. fix

    @anna-parker
    anna-parker committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    814dab2 View commit details
    Browse the repository at this point in the history

  9. fix other edge case

    @anna-parker
    anna-parker committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    255337d View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. Update ena-submission/scripts/ena_submission_helper.py 

    Co-authored-by: Cornelius Roemer <[email protected]>
    @anna-parker @corneliusroemer
    anna-parker and corneliusroemer authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    3b00534 View commit details
    Browse the repository at this point in the history

  2. Update ena-submission/scripts/ena_submission_helper.py

    @anna-parker
    anna-parker authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    315baf3 View commit details
    Browse the repository at this point in the history

  3. Update ena-submission/scripts/ena_submission_helper.py 

    Co-authored-by: Cornelius Roemer <[email protected]>
    @anna-parker @corneliusroemer
    anna-parker and corneliusroemer authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    e780275 View commit details
    Browse the repository at this point in the history

  4. Update ena-submission/scripts/ena_submission_helper.py

    @anna-parker
    anna-parker authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    fe9543f View commit details
    Browse the repository at this point in the history

  5. Update ingest/scripts/prepare_metadata.py

    @anna-parker
    anna-parker authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    9a13990 View commit details
    Browse the repository at this point in the history

  6. Update preprocessing/nextclade/src/loculus_preprocessing/processing_f… 

    …unctions.py
    @anna-parker
    anna-parker authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    c71e6a6 View commit details
    Browse the repository at this point in the history

  7. rename

    @anna-parker
    anna-parker committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    9886d4d View commit details
    Browse the repository at this point in the history

  8. Update reformat_authors_from_genbank_to_loculus

    @anna-parker
    anna-parker committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    1af54f5 View commit details
    Browse the repository at this point in the history

  9. Additionally format authors with correct white space

    @anna-parker
    anna-parker committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    7eed322 View commit details
    Browse the repository at this point in the history

  10. Improve error message

    @anna-parker
    anna-parker committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    b3ab701 View commit details
    Browse the repository at this point in the history

  11. add tests

    @anna-parker
    anna-parker committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    d94e60a View commit details
    Browse the repository at this point in the history

  12. fix missing pattern

    @anna-parker
    anna-parker committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    4a27aaf View commit details
    Browse the repository at this point in the history

  13. improve error logs

    @anna-parker
    anna-parker committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    356b70e View commit details
    Browse the repository at this point in the history

  14. fix error

    @anna-parker
    anna-parker committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    497d09f View commit details
    Browse the repository at this point in the history

  15. Update preprocessing/nextclade/src/loculus_preprocessing/processing_f… 

    …unctions.py
    @anna-parker
    anna-parker authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    5940cbc View commit details
    Browse the repository at this point in the history

  16. improve logging more

    @anna-parker
    anna-parker committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    a44e05a View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2024

  1. feat(ingest): Do not use processed tsv but raw jsonl when ingesting d… 

    …ata from NCBI Virus (#2990)

* Use raw jsonl instead of generated tsv when ingesting data from NCBI virus

* Do not require authors list to end in ';', capitalize names correctly.

* Add tests for capitalization

* Add a warning if author list might be in wrong format

* Add ascii specific warning

* Add tests for warnings and errors

* Only capitalize if full authors string is upper case

* Properly capitalize initial

* Move titlecase option to ingest only - add ingest tests
    @anna-parker
    anna-parker authored Oct 17, 2024
    Configuration menu
    Copy the full SHA
    c5f9966 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into format_authors

    @anna-parker
    anna-parker authored Oct 17, 2024
    Configuration menu
    Copy the full SHA
    b019e96 View commit details
    Browse the repository at this point in the history

  3. Add geoloc metadata parser

    @anna-parker
    anna-parker committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    f07a92f View commit details
    Browse the repository at this point in the history

  4. fix loculus call

    @anna-parker
    anna-parker committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    9433ab6 View commit details
    Browse the repository at this point in the history

  5. Fix tests

    @anna-parker
    anna-parker committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    1a0718f View commit details
    Browse the repository at this point in the history

  6. more updates

    @anna-parker
    anna-parker committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    42ec318 View commit details
    Browse the repository at this point in the history

  7. try again

    @anna-parker
    anna-parker committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    da2e140 View commit details
    Browse the repository at this point in the history

  8. add curl to dockerfile

    @anna-parker
    anna-parker committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    25bb835 View commit details
    Browse the repository at this point in the history

  9. second try

    @anna-parker
    anna-parker committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    0c4ba63 View commit details
    Browse the repository at this point in the history

  10. fix quoting issues with escape characters

    @anna-parker
    anna-parker committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    7fc4c1b View commit details
    Browse the repository at this point in the history

  11. Do not convert ints to floats

    @anna-parker
    anna-parker committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    fa938bc View commit details
    Browse the repository at this point in the history