Issue 99

[HobnobMancer]

Fix Issue #99

Improve handling when incurring errors when retrieving data from NCBI

[HobnobMancer]

The initial problem stated in issue #99 is fixed.

However, GenBank accessions that are no longer listed in NCBI are still retried as many times as defined by args.retries (default 10). This unnecessarily increases running time and demand on the NCBI server.

[HobnobMancer]

Unit test coverage could be increased to >= 60%

[codecov]

Codecov Report

Merging #102 (b3b9dc4) into master (47af82a) will decrease coverage by 2.05%.
The diff coverage is 3.69%.

@@            Coverage Diff             @@
##           master     #102      +/-   ##
==========================================
- Coverage   58.62%   56.57%   -2.06%     
==========================================
  Files          60       61       +1     
  Lines        5337     5541     +204     
==========================================
+ Hits         3129     3135       +6     
- Misses       2208     2406     +198     

[HobnobMancer]

Changes implemented

Changing operation has been successful:

1. Separate invalid IDs to IDs that suffered to failed connections
2. Parse batches containing invalid IDs separately to and before failed connection batches

Downloaded protein sequences are cached to a FASTA file.

Updated information in the docs on caching.

Future developments notes

[1]
Unit tests raise deprecated feature warning:

tests/conftest.py:40
  /home/circleci/cazy_webscraper/tests/conftest.py:40: MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
    Base = declarative_base()

cazy_webscraper currently uses sqlalchemy version >= 1.4.20. This should be changed to ==, and cazy_webscraper should be migrated to sqlalchemy version 2.x on a new branch. (See issue #106)

[2]
Add the protein description from NCBI (retrieved when downloading the sequence record from NCBI) to the local CAZyme database.

Add the protein description to a new column called genbank_description in the Genbanks table.