Release v1.3.0

Release summary for relecov-tools v1.3.0 - 2024-12-23

Modules

• Included files-folder option for read-lab-metadata when no samples_data.json is provided.
• Included folder_names multiple argument for pipeline_manager to specify names of folders to process.
• Included send-mail module: Automated email notification module to generate and send validation reports.

Added Enhancements

• logs-to-excel: Now handles logs with multiple keys and includes folder logs in a new sheet called Global Report.
• Improved logging messages for duplicated sample IDs in read-lab and download modules.
• New method string_to_date() in utils to search for a date pattern within a string.
• Integrated jinja2 for template rendering in the mail module.
• Mail module configurations added to configuration.json.
• Added static method get_invalid_count in log_summary.py.
• Included try-except for every module to catch unexpected errors in main.py.
• Added, removed, and renamed collecting institutions and their cities.
• Updated contact directory to integrate additional institution data.
• Added support for multiple recipients in the email_receiver field.
• Introduced a new Jinja template for successful and error validation reports.
• Modified module logic to dynamically select and render email templates based on user input.
• Enhanced email formatting and added a default CC to mails.
• Validate module now takes an optional argument to select the name of the sheet to check in the Excel file.

Fixes

• Fixed python linting workflow: It was still waiting for .py files.
• files-folder argument: Now works with relative paths in read-lab-metadata.
• check-gzip-integrity(): Now catches any exception in utils.py as it only needs to return True when the file can be decompressed.
• validate module: Does not crash when no METADATA_LAB sheet is found.
• Fixed email module.

Changed

• Pipeline-manager: fields_to_split is now in configuration.json to group samples by those fields.
• Homogenized style of report global report sheet in logs-excel.

relecov-tools v1.2.0

Release Summary for relecov-tools v1.2.0 - 2024-10-11

Modules:

• Wrapper Module: Introduced to launch download, read-lab-metadata, and validate processes sequentially.
• Pipeline Manager: Renamed and updated from launch-pipeline, with enhanced functionality when using tools via CLI.

Enhancements:

• Integrity Checks: Added gzip file integrity check post-download; moved cleaning process to the workflow's end.
• Reconnection Decorator: Introduced in sftp_client.py to handle lost connections.
• Hospital Additions: Added Hospital Universitari Doctor Josep Trueta and Hospital Universitario Araba to laboratory_address.json.
• Metadata Handling:
  • samples_data.json is no longer mandatory as input for read-lab-metadata.
  • Supports alternative column names for distinct headers in read-lab-metadata.
  • Generates new samples_data.json if not provided.
  • Handles missing sample IDs and improved logging.
• Cleaning Process: More accurate by skipping only sequencing files instead of entire folders.
• Logs and Summaries: Single log summaries created for each folder during download; enhanced log handling for download, read-lab-metadata, and wrapper.
• Excel and Messages: Added Excel resize, bracket removal in messages, and improved exception handling in log_summary.py.

Fixes:

• City Name Correction: Corrected wrong city name in laboratory_address.json.
• Layout Detection: Fixed single-paired layout detection issues due to capital letters.
• Error Handling: Improved error handling in merge_logs() and create_logs_excel() methods.
• Empty Rows: Handled multiple empty rows in metadata Excel files.

Changes:

• Configuration Updates:
  • Refactored bioinfo_lab_heading to alt_header_equivalences in configuration.json.
  • Added missing schema fields related to bioinformatics results.
• Metadata Template: Updated and moved to relecov_tools/assets.
• Python Lint: Triggered only when PR includes Python files.
• Concurrency and Testing: Moved concurrency to the whole workflow and updated test datasets.
• Download Skipping: Folders with "invalid_samples" in their name are now skipped during download.
• Warnings and Labels: Some warnings in read-lab-metadata now include labels, and trailing spaces removed.
• Pipeline Data Splitting: Pipeline manager now splits data based on enrichment_panel and version, organizing into separate folders.

Removed:

• Duplicate Tests: Removed redundant tests in test_sftp-handle.
• Deprecated Auto-Release: Deleted non-functional auto-release in pypi_publish.
• Reconnection Decorator: Removed initial sleep time for the reconnection decorator in sftp_client.py, with increased sleep time from the second attempt.

relecov-tools v1.1.0

[v1.1.0] - 2024-09-16

Credits

Code contributions to the hotfix:

• Pablo Mata
• Sara Monzón

Modules

• New logs-to-excel function to create an excel file given a list of log-summary.json files #300

Added enhancements

• Included a way to extract pango-designation version in read-bioinfo-metadata #299
• Now log_summary.py also creates an excel file with the process logs #300
• Read-bioinfo-metadata splits files and data by batch of samples #306
• Included a sleep time in test_sftp-handle to avoid concurrency check failure #308

Fixes

• Fixes in launch_pipeline including creation of samples_id.txt and joined validated json #303
• Fixed failing module_tests.yml workflow due to deprecated upload-artifact version #308

Changed

• Changed pypi_publish action to publish on every release, no need to push tags #308

Removed

• Removed only_samples argument in log_summary.py as it was not used in any module. #300

relecov-tools v1.0.0

This release is the first fully functional version of the relecov-tools package. This means that the workflow for data management can be executed end-to-end with no restrictions from now on.

First dev version

Full phage+ schema, before reducing fields.