Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry logic for file access in micall watcher. #1235

Open
Donaim opened this issue Jan 21, 2025 · 1 comment
Open

Retry logic for file access in micall watcher. #1235

Donaim opened this issue Jan 21, 2025 · 1 comment

Comments

@Donaim
Copy link
Member

Donaim commented Jan 21, 2025

MiCall watcher is designed to run on clusters, and as such, it should acknowledge occasional connectivity failures.

One specific failure, that we found to be common in our deployment, is when one of the "externals" filesystems report files as missing. This results in micall watcher crashing with:

2025-01-21 01:00:04[ERROR]micall.monitor.kive_watcher.wait_for_retry(): Waiting 0:00:05 before retrying.
Traceback (most recent call last):
  File "/usr/local/share/MiCall/micall/monitor/kive_watcher.py", line 531, in poll_runs
    self.check_completed_folders()
  File "/usr/local/share/MiCall/micall/monitor/kive_watcher.py", line 547, in check_completed_folders
    pipeline_group)
  File "/usr/local/share/MiCall/micall/monitor/kive_watcher.py", line 588, in collate_folder
    self.copy_outputs(folder_watcher, scratch_path, target_path)
  File "/usr/local/share/MiCall/micall/monitor/kive_watcher.py", line 602, in copy_outputs
    results_path)
  File "/usr/local/share/MiCall/micall/monitor/kive_watcher.py", line 690, in extract_coverage_maps
    open(target_path, 'wb') as target:
BlockingIOError: [Errno 11] Resource temporarily unavailable: '/Results/version_7.17/denovo/coverage_maps/1943P47H-HIV_S24.HIVB.V3LOOP.details.png'

To solve this, we can implement a retry logic.
Whenever there is an error caused by external hardware, pause for some time, and repeat the action again.

@Donaim
Copy link
Member Author

Donaim commented Jan 21, 2025

This also happens in kive: cfe-lab/Kive#2066

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant