Skip to content

Commit

Permalink
Propagate exceptions raised while fetching dependencies
Browse files Browse the repository at this point in the history
We need to collect the results from the iterator which is produced by
`ThreadPool.map()` to ensure that exceptions raised in the worker
threads are propagated to the main thread.

This commit collects the iterator by calling `list()` on it, which
ensures that any exceptions raised in a worker thread are propagated to
the main thread.
  • Loading branch information
simu committed Sep 18, 2023
1 parent 96a44e1 commit 63574de
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion commodore/dependency_mgmt/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,11 @@ def fetch_parallel(fetch_fun, cfg, to_fetch):
Fetch dependencies in parallel threads with ThreadPoolExecutor.
"""
with ThreadPoolExecutor() as exe:
exe.map(fetch_fun, itertools.repeat(cfg), to_fetch)
# We need to collect the results from the iterator produced by exe.map to ensure that any
# exceptions raised in `fetch_fun` are propagated, cf.
# https://docs.python.org/3/library/concurrent.futures.html#executor-objects. We do so by
# simply materializing the iterator into a list.
list(exe.map(fetch_fun, itertools.repeat(cfg), to_fetch))


def register_components(cfg: Config):
Expand Down

0 comments on commit 63574de

Please sign in to comment.