Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Allow users to configure the internal thread pool (#11) #54

Closed
wants to merge 7 commits into from
+32 −5
Closed

feat: Allow users to configure the internal thread pool (#11) #54

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
7de6eb7
feat: Allow users to configure the internal thread pool (#11)
Oct 8, 2023
2d464f2
Moved executor_kwargs logic to constructor and removed unnecessary file
Oct 12, 2023
d445578
Remove unnecessary configurable-thread-pool file
Oct 12, 2023
4231270
Added executor_kwargs to EarthEngineStore
alhridoy Oct 12, 2023
e643352
Merge branch 'main' into configurable-thread-pool
alhridoy Oct 12, 2023
4cd311d
Added executor_kwargs parameter to EarthEngineStore and open_dataset …
alhridoy Oct 15, 2023
ec80d19
Added executor_kwargs parameter, removed extra space from raw_indexin…
alhridoy Oct 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions configurable-thread-pool
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Branch 'configurable-thread-pool' set up to track remote branch 'main' from 'origin'.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this file.

9 changes: 6 additions & 3 deletions xee/ext.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -634,7 +634,7 @@ def reduce_bands(x, acc):
return target_image

def _raw_indexing_method(
self, key: tuple[Union[int, slice], ...]
self, key: tuple[Union[int, slice], ...], executor_kwargs: Optional[dict] = None
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This extra argument is named well, but I think it should be set in the constructor.

) -> np.typing.ArrayLike:
key, squeeze_axes = self._key_to_slices(key)

Expand Down Expand Up @@ -682,8 +682,11 @@ def _raw_indexing_method(
for _ in range(shape[0])
]

# TODO(#11): Allow users to configure this via kwargs.
with concurrent.futures.ThreadPoolExecutor() as pool:
# If executor_kwargs is None, use an empty dictionary
if executor_kwargs is None:
executor_kwargs = {}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do this bit of logic in the constructor, too.

# Pass executor_kwargs to ThreadPoolExecutor
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment is a bit too verbose for my taste.

with concurrent.futures.ThreadPoolExecutor(**executor_kwargs) as pool:
for (i, j, k), arr in pool.map(
self._make_tile, self._tile_indexes(key[0], bbox)
):
Expand Down