Skip to content

Commit

Permalink
Generate a cache directory even if the "cache" folder is not found (#…
Browse files Browse the repository at this point in the history
…1499)

### What kind of change does this PR introduce?

* Allows the `pytest` setup to load testing data into a cache folder
even if the cache folder does not exist.

### Does this PR introduce a breaking change?

No.

### Other information:

Relevant traceback (conda-forge):
```
==================================== ERRORS ====================================
_________________ ERROR at setup of TestSEuclidean.test_simple _________________
[gw0] linux -- Python 3.8.18 $PREFIX/bin/python3.8

        When running pytest with multiple workers, one worker will copy data remotely to _default_cache_dir while
        other workers wait using lockfile. Once the lock is released, all workers will copy data to their local
        threadsafe_data_dir."""
    
        if (
            not _default_cache_dir.joinpath(helpers.TESTDATA_BRANCH).exists()
            or helpers.PREFETCH_TESTING_DATA
        ):
            if worker_id in "master":
                helpers.populate_testing_data(branch=helpers.TESTDATA_BRANCH)
            else:
>               _default_cache_dir.mkdir(exist_ok=True)

tests/conftest.py:447: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = PosixPath('/home/conda/.cache/xclim-testdata'), mode = 511
parents = False, exist_ok = True

    def mkdir(self, mode=0o777, parents=False, exist_ok=False):
        """
        Create a new directory at this given path.
        """
        if self._closed:
            self._raise_closed()
        try:
>           self._accessor.mkdir(self, mode)
E           FileNotFoundError: [Errno 2] No such file or directory: '/home/conda/.cache/xclim-testdata'

../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.8/pathlib.py:1288: FileNotFoundError
```
  • Loading branch information
Zeitsperre authored Oct 13, 2023
2 parents 25641cc + 33c70f9 commit d53a5b6
Show file tree
Hide file tree
Showing 3 changed files with 3 additions and 2 deletions.
1 change: 1 addition & 0 deletions CHANGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Bug fixes
* Calling a ``sdba.map_blocks``-wrapped function with data chunked along the reduced dimensions will raise an error. This forbids chunking the trained dataset along the distribution dimensions, for example. (:issue:`1481`, :pull:`1482`).
* Optimization of indicators ``huglin_index`` and ``biologically_effective_degree_days`` when used with dask and flox. As a side effect, the indice functions (i.e. under ``xc.indices``) no longer mask incomplete periods. The indicators' output is unchanged under the default "check_missing" setting (:issue:`1494`, :pull:`1495`).
* Fixed ``xclim.indices.run_length.lazy_indexing`` which would sometimes trigger the loading of auxiliary coordinates. (:issue:`1483`, :pull:`1484`).
* Fixed a bug in the `pytest` configuration that could prevent testing data caching from occurring in systems where the platform-dependent cache directory is not found in the user's home. (:issue:`1468`, :pull:`1473`).

Breaking changes
^^^^^^^^^^^^^^^^
Expand Down
2 changes: 1 addition & 1 deletion tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -450,7 +450,7 @@ def gather_session_data(threadsafe_data_dir, worker_id, xdoctest_namespace):
if worker_id in "master":
helpers.populate_testing_data(branch=helpers.TESTDATA_BRANCH)
else:
_default_cache_dir.mkdir(exist_ok=True)
_default_cache_dir.mkdir(exist_ok=True, parents=True)
test_data_being_written = FileLock(_default_cache_dir.joinpath(".lock"))
with test_data_being_written as fl:
# This flag prevents multiple calls from re-attempting to download testing data in the same pytest run
Expand Down
2 changes: 1 addition & 1 deletion xclim/testing/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ def _get(
if not local_file.is_file():
# This will always leave this directory on disk.
# We may want to add an option to remove it.
local_file.parent.mkdir(parents=True, exist_ok=True)
local_file.parent.mkdir(exist_ok=True, parents=True)

url = "/".join((github_url, "raw", branch, fullname.as_posix()))
logger.info(f"Fetching remote file: {fullname.as_posix()}")
Expand Down

0 comments on commit d53a5b6

Please sign in to comment.