Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extracting and reshaping metadata from dynamic paths #377

Merged
merged 64 commits into from
Apr 24, 2023
Merged
Changes from 54 commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
a461719
add code for extracting and reshaping metadata from dynamic paths
bendichter Mar 17, 2023
c12b389
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 17, 2023
3ab6c3f
clean up a bit
bendichter Mar 17, 2023
52e5da6
Merge remote-tracking branch 'origin/dynamic_paths' into dynamic_paths
bendichter Mar 17, 2023
54c8193
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 17, 2023
a78c0ec
added some docstrings
bendichter Mar 17, 2023
ad827e7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 17, 2023
edce0ab
fix import error
bendichter Mar 17, 2023
feef913
Merge remote-tracking branch 'origin/dynamic_paths' into dynamic_paths
bendichter Mar 17, 2023
7fb1c62
Merge branch 'main' into dynamic_paths
bendichter Mar 22, 2023
7d8d216
fix tests to handle lists of arbitrary order
bendichter Mar 22, 2023
550e7f7
Merge remote-tracking branch 'origin/dynamic_paths' into dynamic_paths
bendichter Mar 22, 2023
90fd792
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 22, 2023
1e844f9
fix typo
bendichter Mar 22, 2023
7d96877
Merge remote-tracking branch 'origin/dynamic_paths' into dynamic_paths
bendichter Mar 22, 2023
3cbd2f5
a bit of renaming
bendichter Mar 22, 2023
363ed88
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 22, 2023
1be8ca1
add some comments
bendichter Mar 22, 2023
3507f54
Merge remote-tracking branch 'origin/dynamic_paths' into dynamic_paths
bendichter Mar 22, 2023
7968e0d
generalize paths for windows
bendichter Mar 22, 2023
b6e376a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 22, 2023
f8c8624
create LocalPathExpander and GoogleDrivePathExpander
bendichter Mar 23, 2023
1f5d68c
Merge remote-tracking branch 'origin/dynamic_paths' into dynamic_paths
bendichter Mar 23, 2023
ef073f9
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 23, 2023
1785efa
Fix tests and code for latest path_expansion exports and arguments
garrettmflynn Apr 7, 2023
622e507
Update src/neuroconv/utils/path_expansion.py
bendichter Apr 8, 2023
cabd7a6
Update src/neuroconv/utils/path_expansion.py
bendichter Apr 8, 2023
93175fb
Merge branch 'main' into dynamic_paths
garrettmflynn Apr 10, 2023
593e0c6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 10, 2023
14570cb
Return absolute paths
garrettmflynn Apr 10, 2023
2e69513
Merge branch 'dynamic_paths' of https://github.com/catalystneuro/neur…
garrettmflynn Apr 10, 2023
2d80dfe
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 10, 2023
b4ef6aa
Merge branch 'main' into dynamic_paths
CodyCBakerPhD Apr 10, 2023
4cfeb28
Merge branch 'main' into dynamic_paths
CodyCBakerPhD Apr 13, 2023
86e6922
Update src/neuroconv/utils/path_expansion.py
bendichter Apr 14, 2023
d84755a
add DeelDict, rmv google drive path expander
bendichter Apr 14, 2023
e020910
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 14, 2023
7306a65
Merge branch 'deep_dict' into dynamic_paths
CodyCBakerPhD Apr 14, 2023
4935234
Update src/neuroconv/utils/path_expansion.py
bendichter Apr 17, 2023
8aad555
Update src/neuroconv/utils/path_expansion.py
bendichter Apr 17, 2023
dc9f934
Merge branch 'main' into dynamic_paths
bendichter Apr 17, 2023
db81aa6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 17, 2023
248678e
fix tests for absolute paths
bendichter Apr 17, 2023
5ff50ec
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 17, 2023
d5aa48d
rmv unused import
bendichter Apr 17, 2023
26a0b2a
Merge remote-tracking branch 'origin/dynamic_paths' into dynamic_paths
bendichter Apr 17, 2023
861b45b
Update src/neuroconv/utils/path_expansion.py
bendichter Apr 19, 2023
c95ca07
Update tests/test_minimal/test_utils/test_expand_paths.py
bendichter Apr 19, 2023
33a0786
Update path_expansion.py
CodyCBakerPhD Apr 19, 2023
0318e86
move to tools
CodyCBakerPhD Apr 20, 2023
9ae99b8
Merge branch 'main' into dynamic_paths
CodyCBakerPhD Apr 20, 2023
8a8aac4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 20, 2023
c5f1bb0
fix import testts
CodyCBakerPhD Apr 20, 2023
ddccef5
Update src/neuroconv/tools/path_expansion.py
bendichter Apr 24, 2023
e5a4a64
Update src/neuroconv/tools/path_expansion.py
bendichter Apr 24, 2023
7352a3a
Update src/neuroconv/tools/path_expansion.py
bendichter Apr 24, 2023
ef4fffe
Update src/neuroconv/tools/path_expansion.py
bendichter Apr 24, 2023
acc7c50
Update path_expansion.py
bendichter Apr 24, 2023
fd03d19
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 24, 2023
46564d1
Update src/neuroconv/tools/path_expansion.py
CodyCBakerPhD Apr 24, 2023
aa52a0f
Merge branch 'main' into dynamic_paths
CodyCBakerPhD Apr 24, 2023
5bb4819
Update imports.py
CodyCBakerPhD Apr 24, 2023
c36e229
fix tests to new name
CodyCBakerPhD Apr 24, 2023
5f12068
Merge branch 'main' into dynamic_paths
CodyCBakerPhD Apr 24, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -29,6 +29,7 @@
* Adds `MockRecordingInterface` as a general testing mechanism for ecephys interfaces [PR #395](https://github.com/catalystneuro/neuroconv/pull/395).
* `metadata` returned by `DataInterface.get_metadata()` is now a `DeepDict` object, making it easier to add and adjust metadata. [PR #404](https://github.com/catalystneuro/neuroconv/pull/404).
* The `OpenEphysLegacyRecordingInterface` is now extracts the `session_start_time` in `get_metadata()` from `Neo` (`OpenEphysRawIO`) and does not depend on `pyopenephys` anymore. [PR #410](https://github.com/catalystneuro/neuroconv/pull/410)
* Added `expand_paths`. [PR #377](https://github.com/catalystneuro/neuroconv/pull/377)

### Testing
* The tests for `automatic_dandi_upload` now follow up-to-date DANDI validation rules for file name conventions. [PR #310](https://github.com/catalystneuro/neuroconv/pull/310)
1 change: 1 addition & 0 deletions requirements-minimal.txt
Original file line number Diff line number Diff line change
@@ -11,3 +11,4 @@ psutil>=5.8.0
tqdm>=4.60.0
dandi>=0.46.2
pandas
parse
1 change: 1 addition & 0 deletions src/neuroconv/tools/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
from .importing import get_package
from .path_expansion import LocalPathExpander
82 changes: 82 additions & 0 deletions src/neuroconv/tools/path_expansion.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
import abc
import os
from pathlib import Path
from typing import Iterable, List, Union

from parse import parse
from pydantic import DirectoryPath, FilePath

from ..utils import DeepDict


class AbstractPathExpander(abc.ABC):
def extract_metadata(self, folder, format_: str):
format_ = format_.replace("\\", os.sep) # Actual character is a single back-slash; first is an escape for that
format_ = format_.replace("/", os.sep) # our f-string uses '/' to communicate os-independent separators
for filepath in self.list_directory(folder):
result = parse(format_, filepath)
if result:
yield filepath, result.named
bendichter marked this conversation as resolved.
Show resolved Hide resolved

@abc.abstractmethod
def list_directory(self, folder: DirectoryPath) -> Iterable[FilePath]:
bendichter marked this conversation as resolved.
Show resolved Hide resolved
"""
List all folders and files in a directory recursively.

Parameters
----------
folder : FilePath or DirectoryPath
The base folder whose contents will be iterated recursively.

Yields
-------
sub_paths : iterable of strings
Generator that yields all sub-paths of file and folders from the common root `folder`.
"""
pass

def expand_paths(self, source_data_spec: dict) -> List[DeepDict]:
bendichter marked this conversation as resolved.
Show resolved Hide resolved
"""
Match paths in a directory to specs and extract metadata from the paths.

Parameters
----------
source_data_spec : dict
Source spec.

Returns
-------
deep_dicts : list of DeepDict objects

Examples
--------
>>> path_expander.expand_paths(
... dict(
... spikeglx=dict(
... folder="source_folder",
... paths=dict(
... file_path="sub-{subject_id}/sub-{subject_id}_ses-{session_id}"
... )
... )
... )
... )
"""
out = DeepDict()
for interface, source_data in source_data_spec.items():
for path_type in ("file_path", "folder_path"):
if path_type in source_data:
for path, metadata in self.extract_metadata(source_data["folder"], source_data[path_type]):
key = tuple(sorted(metadata.items()))
out[key]["source_data"][interface][path_type] = os.path.join(
source_data["folder"], path
) # return the absolute path
if "session_id" in metadata:
out[key]["metadata"]["NWBFile"]["session_id"] = metadata["session_id"]
if "subject_id" in metadata:
out[key]["metadata"]["Subject"]["subject_id"] = metadata["subject_id"]
return list(dict(out).values())


class LocalPathExpander(AbstractPathExpander):
def list_directory(self, folder: Union[FilePath, DirectoryPath]) -> Iterable[str]:
return (str(path.relative_to(folder)) for path in Path(folder).rglob("*"))
CodyCBakerPhD marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 1 addition & 1 deletion src/neuroconv/utils/__init__.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
from .checks import calculate_regular_series_rate
from .dict import (
DeepDict,
append_replace_dict_in_list,
dict_deep_update,
exist_dict_in_list,
load_dict_from_file,
)
from .globbing import decompose_f_string, parse_f_string
from .json_schema import (
NWBMetaDataEncoder,
fill_defaults,
1 change: 1 addition & 0 deletions src/neuroconv/utils/dict.py
Original file line number Diff line number Diff line change
@@ -3,6 +3,7 @@
import warnings
from collections import defaultdict
from copy import deepcopy
from ctypes import Union
from pathlib import Path
from typing import Any, Optional, Union

89 changes: 0 additions & 89 deletions src/neuroconv/utils/globbing.py

This file was deleted.

61 changes: 61 additions & 0 deletions tests/test_minimal/test_tools/test_expand_paths.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
import os
from pathlib import Path

from neuroconv.tools import LocalPathExpander


def test_expand_paths(tmpdir):
expander = LocalPathExpander()

# set up directory for parsing
base = Path(tmpdir)
for subject_id in ("001", "002"):
Path.mkdir(base / f"sub-{subject_id}")
for session_id in ("101", "102"):
Path.mkdir(base / f"sub-{subject_id}" / f"session_{session_id}")
(base / f"sub-{subject_id}" / f"session_{session_id}" / "abc").touch()
(base / f"sub-{subject_id}" / f"session_{session_id}" / "xyz").touch()

# run path parsing
out = expander.expand_paths(
dict(
aa=dict(folder=base, file_path="sub-{subject_id:3}/session_{session_id:3}/abc"),
bb=dict(folder=base, file_path="sub-{subject_id:3}/session_{session_id:3}/xyz"),
),
)

expected = [
{
"source_data": {
"aa": {"file_path": str(base / "sub-002" / "session_101" / "abc")},
"bb": {"file_path": str(base / "sub-002" / "session_101" / "xyz")},
},
"metadata": {"NWBFile": {"session_id": "101"}, "Subject": {"subject_id": "002"}},
},
{
"source_data": {
"aa": {"file_path": str(base / "sub-002" / "session_102" / "abc")},
"bb": {"file_path": str(base / "sub-002" / "session_102" / "xyz")},
},
"metadata": {"NWBFile": {"session_id": "102"}, "Subject": {"subject_id": "002"}},
},
{
"source_data": {
"aa": {"file_path": str(base / "sub-001" / "session_101" / "abc")},
"bb": {"file_path": str(base / "sub-001" / "session_101" / "xyz")},
},
"metadata": {"NWBFile": {"session_id": "101"}, "Subject": {"subject_id": "001"}},
},
{
"source_data": {
"aa": {"file_path": str(base / "sub-001" / "session_102" / "abc")},
"bb": {"file_path": str(base / "sub-001" / "session_102" / "xyz")},
},
"metadata": {"NWBFile": {"session_id": "102"}, "Subject": {"subject_id": "001"}},
},
]

# test results
for x in out:
assert x in expected
assert len(out) == len(expected)
91 changes: 0 additions & 91 deletions tests/test_minimal/test_utils/test_globbing_utils.py

This file was deleted.