Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Add tests for reading from S3 bucket with references #252

Open
rly opened this issue Dec 19, 2024 · 0 comments
Open

[Feature]: Add tests for reading from S3 bucket with references #252

rly opened this issue Dec 19, 2024 · 0 comments
Assignees
Labels
category: enhancement improvements of code or code behavior
Milestone

Comments

@rly
Copy link
Contributor

rly commented Dec 19, 2024

What would you like to see added to HDMF-ZARR?

See #238 and NeurodataWithoutBorders/nwb_benchmarks#77

Error:

For parameters: 's3://dandiarchive/zarr/18e75d22-f527-4051-a4c8-c7e0f1e7dad1/'
             . Using tshark at: /Applications/Wireshark.app/Contents/MacOS/tshark
             Traceback (most recent call last):
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv/benchmark.py", line 68, in <module>
                 main()
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv/benchmark.py", line 60, in main
                 commands[mode](args)
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv_runner/run.py", line 72, in _run
                 result = benchmark.do_run()
                          ^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv_runner/benchmarks/_base.py", line 661, in do_run
                 return self.run(*self._current_params)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv_runner/benchmarks/track.py", line 64, in run
                 return self.func(*param)
                        ^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv_runner/benchmarks/mark.py", line 105, in wrapper
                 return func(*args, **kwargs)
                        ^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv_runner/benchmarks/mark.py", line 164, in wrapper
                 return func(*args, **kwargs)
                        ^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/nwb_benchmarks/src/nwb_benchmarks/benchmarks/network_tracking_remote_file_reading.py", line 327, in track_network_activity_during_read
                 self.nwbfile, self.io = read_zarr_nwbfile(s3_url=s3_url, mode="r")
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/nwb_benchmarks/src/nwb_benchmarks/core/_streaming.py", line 264, in read_zarr_nwbfile
                 nwbfile = io.read()
                           ^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/hdmf/utils.py", line 668, in func_call
                 return func(args[0], **pargs)
                        ^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/hdmf/backends/io.py", line 56, in read
                 f_builder = self.read_builder()
                             ^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/hdmf/utils.py", line 668, in func_call
                 return func(args[0], **pargs)
                        ^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/hdmf-zarr/src/hdmf_zarr/backend.py", line 1467, in read_builder
                 f_builder = self.__read_group(self.__file, ROOT_NAME)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/hdmf-zarr/src/hdmf_zarr/backend.py", line 1545, in __read_group
                 sub_builder = self.__read_group(sub_group, sub_name)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/hdmf-zarr/src/hdmf_zarr/backend.py", line 1545, in __read_group
                 sub_builder = self.__read_group(sub_group, sub_name)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/hdmf-zarr/src/hdmf_zarr/backend.py", line 1554, in __read_group
                 self.__read_links(zarr_obj=zarr_obj, parent=ret)
               File "/Users/rly/Documents/NWB/hdmf-zarr/src/hdmf_zarr/backend.py", line 1573, in __read_links
                 target_name, target_zarr_obj = self.resolve_ref(link)
                                                ^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/hdmf-zarr/src/hdmf_zarr/backend.py", line 819, in resolve_ref
                 target_zarr_obj = self.__open_file_consolidated(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/hdmf-zarr/src/hdmf_zarr/backend.py", line 566, in __open_file_consolidated
                 return zarr.open_consolidated(
                        ^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/zarr/convenience.py", line 1345, in open_consolidated
                 store = normalize_store_arg(
                         ^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/zarr/storage.py", line 201, in normalize_store_arg
                 return normalize_store(store, storage_options, mode)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/zarr/storage.py", line 173, in _normalize_store_arg_v2
                 raise ValueError("storage_options passed with non-fsspec path")
             ValueError: storage_options passed with non-fsspec path

Is your feature request related to a problem?

No response

What solution would you like?

Add tests for this

Do you have any interest in helping implement the feature?

Yes.

@rly rly added the category: enhancement improvements of code or code behavior label Dec 19, 2024
@rly rly added this to the Future milestone Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: enhancement improvements of code or code behavior
Projects
None yet
Development

No branches or pull requests

2 participants