Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chunked writing of h5py.Dataset and zarr.Array #1624

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Merge branch 'main' into more-efficient-dense-writing

690b682
Select commit
Loading
Failed to load commit list.
Open

Chunked writing of h5py.Dataset and zarr.Array #1624

Merge branch 'main' into more-efficient-dense-writing
690b682
Select commit
Loading
Failed to load commit list.
Azure Pipelines / scverse.anndata failed Nov 15, 2024 in 17m 57s

Build #20241115.9 had test failures

Details

Tests

  • Failed: 3,401 (16.87%)
  • Passed: 13,992 (69.38%)
  • Other: 2,773 (13.75%)
  • Total: 20,166
Code coverage

  • 4676 of 6082 line covered (76.88%)

Annotations

Check failure on line 73759 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / scverse.anndata

Build log #L73759

Bash exited with code '1'.

Check failure on line 73845 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / scverse.anndata

Build log #L73845

Bash exited with code '1'.

Check failure on line 73835 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / scverse.anndata

Build log #L73835

Bash exited with code '1'.

Check failure on line 73839 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / scverse.anndata

Build log #L73839

Bash exited with code '1'.

Check failure on line 1 in anndata._core.sparse_dataset.sparse_dataset

See this annotation in the file changed.

@azure-pipelines azure-pipelines / scverse.anndata

anndata._core.sparse_dataset.sparse_dataset

607     Example
608     -------
609 
610     First we'll need a stored dataset:
611 
612     >>> import scanpy as sc
613     >>> import h5py
614     >>> from anndata.io import sparse_dataset
615     >>> from anndata.io import read_elem
616     >>> sc.datasets.pbmc68k_reduced().raw.to_adata().write_h5ad("pbmc.h5ad")
UNEXPECTED EXCEPTION: TypeError("h5py._hl.group.Group.create_dataset() got multiple values for keyword argument 'dtype'")
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/doctest.py", line 1368, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest anndata._core.sparse_dataset.sparse_dataset[4]>", line 1, in <module>
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_core/anndata.py", line 1865, in write_h5ad
    write_h5ad(
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/h5ad.py", line 90, in write_h5ad
    write_elem(f, "X", adata.X, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 488, in write_elem
    Writer(_REGISTRY).write_elem(store, k, elem, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py", line 248, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 355, in write_elem
    return write_func(store, k, elem, dataset_kwargs=dataset_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 71, in wrapper
    result = func(g, k, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 649, in write_sparse_compressed
    _writer.write_elem(
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py", line 248, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 355, in write_elem
    return write_func(store, k, elem, dataset_kwargs=dataset_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 71, in wrapper
    result = func(g, k, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 394, in write_basic
    f.create_dataset(k, data=elem, **dataset_kwargs, dtype=dtype)
TypeError: h5py._hl.group.Group.create_dataset() got multiple values for keyword argument 'dtype'
Error raised while writing key 'indptr' of <class 'h5py._hl.group.Group'> to /X
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_core/sparse_dataset.py:616: UnexpectedException
Raw output
607     Example
608     -------
609 
610     First we'll need a stored dataset:
611 
612     >>> import scanpy as sc
613     >>> import h5py
614     >>> from anndata.io import sparse_dataset
615     >>> from anndata.io import read_elem
616     >>> sc.datasets.pbmc68k_reduced().raw.to_adata().write_h5ad("pbmc.h5ad")
UNEXPECTED EXCEPTION: TypeError("h5py._hl.group.Group.create_dataset() got multiple values for keyword argument 'dtype'")
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/doctest.py", line 1368, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest anndata._core.sparse_dataset.sparse_dataset[4]>", line 1, in <module>
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_core/anndata.py", line 1865, in write_h5ad
    write_h5ad(
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/h5ad.py", line 90, in write_h5ad
    write_elem(f, "X", adata.X, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 488, in write_elem
    Writer(_REGISTRY).write_elem(store, k, elem, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py", line 248, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 355, in write_elem
    return write_func(store, k, elem, dataset_kwargs=dataset_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 71, in wrapper
    result = func(g, k, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 649, in write_sparse_compressed
    _writer.write_elem(
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py", line 248, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 355, in write_elem
    return write_func(store, k, elem, dataset_kwargs=dataset_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 71, in wrapper
    result = func(g, k, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 394, in write_basic
    f.create_dataset(k, data=elem, **dataset_kwargs, dtype=dtype)
TypeError: h5py._hl.group.Group.create_dataset() got multiple values for keyword argument 'dtype'
Error raised while writing key 'indptr' of <class 'h5py._hl.group.Group'> to /X
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_core/sparse_dataset.py:616: UnexpectedException

Check failure on line 1 in anndata._io.specs.registry.read_elem_as_dask

See this annotation in the file changed.

@azure-pipelines azure-pipelines / scverse.anndata

anndata._io.specs.registry.read_elem_as_dask

413     >>> import tempfile
414     >>> import anndata as ad
415     >>> import zarr
416 
417     >>> tmp_path = tempfile.gettempdir()
418     >>> zarr_path = tmp_path + "/adata.zarr"
419 
420     >>> adata = pbmc3k()
421     >>> adata.layers["dense"] = adata.X.toarray()
422     >>> adata.write_zarr(zarr_path)
UNEXPECTED EXCEPTION: TypeError("zarr.hierarchy.Group.create_dataset() got multiple values for keyword argument 'dtype'")
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/doctest.py", line 1368, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest anndata._io.specs.registry.read_elem_as_dask[8]>", line 1, in <module>
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_core/anndata.py", line 1928, in write_zarr
    write_zarr(store, self, chunks=chunks)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/zarr.py", line 50, in write_zarr
    write_dispatched(f, "/", adata, callback=callback, dataset_kwargs=ds_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/experimental/_dispatch_io.py", line 77, in write_dispatched
    writer.write_elem(store, key, elem, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py", line 248, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 356, in write_elem
    return self.callback(
           ^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/zarr.py", line 48, in callback
    func(s, k, elem, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 71, in wrapper
    result = func(g, k, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 277, in write_anndata
    _writer.write_elem(g, "X", adata.X, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py", line 248, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 356, in write_elem
    return self.callback(
           ^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/zarr.py", line 48, in callback
    func(s, k, elem, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 71, in wrapper
    result = func(g, k, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 649, in write_sparse_compressed
    _writer.write_elem(
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py", line 248, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 356, in write_elem
    return self.callback(
           ^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/zarr.py", line 48, in callback
    func(s, k, elem, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 71, in wrapper
    result = func(g, k, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site
Raw output
413     >>> import tempfile
414     >>> import anndata as ad
415     >>> import zarr
416 
417     >>> tmp_path = tempfile.gettempdir()
418     >>> zarr_path = tmp_path + "/adata.zarr"
419 
420     >>> adata = pbmc3k()
421     >>> adata.layers["dense"] = adata.X.toarray()
422     >>> adata.write_zarr(zarr_path)
UNEXPECTED EXCEPTION: TypeError("zarr.hierarchy.Group.create_dataset() got multiple values for keyword argument 'dtype'")
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/doctest.py", line 1368, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest anndata._io.specs.registry.read_elem_as_dask[8]>", line 1, in <module>
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_core/anndata.py", line 1928, in write_zarr
    write_zarr(store, self, chunks=chunks)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/zarr.py", line 50, in write_zarr
    write_dispatched(f, "/", adata, callback=callback, dataset_kwargs=ds_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/experimental/_dispatch_io.py", line 77, in write_dispatched
    writer.write_elem(store, key, elem, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py", line 248, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 356, in write_elem
    return self.callback(
           ^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/zarr.py", line 48, in callback
    func(s, k, elem, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 71, in wrapper
    result = func(g, k, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 277, in write_anndata
    _writer.write_elem(g, "X", adata.X, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py", line 248, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 356, in write_elem
    return self.callback(
           ^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/zarr.py", line 48, in callback
    func(s, k, elem, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 71, in wrapper
    result = func(g, k, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 649, in write_sparse_compressed
    _writer.write_elem(
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py", line 248, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 356, in write_elem
    return self.callback(
           ^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/zarr.py", line 48, in callback
    func(s, k, elem, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 71, in wrapper
    result = func(g, k, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site

Check failure on line 1 in anndata.experimental.merge.concat_on_disk

See this annotation in the file changed.

@azure-pipelines azure-pipelines / scverse.anndata

anndata.experimental.merge.concat_on_disk

515     ...         for data in r.iter_bytes():
516     ...             f.write(data)
517     ...     return out_path
518     >>> path_b_cells = get_cellxgene_data('a93eab58-3d82-4b61-8a2f-d7666dcdb7c4')
519     >>> path_fetal = get_cellxgene_data('d170ff04-6da0-4156-a719-f8e1bbefbf53')
520 
521     Now we can concatenate them on-disk:
522 
523     >>> import anndata as ad
524     >>> ad.experimental.concat_on_disk(
UNEXPECTED EXCEPTION: TypeError("h5py._hl.group.Group.create_dataset() got multiple values for keyword argument 'dtype'")
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/doctest.py", line 1368, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest anndata.experimental.merge.concat_on_disk[7]>", line 1, in <module>
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/experimental/merge.py", line 634, in concat_on_disk
    _write_concat_arrays(
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/experimental/merge.py", line 300, in _write_concat_arrays
    write_concat_sparse(
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/experimental/merge.py", line 226, in write_concat_sparse
    write_elem(
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 488, in write_elem
    Writer(_REGISTRY).write_elem(store, k, elem, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py", line 248, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 355, in write_elem
    return write_func(store, k, elem, dataset_kwargs=dataset_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 71, in wrapper
    result = func(g, k, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 649, in write_sparse_compressed
    _writer.write_elem(
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py", line 248, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 355, in write_elem
    return write_func(store, k, elem, dataset_kwargs=dataset_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 71, in wrapper
    result = func(g, k, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 394, in write_basic
    f.create_dataset(k, data=elem, **dataset_kwargs, dtype=dtype)
TypeError: h5py._hl.group.Group.create_dataset() got multiple values for keyword argument 'dtype'
Error raised while writing key 'indptr' of <class 'h5py._hl.group.Group'> to /X
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/experimental/merge.py:524: UnexpectedException
Raw output
515     ...         for data in r.iter_bytes():
516     ...             f.write(data)
517     ...     return out_path
518     >>> path_b_cells = get_cellxgene_data('a93eab58-3d82-4b61-8a2f-d7666dcdb7c4')
519     >>> path_fetal = get_cellxgene_data('d170ff04-6da0-4156-a719-f8e1bbefbf53')
520 
521     Now we can concatenate them on-disk:
522 
523     >>> import anndata as ad
524     >>> ad.experimental.concat_on_disk(
UNEXPECTED EXCEPTION: TypeError("h5py._hl.group.Group.create_dataset() got multiple values for keyword argument 'dtype'")
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/doctest.py", line 1368, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest anndata.experimental.merge.concat_on_disk[7]>", line 1, in <module>
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/experimental/merge.py", line 634, in concat_on_disk
    _write_concat_arrays(
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/experimental/merge.py", line 300, in _write_concat_arrays
    write_concat_sparse(
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/experimental/merge.py", line 226, in write_concat_sparse
    write_elem(
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 488, in write_elem
    Writer(_REGISTRY).write_elem(store, k, elem, dataset_kwargs=dataset_kwargs)
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py", line 248, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 355, in write_elem
    return write_func(store, k, elem, dataset_kwargs=dataset_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 71, in wrapper
    result = func(g, k, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 649, in write_sparse_compressed
    _writer.write_elem(
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py", line 248, in func_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 355, in write_elem
    return write_func(store, k, elem, dataset_kwargs=dataset_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py", line 71, in wrapper
    result = func(g, k, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/methods.py", line 394, in write_basic
    f.create_dataset(k, data=elem, **dataset_kwargs, dtype=dtype)
TypeError: h5py._hl.group.Group.create_dataset() got multiple values for keyword argument 'dtype'
Error raised while writing key 'indptr' of <class 'h5py._hl.group.Group'> to /X
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/experimental/merge.py:524: UnexpectedException

Check failure on line 1 in test_awkward_io_view

See this annotation in the file changed.

@azure-pipelines azure-pipelines / scverse.anndata

test_awkward_io_view

TypeError: h5py._hl.group.Group.create_dataset() got multiple values for keyword argument 'dtype'
Error raised while writing key 'indptr' of <class 'h5py._hl.group.Group'> to /X
Raw output
tmp_path = PosixPath('/tmp/pytest-of-vsts/pytest-0/test_awkward_io_view0')

    def test_awkward_io_view(tmp_path):
        """Check that views are converted to actual arrays on save, i.e. the _view_args and __list__ parameters are removed"""
        adata = gen_adata((3, 3), varm_types=(), obsm_types=(AwkArray,), layers_types=())
    
        v = adata[1:]
        adata_path = tmp_path / "adata.h5ad"
>       v.write_h5ad(adata_path)

tests/test_awkward.py:259: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_core/anndata.py:1865: in write_h5ad
    write_h5ad(
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/h5ad.py:90: in write_h5ad
    write_elem(f, "X", adata.X, dataset_kwargs=dataset_kwargs)
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py:488: in write_elem
    Writer(_REGISTRY).write_elem(store, k, elem, dataset_kwargs=dataset_kwargs)
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py:248: in func_wrapper
    return func(*args, **kwargs)
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py:355: in write_elem
    return write_func(store, k, elem, dataset_kwargs=dataset_kwargs)
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py:71: in wrapper
    result = func(g, k, *args, **kwargs)
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/methods.py:649: in write_sparse_compressed
    _writer.write_elem(
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/utils.py:248: in func_wrapper
    return func(*args, **kwargs)
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py:355: in write_elem
    return write_func(store, k, elem, dataset_kwargs=dataset_kwargs)
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/registry.py:71: in wrapper
    result = func(g, k, *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

f = <Closed HDF5 group>, k = '/X/indptr', elem = array([0, 3, 4], dtype=int32)

    @_REGISTRY.register_write(H5Group, views.ArrayView, IOSpec("array", "0.2.0"))
    @_REGISTRY.register_write(H5Group, np.ndarray, IOSpec("array", "0.2.0"))
    @_REGISTRY.register_write(H5Group, np.ma.MaskedArray, IOSpec("array", "0.2.0"))
    @_REGISTRY.register_write(ZarrGroup, views.ArrayView, IOSpec("array", "0.2.0"))
    @_REGISTRY.register_write(ZarrGroup, np.ndarray, IOSpec("array", "0.2.0"))
    @_REGISTRY.register_write(ZarrGroup, np.ma.MaskedArray, IOSpec("array", "0.2.0"))
    @_REGISTRY.register_write(ZarrGroup, ZarrArray, IOSpec("array", "0.2.0"))
    @_REGISTRY.register_write(ZarrGroup, H5Array, IOSpec("array", "0.2.0"))
    def write_basic(
        f: GroupStorageType,
        k: str,
        elem: views.ArrayView | np.ndarray | h5py.Dataset | np.ma.MaskedArray | ZarrArray,
        *,
        _writer: Writer,
        dataset_kwargs: Mapping[str, Any] = MappingProxyType({}),
    ):
        """Write methods which underlying library handles natively."""
        dtype = dataset_kwargs.get("dtype", elem.dtype)
>       f.create_dataset(k, data=elem, **dataset_kwargs, dtype=dtype)
E       TypeError: h5py._hl.group.Group.create_dataset() got multiple values for keyword argument 'dtype'
E       Error raised while writing key 'indptr' of <class 'h5py._hl.group.Group'> to /X

/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages/anndata/_io/specs/methods.py:394: TypeError