Update Scanpy to 1.10, preparing for GTA

[pavanvidem]

FOR CONTRIBUTOR:

• I have read the CONTRIBUTING.md document and this tool is appropriate for the tools-iuc repo.
• License permits unrestricted use (educational + commercial)
• This PR adds a new tool or tool collection
• This PR updates an existing tool or tool collection
• This PR does something else (explain below)

[bgruening]

@pavanvidem not sure if you have seen it, but we also need a rebase here.

[nilchia]

mnnpy conda recipe needs update

[bgruening]

Please rebase this PR

[bgruening]

What do you mean with mnnoy update? What is broken?

[nilchia]

All tests pass now :D
I'll rebase the PR and resolve the conflicts next.

[nilchia]

I removed sc.pl.dpt_groups_pseudotime() because it has problem in upstream source code.
It is incompatible with pandas version I think. The way scanpy tries to index a pandas.series object is not supported by pandas.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nilchia/miniconda3/envs/scanpy/lib/python3.12/site-packages/legacy_api_wrap/__init__.py", line 80, in fn_compatible
    return fn(*args_all, **kw)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/nilchia/miniconda3/envs/scanpy/lib/python3.12/site-packages/scanpy/plotting/_tools/__init__.py", line 276, in dpt_groups_pseudotime
    timeseries_subplot(
  File "/home/nilchia/miniconda3/envs/scanpy/lib/python3.12/site-packages/scanpy/plotting/_utils.py", line 140, in timeseries_subplot
    X = X[:, None]
        ~^^^^^^^^^
  File "/home/nilchia/miniconda3/envs/scanpy/lib/python3.12/site-packages/pandas/core/series.py", line 1153, in __getitem__
    return self._get_with(key)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/nilchia/miniconda3/envs/scanpy/lib/python3.12/site-packages/pandas/core/series.py", line 1163, in _get_with
    return self._get_values_tuple(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nilchia/miniconda3/envs/scanpy/lib/python3.12/site-packages/pandas/core/series.py", line 1203, in _get_values_tuple
    disallow_ndim_indexing(result)
  File "/home/nilchia/miniconda3/envs/scanpy/lib/python3.12/site-packages/pandas/core/indexers/utils.py", line 341, in disallow_ndim_indexing
    raise ValueError(
ValueError: Multi-dimensional indexing (e.g. obj[:, None]) is no longer supported. Convert to a numpy array before indexing instead.
>>> 

[bgruening]

Try using numpy <2.0. Maybe that helps.

[nilchia]

my package versions are these:

numpy                     1.26.4          py312heda63a1_0    conda-forge
pandas                    2.2.2           py312h1d6d2e6_1    conda-forge
scanpy                    1.10.2             pyhd8ed1ab_0    conda-forge

[nilchia]

I did rebase and resolved conflicts locally but still, here it said there was a conflict :(

I reset back to my last commit.

[nilchia]

The tests added from #6113 need a proper input file I guess.
error:

KeyError: "Could not find keys '['C1orf54', 'CLEC9A', 'CPNE3', 'DNASE1L3', 'HSP90B1', 'ID2', 'IGLL1', 'IRF8', 'LGALS1', 'NASP', 'PCNA', 'PPIB', 'RNASE6', 'SEC11C', 'SNX3', 'TK1']' in columns of `adata.obs` or in adata.var_names."

[nilchia]

Please don't merge. I'll add h5 assert.

[nilchia]

I added the h5 assert.
However, for some functions, this assertion is not a good test because the function does not add any layer or key to the h5ad file but only changes the contents of the keys.
Functions:
pp.downsample_counts
filter_marker
pp.sqrt
external.pp.magic 'all_genes'
pp.regress_out
pp.combat

[nilchia]

in filter.xml, the real output of filter_marker is the tabular file and not the adata, right?

[nilchia]

.. ERROR: Error '403 Client Error: Forbidden for url: https://www.biorxiv.org/content/10.1101/298430v1' accessing https://www.biorxiv.org/content/10.1101/298430v1

What is wrong with the link? It was the same error with the previous link.

[bgruening]

No idea, but the preprint is published by now and you should link the final paper https://www.nature.com/articles/nbt.4314

[pavanvidem]

in filter.xml, the real output of filter_marker is the tabular file and not the adata, right?

Yes, it is is. Good catch! It also updates the anndata but tabular file is the actual output to test. Thanks!

[pavanvidem]

I added the h5 assert. However, for some functions, this assertion is not a good test because the function does not add any layer or key to the h5ad file but only changes the contents of the keys. Functions: pp.downsample_counts filter_marker pp.sqrt external.pp.magic with t=-1 pp.regress_out pp.combat

Yes, all those functions only update the count matrix, not the annotations. If we don't want to save the test data then check whether .X exists which should be there. I don't know what is a good test here.

[nilchia]

I added test for those. Is it good?

[pavanvidem]

Here we are writing anndata info to a file and then to stdout. So for all tests where there are no changes in the annotations, you can also check the matrix size in stdout. Something like:

<assert_stdout>
    <has_text_matching expression="336 × 11"/>
</assert_stdout>

[nilchia]

Do you mean instead of the min() max() ... ?
But for those functions the matrix size does not change.
Only the values of adata.X is changed.

I used this assert for filter_cells/genes and downsample that the data size changes

[nilchia]

@pavanvidem @bgruening
Should I add/modify anything here? or is it ready to merge?

[pavanvidem]

Can you please check if this was fixed? #6302

Otherwise, good to merge.

[nilchia]

Actually, I didn't understand what was the issue there. Did the config file have a problem? or the adata was not defined?

[bgruening]

NameError: name 'adata' is not defined, can you check if you still have this variable somewhere and if you have a test for this?

So many things have changed, I guess we can proceed.

[nilchia]

Almost all functions use this variable. I guess we don't have any problems with that.

[pavanvidem]

Agree. Let's proceed. The error was in the plot tool. None of the functions in the plot tool change the anndata object. So it should not affect the analysis. The worst thing that can happen is that no plot is generated which we can investigate later.

[bgruening]

Awesome!

[pavanvidem]

Thanks, @nilchia and @bgruening for helping out!!