Add support for sparse arrays (pydata) to .varp and .obsp fields

[jkobject]

Please describe your wishes and possible alternatives to achieve the desired result.

scipy.sparse only supports 2D arrays. Whereas varp and obsp support N-D arrays.
Most of the time we would want them to be sparse.
But right now anndata prevent the varp and obsp fields to contain the pydata/sparse formats.

Not all but many features are available on these matrix formats. Are there missing features that would be required as of now that would prevent this integration?

[flying-sheep]

Ah, you’re talking about https://sparse.pydata.org/en/stable/.

We haven’t talked about this much yet, but it’s definitely worth discussing.

[jkobject]

Let me know if I can help since it doesn't seem like too complicated to add and would be worth it for some applications :)

If you can point me to some of files to be looking at for updates I can have a look

[ilan-gold]

@jkobject If I remember, our main stumbling block with this up until now has been the dependency on numba, which is too much for anndata.

But we could do this as an optional dep. In this case, we should think more broadly about a registration mechanism for choosing which array type gets returned from i/o. This would be helpful for GPU.

[ivirshup]

I would add that last I talked to the authors of pydata/sparse they were looking at a full rewrite. So it might be in a rocky state for a while.

[ilan-gold]

@ivirshup it loooks like the package has a backend for finch now...could be worth trying out actually...would be huge for array-api compat!