Compatibility and tests against Numpy 2.0.0rc1

[dhomeier]

Description

Installing numpy dev in CI is currently blocked by various upstream packages pinned to numpy < 2.0; adding those to the developer versions.
This would update both dev jobs to numpy>=0.0.dev0; restrict to a new env npdev instead?

• pandas has no compatible wheel and a hard dependency on numpy < 2.0.
• contourpy has no compatible wheel and a hard dependency on numpy < 2.0.
• shapely has no compatible wheel for numpy < 2.0 (Wheel that is compatible with numpy 2.0.dev ?  shapely/shapely#1949), build depends on libgeos.
• fast-histogram has no compatible wheel.
• casa-formats-io has no compatible wheel and a pinned dependency on numpy < 2.0.

Only workaround for the pins seems to be to install with --no-deps (pandas-dev/pandas#55488) which in turn requires every single test dependency to be pulled in explicitly; if anyone has an idea how to run a first pass without --no-deps an then a second pip install to override the numpy pin, that could be simplified a bit.

Building shapely seems to require compiling its own libgeos installation per its
install script; unless I find a way to get an apt install for that, this is a showstopper for now.
(It seems to be available, but would require pointing the CI to fetch from an alternative repo).

[pllim]

For pandas and matplotlib, I did a hack in my packages to let them downgrade numpy first but then force numpy dev reinstall right before test start running and ignore the pip warning. It's not pretty but easier than --no-deps .

shapely is indeed more troublesome. They never replied to my request.

[dhomeier]

Running now with numpy 2.0.0rc1, astropy 6.1.0rc1 and scipy 1.13.0; fast-histogram still needs its own build (see astrofrog/fast-histogram#86 (comment)); have not checked for released versions of other dependencies yet.
The actual glue test failures all seem to be on np.array(copy=False) like
https://github.com/glue-viz/glue/actions/runs/8709773487/job/23890261129?pr=2471#step:10:311
The fix looks straightforward enough, going to look into that next.

[dhomeier]

@astrofrog I have used a simplified version of the Scipy fix, since we should not need to worry about earlier numpy 2.0.0dev versions that did not support copy=None yet.
The new casa-formats-io wheel on PyPI still depends on and was built against numpy 1.26 – I could not follow the cibuildwheel process entirely, but it is also testing against that:
https://github.com/radio-astro-tools/casa-formats-io/actions/runs/8720397145/job/23921900123#step:7:371
perhaps need to remove the python_version<'3.9' case from pyproject.toml altogether?

Shapely has a new release built on shapely/shapely#2033 but seems to suffer the same problems with wheels for older Pythons being pulled in. – may just have had some delays in uploading all wheels, so for all other dependencies the current releases or candidates now work. 🚀

Before merging we should decide which 0.0.dev0 versions to keep for testing in the dev env (numpy, scipy, astropy?)

Don't have a way to debug the pandas-related Windows failures.

[astrofrog]

@dhomeier - are you sure it's still not working with casa-formats-io v0.3.0? There is a devdeps job there that shows that it works fine with numpy 2.0.0rc1 at runtime: https://github.com/radio-astro-tools/casa-formats-io/actions/runs/8720397149/job/23921906624

[astrofrog]

Can you double check that v0.3.0 works now?

[dhomeier]

I've tried to pip install on macOS several times now, always getting a numpy 1.26 version

[dhomeier]

Maybe a caching problem just as with shapely; seems to be running here as well, but then the installed versions of other deps don't make any sense with what is in tox.ini now. Rebasing to try to get this sorted...

[pllim]

Would this help?

Suggested change

	dev: python -I -m pip install -v
	dev: python -I -m pip install -v --pre

[dhomeier]

Just removed this as we should no longer need any git installs; the >=0.0.dev0 versions apparently install without it 🤪

[dhomeier]

https://github.com/glue-viz/glue/actions/runs/8723259903/job/23931144239?pr=2471 looks OK now

[astrofrog]

@dhomeier is this ok to merge? (If so feel free to!)

[dhomeier]

Yes, think so – I made on run with pandas dev to check the Windows failures (still persisted), but don't think we need that in general; thanks!