Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for NumPy 2.0 in Neo Core #1490

Merged
merged 40 commits into from
Dec 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
a50e343
update core tests-check numpy 2.0
zm711 Jun 18, 2024
f9fb7ec
Merge branch 'master' into numpy-2-0
zm711 Aug 27, 2024
5cf94d4
Merge branch 'master' into numpy-2-0
zm711 Oct 1, 2024
9f0c35e
add back in 2.0 test
zm711 Oct 1, 2024
dec7667
change dependency for testing
zm711 Oct 1, 2024
088a26b
Merge branch 'master' into numpy-2-0
zm711 Oct 17, 2024
5b5c6c0
update exclusion list for 2.0
zm711 Oct 17, 2024
24f0e63
add python 3.13
zm711 Oct 17, 2024
3bcbe1c
oops
zm711 Oct 17, 2024
a22c61b
no 2.1 for now
zm711 Oct 17, 2024
5dc0dc8
fix 1.24.4
zm711 Oct 17, 2024
119e715
fix pyproject
zm711 Oct 17, 2024
b27f880
oops
zm711 Oct 17, 2024
c9113f6
fail-fast false
zm711 Oct 17, 2024
11d9d72
update io-test
zm711 Oct 17, 2024
0160830
spiketrain set due to new numpy string
zm711 Oct 17, 2024
10605e3
fix the casting of times
zm711 Oct 17, 2024
559b739
more tests for core
zm711 Oct 17, 2024
34771c7
try another action trick
zm711 Oct 17, 2024
08d9a2e
fix matrix call
zm711 Oct 17, 2024
18cf5ea
try limiting to 1.26.4 during install
zm711 Oct 17, 2024
f36daa8
last attempt to get action working
zm711 Oct 17, 2024
8e6c3d4
add numpy version there.
zm711 Oct 17, 2024
b67684b
try the .yml env file
zm711 Oct 17, 2024
7154a18
another attempt to get numpy to work.
zm711 Oct 18, 2024
8df5dae
add -y
zm711 Oct 18, 2024
d360787
remove - y
zm711 Oct 18, 2024
12cdec9
try not to download dhn_med_py
zm711 Oct 18, 2024
b6d6bb8
update action version number
zm711 Oct 18, 2024
a3b753b
don't update miniconda version
zm711 Oct 18, 2024
2743cdf
Merge branch 'master' into numpy-2-0
zm711 Oct 18, 2024
f4545c2
update rtd yaml
zm711 Oct 18, 2024
4b0fc8e
revert to 3.11, errors with 3.12
zm711 Oct 18, 2024
7d83da0
Merge branch 'master' into numpy-2-0
zm711 Oct 20, 2024
5930673
Merge branch 'master' into numpy-2-0
zm711 Oct 21, 2024
d8edd68
fix pyproject after release
zm711 Oct 21, 2024
5209596
Merge branch 'master' into numpy-2-0
zm711 Oct 21, 2024
9a1962b
Merge branch 'master' into numpy-2-0
zm711 Oct 28, 2024
7449082
Merge branch 'master' into numpy-2-0
zm711 Nov 25, 2024
b078766
Merge branch 'master' into numpy-2-0
zm711 Dec 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 20 additions & 5 deletions .github/workflows/core-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,28 +25,43 @@ jobs:
fail-fast: true
matrix:
os: ["ubuntu-latest", "windows-latest", "macos-latest"]
python-version: ['3.9', '3.10', '3.11', '3.12']
numpy-version: ['1.22.4', '1.23.5', '1.24.1', '1.25.1', '1.26.4']
python-version: ['3.9', '3.10', '3.11', '3.12', '3.13']
numpy-version: ['1.22.4', '1.23.5', '1.24.4', '1.25.1', '1.26.4', '2.0.2','2.1']
# numpy 1.22: 3.10, 1.23: 3.11, 1.24: 3.11, 1.25: 3.11, 1.26: 3.12
exclude:
- python-version: '3.9'
numpy-version: '2.1'
- python-version: '3.11'
numpy-version: '1.22.4'
- python-version: '3.12'
numpy-version: '1.22.4'
- python-version: '3.12'
numpy-version: '1.23.5'
- python-version: '3.12'
numpy-version: '1.24.1'
numpy-version: '1.24.4'
- python-version: '3.12'
numpy-version: '1.25.1'
- python-version: '3.13'
numpy-version: '1.22.4'
- python-version: '3.13'
numpy-version: '1.23.5'
- python-version: '3.13'
numpy-version: '1.24.4'
- python-version: '3.13'
numpy-version: '1.25.1'
- python-version: '3.13'
numpy-version: '1.26.4'
- python-version: '3.13'
numpy-version: '2.0.2'

steps:
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Install numpy ${{ matrix.numpy-version }}
run: |
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/io-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,15 @@ jobs:
strategy:
fail-fast: true
matrix:
python-version: ['3.9', '3.11.9']
python-version: ['3.9', '3.12']
defaults:
# by default run in bash mode (required for conda usage)
run:
shell: bash -l {0}
steps:

- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Get current year-month
id: date
Expand All @@ -38,7 +38,7 @@ jobs:
run: |
echo "dataset_hash=$(git ls-remote https://gin.g-node.org/NeuralEnsemble/ephy_testing_data.git HEAD | cut -f1)" >> $GITHUB_OUTPUT

- uses: actions/cache/restore@v3
- uses: actions/cache/restore@v4
# Loading cache of ephys_testing_dataset
id: cache-datasets
with:
Expand Down
3 changes: 3 additions & 0 deletions environment_testing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,6 @@ channels:
dependencies:
- datalad
- pip
# temporary have this here for IO testing while we decide how to deal with
# external packages not 2.0 ready
- numpy=1.26.4
2 changes: 1 addition & 1 deletion neo/core/spiketrain.py
Original file line number Diff line number Diff line change
Expand Up @@ -979,7 +979,7 @@ def _merge_array_annotations(self, others, sorting=None):

omitted_keys_other = [
key
for key in np.unique([key for other in others for key in other.array_annotations])
for key in set([key for other in others for key in other.array_annotations])
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only actual change to code base. numpy 2.0 returns this as a np.str_ thing rather than as the actual string. Using set instead of np.unique returns the string both >2.0 and < 2.0 for NumPy. Maybe a slight performance hit, but probably not too bad since the annotations should be small either way and this is in a list and not array.

if key not in self.array_annotations
]

Expand Down
10 changes: 9 additions & 1 deletion neo/test/coretest/test_spiketrain.py
Original file line number Diff line number Diff line change
Expand Up @@ -313,6 +313,10 @@ def test__create_from_quantity_array(self):

def test__create_from_quantity_array_with_dtype(self):
times = np.arange(10, dtype="f4") * pq.ms
# this step is required for NumPy 2.0 which now casts to float64 in the case either value/array
# is float64 even if not necessary
# https://numpy.org/devdocs/numpy_2_0_migration_guide.html
times = times.astype(dtype="f4")
Comment on lines 315 to +319
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on discussion with Sam my comment may not have been clear enough. The problem is that
1* pq.ms is at baseline at 'float64' and as of numpy 2.0+ everything is up-cast to the bigger dtype even if not necessarily. So in old versions of our test times would remain a float32, but with new versions of numpy make it into a 'float64'. I did the astype to keep our original idea explict, but we could think of other rewrites of this test to maintain the dtype test.

t_start = 0.0 * pq.s
t_stop = 12.0 * pq.ms
train1 = SpikeTrain(times, t_start=t_start, t_stop=t_stop)
Expand Down Expand Up @@ -343,6 +347,10 @@ def test__create_from_quantity_array_no_start_stop_units(self):

def test__create_from_quantity_array_no_start_stop_units_with_dtype(self):
times = np.arange(10, dtype="f4") * pq.ms
# this step is required for NumPy 2.0 which now casts to float64 in the case either value/array
# is float64 even if not necessary
# https://numpy.org/devdocs/numpy_2_0_migration_guide.html
times = times.astype(dtype="f4")
t_start = 0.0
t_stop = 12.0
train1 = SpikeTrain(times, t_start=t_start, t_stop=t_stop)
Expand Down Expand Up @@ -1143,7 +1151,7 @@ def test_merge_multiple(self):
expected *= time_unit
sorting = np.argsort(expected)
expected = expected[sorting]
np.testing.assert_array_equal(result.times, expected)
np.testing.assert_array_equal(result.times.magnitude, expected.magnitude)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is necessary because as of numpy 2.0 because numpy assert_array_equal actually checks that the quantities are the same which leads to a units issue. If someone else understands this better please post here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is annoying @apdavison any comments ?


# Make sure array annotations are merged correctly
self.assertTrue("label" not in result.array_annotations)
Expand Down
8 changes: 4 additions & 4 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ version = "0.14.0.dev0"
authors = [{name = "Neo authors and contributors"}]
description = "Neo is a package for representing electrophysiology data in Python, together with support for reading a wide range of neurophysiology file formats"
readme = "README.rst"
requires-python = ">=3.9,<3.13" # 3.13 will require NumPy > 2.0 (Windows issue in CI)
requires-python = ">=3.9"
license = {text = "BSD 3-Clause License"}
classifiers = [
"Development Status :: 4 - Beta",
Expand All @@ -23,7 +23,7 @@ classifiers = [

dependencies = [
"packaging",
"numpy>=1.22.4,<2.0.0",
"numpy>=1.22.4",
"quantities>=0.16.1"
]

Expand All @@ -45,7 +45,7 @@ iocache = [
]

test = [
"dhn_med_py<2.0", # ci failing with 2.0 test future version when stable
# "dhn_med_py<2.0", # ci failing with 2.0 test future version when stable
"pytest",
"pytest-cov",
# datalad # this dependency is covered by conda (environment_testing.yml)
Expand Down Expand Up @@ -105,7 +105,7 @@ plexon2 = ["zugbruecke>=0.2; sys_platform!='win32'", "wenv; sys_platform!='win32
all = [
"coverage",
"coveralls",
"dhn_med_py<2.0", # ci failing with 2.0 test future version when stable
# "dhn_med_py<2.0", # ci failing with 2.0 test future version when stable
"h5py",
"igor2",
"ipython",
Expand Down
Loading