Scikit build core (#2121)

- Replaces `scikit-build` with `scikit-build-core`, which is beta-ish.
- Delete stale `setup.py`
- Create `scripts/test_executables.sh`, with a `modcc` and `a-b-c` test.
Call script in all workflows (CMake|Pip|Spack).
- Suppress setting `march` and `mtune` to `native` on versions of Apple
Clang <15. Only explicit targets are allowed, but since Apple Clang v15
`native` _is_ valid, and we only really use Apple Clang for the
universal Python wheels, this seems an acceptably simple solution.

Downsides
1.
https://scikit-build-core.readthedocs.io/en/latest/configuration.html#dynamic-metadata-wip
Looks like we cannot use `VERSION` to populate the `pyroject.toml`
version string, so it is duplicated.
2. Writing outside of the Python package dir, therefore not copy a-b-c
and modcc to /bin. Workaround is to make them callable through a
function, and bind the function to a script. There is a way to write
"scripts" (i.e. executables) outside of the package dir, but then
headers and such are still not copied. We could deduce the correct paths
for all outputs (libraries, documentation) from the env var, but that
seems no less brittle than the proposed solution, used by the CMake
project itself. See
scikit-build/scikit-build-core#194 (comment)

Fixes #2109

---------

Co-authored-by: Simon Frasch <[email protected]>

.github/workflows/release.yml

@@ -13,7 +13,7 @@ on:
 jobs:
  get_timestamp:
  name: Prep VERSION
- runs-on: ubuntu-20.04
+ runs-on: ubuntu-latest
 
  steps:
  - name: Get Arbor
@@ -34,7 +34,7 @@ jobs:
  runs-on: ${{ matrix.os }}
  strategy:
  matrix:
- os: [ubuntu-20.04, macos-latest]
+ os: [ubuntu-latest, macos-latest]
 
  steps:
  - name: Get Arbor
@@ -61,9 +61,11 @@ jobs:
  build_sdist:
  name: Build sdist
  needs: get_timestamp
- runs-on: ubuntu-20.04
+ runs-on: ubuntu-latest
 
  steps:
+ - name: Update pip and setup venv
+ run: python -m pip install --upgrade pip && python -m venv ~/env && . ~/env/bin/activate && echo PATH=$PATH >> $GITHUB_ENV
  - name: Get packages
  run: python3 -m pip install build
  - name: Get Arbor
@@ -86,14 +88,16 @@ jobs:
  run: python3 -m unittest discover -v -s python
  - name: Run Python examples
  run: scripts/run_python_examples.sh
+ - name: Test executables
+ run: scripts/test_executables.sh
  - uses: actions/upload-artifact@v3
  with:
  name: dist
  path: dist/*.tar.gz
 
  upload_test_pypi:
  name: upload to test pypi
- runs-on: ubuntu-20.04
+ runs-on: ubuntu-latest
  needs: [build_binary_wheels, build_sdist]
  steps:
  - uses: actions/download-artifact@v3
@@ -110,7 +114,7 @@ jobs:
  make_release:
  name: draft new GitHub release
  if: startsWith(github.ref, 'refs/tags/v')
- runs-on: ubuntu-20.04
+ runs-on: ubuntu-latest
  needs: [build_binary_wheels, build_sdist]
  steps:
  - name: "Clone w/ submodules"

.github/workflows/spack.yml

@@ -25,8 +25,13 @@ jobs:
  uses: actions/cache@v3
  with:
  path: ~/.spack-cache
- key: arbor-cache-${{ matrix.os }}-${{ github.run_id }}
- restore-keys: arbor-cache-${{ matrix.os }}-
+ key: spack-cache-${{ matrix.os }}-${{ github.run_id }}
+ restore-keys: spack-cache-${{ matrix.os }}-
 
  - name: Build Arbor's Spack package against the develop branch
  run: arbor/scripts/build_spack_package.sh arbor develop
+
+ - name: Run Python examples
+ run: scripts/run_python_examples.sh
+ - name: Test executables
+ run: scripts/test_executables.sh

.github/workflows/test-everything.yml

@@ -48,7 +48,7 @@ jobs:
  os: "ubuntu-22.04",
  cc: "gcc-12",
  cxx: "g++-12",
- py: "3.10",
+ py: "3.11",
  cmake: "3.22.x",
  mpi: "ON",
  simd: "OFF"
@@ -58,7 +58,7 @@ jobs:
  os: "ubuntu-22.04",
  cc: "gcc-12",
  cxx: "g++-12",
- py: "3.10",
+ py: "3.11",
  cmake: "3.22.x",
  mpi: "OFF",
  simd: "ON"
@@ -68,17 +68,17 @@ jobs:
  os: "ubuntu-22.04",
  cc: "clang-14",
  cxx: "clang++-14",
- py: "3.10",
+ py: "3.11",
  cmake: "3.22.x",
  mpi: "ON",
  simd: "OFF"
  }
  - {
  name: "MacOS Max",
- os: "macos-12",
+ os: "macos-13",
  cc: "clang",
  cxx: "clang++",
- py: "3.10",
+ py: "3.11",
  cmake: "3.22.x",
  mpi: "ON",
  simd: "OFF"
@@ -189,10 +189,8 @@ jobs:
  run: mpirun -n 4 -oversubscribe python -m unittest discover -v -s python
  # - name: Run Python examples (plotting leads to time outs on macos, the step under testpip is enough)
  # run: scripts/run_python_examples.sh
- - name: Build and test a catalogue
- run: |
- arbor-build-catalogue -v default mechanisms/default
- ./scripts/test-catalogue.py ./default-catalogue.so
+ - name: Test executables
+ run: ./scripts/test_executables.sh
  testpip:
  name: "Pip build test + Python examples test"
  runs-on: ${{ matrix.os }}
@@ -224,6 +222,8 @@ jobs:
  run: python -m unittest discover -v -s python
  - name: Run Python examples
  run: scripts/run_python_examples.sh
+ - name: Test executables
+ run: scripts/test_executables.sh
  testdocs:
  name: "Docs build test"
  runs-on: ${{ matrix.os }}

README.md

@@ -1,5 +1,5 @@
 [![ci](https://github.com/arbor-sim/arbor/actions/workflows/test-everything.yml/badge.svg)](https://github.com/arbor-sim/arbor/actions/workflows/test-everything.yml)
-[![pythonwheels](https://github.com/arbor-sim/arbor/actions/workflows/ciwheel.yml/badge.svg)](https://github.com/arbor-sim/arbor/actions/workflows/ciwheel.yml)
+[![pythonwheels](https://github.com/arbor-sim/arbor/actions/workflows/release.yml/badge.svg)](https://github.com/arbor-sim/arbor/actions/workflows/release.yml)
 [![gitpod](https://img.shields.io/badge/Gitpod-Ready--to--Code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/arbor-sim/arbor)
 [![docs](https://readthedocs.org/projects/arbor/badge/?version=latest)](https://docs.arbor-sim.org/en/latest/)
 [![gitter](https://badges.gitter.im/arbor-sim/community.svg)](https://gitter.im/arbor-sim/community)

cmake/CompilerOptions.cmake

@@ -111,12 +111,17 @@ function(set_arch_target optvar arch)
  string(REGEX REPLACE "-.*" "" target_model "${target}")
 
  # Use -mcpu for all supported targets _except_ for x86 and Apple arm64, where it should be -march.
- if("${target}" MATCHES "aarch64-apple-darwin" OR "${target}" MATCHES "arm64-apple-darwin")
- set(arch_opt "-march=${arch} -mtune=${arch}")
- elseif(target_model MATCHES "x86|i[3456]86" OR target_model MATCHES "amd64" OR target_model MATCHES "aarch64")
- set(arch_opt "-march=${arch} -mtune=${arch}")
+
+ if (CMAKE_CXX_COMPILER_ID MATCHES "AppleClang" AND CMAKE_CXX_COMPILER_VERSION LESS 15)
+ set(arch_opt "")
  else()
- set(arch_opt "-mcpu=${arch}")
+ if("${target}" MATCHES "aarch64-apple-darwin" OR "${target}" MATCHES "arm64-apple-darwin")
+ set(arch_opt "-march=${arch} -mtune=${arch}")
+ elseif(target_model MATCHES "x86|i[3456]86" OR target_model MATCHES "amd64" OR target_model MATCHES "aarch64")
+ set(arch_opt "-march=${arch} -mtune=${arch}")
+ else()
+ set(arch_opt "-mcpu=${arch}")
+ endif()
  endif()
  endif()
 

doc/contrib/release.rst

@@ -48,6 +48,7 @@ Update tags/versions and test
 #. Create new temp-branch ending in ``-rc``. E.g. ``v0.6-rc``
 #. Bump the ``VERSION`` file:
 
+ - For as long as `scikit-build-core` does not support loading fields from external files, also bump in `pyproject.toml`
  - See also :ref:`dev-version`
  - Append ``-rc``. (Make sure there's no ``-dev``)
 

doc/install/build_install.rst

@@ -440,16 +440,16 @@ CMake ``ARB_WITH_PYTHON`` option:
 By default ``ARB_WITH_PYTHON=OFF``. When this option is turned on, a Python module called :py:mod:`arbor` is built.
 
 A specific version of Python can be set when configuring with CMake using the
-``PYTHON_EXECUTABLE`` variable. For example, to use Python 3.8 installed on a Linux
-system with the executable in ``/usr/bin/python3.8``:
+``PYTHON_EXECUTABLE`` variable. For example, to use Python 3.11 installed on a Linux
+system with the executable in ``/usr/bin/python3.11``:
 
 .. code-block:: bash
 
- cmake .. -DARB_WITH_PYTHON=ON -DPYTHON_EXECUTABLE=/usr/bin/python3.8
+ cmake .. -DARB_WITH_PYTHON=ON -DPYTHON_EXECUTABLE=/usr/bin/python3.11
 
 By default the Python package will be installed in the appropriate sub-directory
 inside ``CMAKE_INSTALL_PREFIX``, determined by querying Python's sysconfig library.
-For example ``${CMAKE_INSTALL_PREFIX}/lib/python3.9/site-packages/``.
+For example ``${CMAKE_INSTALL_PREFIX}/lib/python3.10/site-packages/``.
 
 To install the module in a different location, independent of ``CMAKE_INSTALL_PREFIX``,
 use ``ARB_PYTHON_LIB_PATH`` to specify the location where the Python module is to be installed.
@@ -463,10 +463,10 @@ use ``ARB_PYTHON_LIB_PATH`` to specify the location where the Python module is t
  Therefore, correct installation of the Python package to any other location using ``CMAKE_INSTALL_PREFIX``,
  such as user directory (e.g. `~/.local`), a Python or Conda virtual environment, may result in installation to a wrong path.
 
- ``python3 -m site --user-site`` (for user installations) or a path from ``python3 -c 'import site; print(site.getsitepackages())'``
+ ``python -m site --user-site`` (for user installations) or a path from ``python -c 'import site; print(site.getsitepackages())'``
  (for virtual environment installation) can be used in combination with ``ARB_PYTHON_LIB_PATH``.
 
- In addition, installation via ``pip`` or ``python setup.py`` is guaranteed to find the right path. Please refer to the
+ In addition, installation via ``pip`` is guaranteed to find the right path. Please refer to the
  :ref:`Python installation instruction <in_python_custom>`.
 
 
@@ -475,20 +475,18 @@ use ``ARB_PYTHON_LIB_PATH`` to specify the location where the Python module is t
  # A demonstration using ARB_PYTHON_LIB_PATH
 
  # Set up your venv.
- mkdir myenv
- cd myenv/
- python3 -m venv env
+ python -m venv env
  source env/bin/activate
 
  # Install dependencies
- pip3 install numpy
+ pip install numpy
 
  # Obtain arbor
  git clone --recursive [email protected]:arbor-sim/arbor.git
 
  # Manually set the prefix under which the python package will be installed.
  # In this case, the first directory found by querying Python's list of site-package directories.
- pyprefix=`python3 -c 'import site; print(site.getsitepackages()[0])'`
+ pyprefix=`python -c 'import site; print(site.getsitepackages()[0])'`
 
  # Setup CMake
  mkdir build
@@ -502,7 +500,7 @@ use ``ARB_PYTHON_LIB_PATH`` to specify the location where the Python module is t
  make install
 
  # Test it out!
- python -c "import arbor; print(arbor.__config__)"
+ python -c "import arbor; arbor.print_config()"
 
 
 The Arbor Python wrapper has optional support for mpi4py, though
@@ -516,15 +514,14 @@ variable before configuring and building Arbor:
 .. code-block:: bash
 
  # search for path tp python's site-package mpi4py
- for p in `python3 -c 'import sys; print("\n".join(sys.path))'`; do echo ===== $p; ls $p | grep mpi4py; done
-
- ===== /path/to/python3/site-packages
+ $ for p in `python -c 'import sys; print("\n".join(sys.path))'`; do echo ===== $p; ls $p | grep mpi4py; done
+ ===== /path/to/python/site-packages
  mpi4py
 
  # set CPATH and run cmake
- export CPATH="/path/to/python3/site-packages/mpi4py/include/:$CPATH"
+ $ export CPATH="/path/to/python/site-packages/mpi4py/include/:$CPATH"
 
- cmake -DARB_WITH_PYTHON=ON -DARB_WITH_MPI=ON
+ $ cmake -DARB_WITH_PYTHON=ON -DARB_WITH_MPI=ON
 
 .. _install-neuroml:
 

doc/install/gui.rst

@@ -30,7 +30,7 @@ at the same locations at our `GitHub Releases page <https://github.com/arbor-sim
 These versions are self-contained and ready for launch.
 
 To get a feel for the workflow with Arbor-GUI, you can
-take a look at [the tutorial](https://docs.arbor-sim.org/en/latest/tutorial/single_cell_gui.html).
+take a look at :ref:`the tutorial <tutorialgui>`.
 
 The source code is available `at GitHub <https://github.com/arbor-sim/gui>`_.
 We welcome bug reports and feature requests, please use the issue

doc/install/python.rst

@@ -22,16 +22,19 @@ The easiest way to get Arbor is with
 
 .. code-block:: bash
 
- pip3 install arbor
+ # Recommended but optional: install Arbor in a virtual environment
+ python -m venv arbor_env
+ source arbor_env/bin/activate
+ # Download and install Arbor
+ pip install arbor
 
 To test that Arbor is available, try the following in a Python interpreter
 to see information about the version and enabled features:
 
 .. code-block:: python
 
  >>> import arbor
- >>> print(arbor.__version__)
- >>> print(arbor.__config__)
+ >>> arbor.print_config()
 
 You are now ready to use Arbor! You can continue reading these documentation pages, have a look at the
 :ref:`Python API reference<pyoverview>`, or visit the :ref:`tutorial`.
@@ -57,7 +60,7 @@ the master branch in our git repository, you can run:
 
 .. code-block:: bash
 
- pip3 install git+https://github.com/arbor-sim/arbor.git
+ pip install git+https://github.com/arbor-sim/arbor.git
 
 If you want to work on Arbor's code, you can get a copy of our repo and point `pip` at the local directory:
 
@@ -66,7 +69,7 @@ If you want to work on Arbor's code, you can get a copy of our repo and point `p
  # get your copy of the Arbor source
  git clone https://github.com/arbor-sim/arbor.git --recursive
  # make your changes and then instruct pip to build and install the local source
- pip3 install ./arbor/
+ pip install ./arbor/
 
 Every time you make changes to the code, you'll have to repeat the second step.
 
@@ -81,12 +84,6 @@ To enable more, they must be placed in the ``CMAKE_ARGS`` environment variable.
 The simplest way to do this is by prepending the ``pip`` command with ``CMAKE_ARGS=""``,
 where you place the arguments separated by space inside the quotes.
 
-.. Note::
-
- If you run into build issues while experimenting with build options, be sure
- to remove the ``_skbuild`` directory. If you had Arbor installed already,
- you may need to remove it first before you can (re)compile it with the flags you need.
-
 The following flags can be used to configure the installation:
 
 * ``ARB_WITH_MPI=<ON|OFF>``: Enable MPI support, requires MPI library. Default
@@ -106,18 +103,10 @@ The following flags can be used to configure the installation:
 
 .. note::
 
- There are more, advanced flags that can be set. We are using ``scikit-build``
+ There are more, advanced flags that can be set. We are using ``scikit-build-core``
  and ``CMake`` under the hood, so all flags and options valid in ``CMake`` can
  be used in this fashion.
 
- Allthough the
- `scikit-build documentation <https://scikit-build.readthedocs.io/en/latest/usage.html#environment-variable-configuration>`_
- mentions that you can also pass the build options with ``--install-option=""``,
- this will cause ``pip`` to build all dependencies, including all build-dependencies,
- instead of downloading them from PyPI.
- ``CMAKE_ARGS=""`` saves you the build time, and also downloading and setting up the dependencies they in turn require to be present.
- Setting ``CMAKE_ARGS=""`` is in addition compatible with build front-ends like `build <https://pypa-build.readthedocs.io>`_.
-
  Detailed instructions on how to install using CMake are in the :ref:`Python
  configuration <install-python>` section of the :ref:`installation guide
  <in_build_install>`. CMake is recommended if you need more control over
@@ -130,40 +119,40 @@ In the examples below we assume you are installing from a local copy.
 
 .. code-block:: bash
 
- pip3 install ./arbor
+ pip install ./arbor
 
 **With MPI support**. This might require loading an MPI module or setting the ``CC`` and ``CXX``
 :ref:`environment variables <install-mpi>`:
 
 .. code-block:: bash
 
- CMAKE_ARGS="-DARB_WITH_MPI=ON" pip3 install ./arbor
+ CMAKE_ARGS="-DARB_WITH_MPI=ON" pip install ./arbor
 
 **Compile with** :ref:`vectorization <install-vectorize>` on a system with a SkyLake
 :ref:`architecture <install-architecture>`:
 
 .. code-block:: bash
 
- CMAKE_ARGS="-DARB_VECTORIZE=ON -DARB_ARCH=skylake" pip3 install ./arbor
+ CMAKE_ARGS="-DARB_VECTORIZE=ON -DARB_ARCH=skylake" pip install ./arbor
  
 **Enable NVIDIA GPUs (compiled with nvcc)**. This requires the :ref:`CUDA toolkit <install-gpu>`:
 
 .. code-block:: bash
 
- CMAKE_ARGS="-DARB_GPU=cuda" pip3 install ./arbor
+ CMAKE_ARGS="-DARB_GPU=cuda" pip install ./arbor
 
 **Enable NVIDIA GPUs (compiled with clang)**. This also requires the :ref:`CUDA toolkit <install-gpu>`:
 
 .. code-block:: bash
 
- CMAKE_ARGS="-DARB_GPU=cuda-clang" pip3 install ./arbor
+ CMAKE_ARGS="-DARB_GPU=cuda-clang" pip install ./arbor
 
 **Enable AMD GPUs (compiled with hipcc)**. This requires setting the ``CC`` and ``CXX``
 :ref:`environment variables <install-gpu>`:
 
 .. code-block:: bash
 
- CC=clang CXX=hipcc CMAKE_ARGS="-DARB_GPU=hip" pip3 install ./arbor
+ CC=clang CXX=hipcc CMAKE_ARGS="-DARB_GPU=hip" pip install ./arbor
 
 Note on performance
 -------------------