-
Notifications
You must be signed in to change notification settings - Fork 4
Build
Synergia3 is available on:
-
Docker images for
x86_64_v2ISA, CUDA-11 and CUDA-12: See here.
Available in the synergia2-provisioning repository.
export LOCAL_ROOT=/path/to/local/packages
Git 2.13 and later,
git clone -b devel3 --recurse-submodules https://github.com/fnalacceleratormodeling/synergia2.git
Or manually init and update submodules,
git clone -b devel3 https://github.com/fnalacceleratormodeling/synergia2.git
cd synergia2
git submodule update --init --recursive
In one command
git pull --recurse-submodules
or,
git pull
git submodule update
pixi is a package management tool that attempts to present a modern take on conda pacakge management. Here is an article that describes the benefits of using it over conda/mamba.
To obtain pixi, run:
curl -fsSL https://pixi.sh/install.sh | bash
We currently have a pixi.toml definition file in the source tree that already defines the following pixi environments:
-
cpu(alsodefault) cuda
There are also the following predefined tasks (with dependencies defined among them as appropriate):
- build build synergia3
- clean delete build/install directories
- cmake run cmake to generate the build recipes
- install install synergia3
- test run test suite
- test-failed re-run failed tests with increased verbosity
- synergia launch bash shell with synergia environment enabled
For CPU based platforms, the process to configure, build, install and run the test suite synergia3 is:
pixi run test
For CUDA-based GPU platforms, the command is:
pixi run -e cmake test
To run a synergia script:
pixi run synergia
cd build_pixi/examples/fodo
python fodo.py
CC=gcc CXX=g++ \
cmake -DCMAKE_INSTALL_PREFIX=$LOCAL_ROOT \
-DCMAKE_BUILD_TYPE=Release \
-DFFTW3_LIBRARY_DIRS=$LOCAL_ROOT/lib \
-DHDF5_ROOT=$LOCAL_ROOT \
-DKokkos_ENABLE_OPENMP=ON \
/path/to/synergia/
Kokkos options:
cmake -DKokkos_ENABLE_OPENMP=ON/OFF
cmake -DKokkos_ENABLE_CUDA=ON/OFF
...
Enable/disable Python bindings:
cmake -DBUILD_PYTHON_BINDINGS=on|off # default is on
Vectorization flags (on M1 Mac must set -DGSV=DOUBLE):
cmake -DGSV=DOUBLE|SSE|AVX|AVX512
For ARM or CUDA, use DOUBLE, for x86_64 use AVX seems to give best performance
Enable/disable simple timer profiling:
cmake -DSIMPLE_TIMER=on|off # default is off
cmake -DKokkos_ENABLE_OPENMP=on|off # default is off
cmake -DKokkos_ENABLE_CUDA=on|off
Please ensure that both OpenMP and CUDA are not enabled together.
Paddings need to be turned off in the CUDA build due to a Kokkos bug https://github.com/kokkos/kokkos/issues/2995
cmake -DALLOW_PADDING=ON|OFF
Use OFF for ARM or CUDA, ON for x86_64
We allow external installations of cereal and pybind11 (for building the python bindings) if preferred by the user. These can be passed by the flags -DUSE_EXTERNAL_CEREAL/PYBIND11=on and ensuring that CMake can find them. The defaul behavior is to fetch the latest versions of these packages during configuration.
When running on a cluster that uses slurm as the resource allocation manger/job launch tool, we assume that the default launch flags for MPI programs is srun --mpi=pmix_v3. If you are using a different version of pmi2/pmix, please pass -DSRUN_MPI_PMIX=....
Out general philosophy is to use the package manager to install as many requirements as possible.
sudo apt install \
cmake \
cython3 \
g++ \
hdf5-tools \
libfftw3-dev \
libfftw3-mpi-dev \
libgsl-dev \
libhdf5-dev \
libhdf5-openmpi-dev \
libopenmpi-dev \
libpython3-dev \
pkg-config \
python3-dev \
python3-matplotlib \
python3-mpi4py \
python3-pkgconfig \
python3-pyparsing \
python3-pytest \
python3-tables \
python3.8-venv
# We recommend a python virtual environment for managing module versions.
python3 -m venv --system-site-packages synergia-env
source synergia-env/bin/activate
# We do not install h5py using pip because the pip-installed version may not
# not use a version of HDF5 that matches what we have from the package manager.
i# Instead, we build our own from source.
mkdir tmp
cd tmp
wget https://github.com/h5py/h5py/archive/refs/tags/3.7.0.tar.gz # check for the most recent version
tar xf 3.7.0.tar.gz
cd h5py-3.7.0
HDF5_DIR=/usr/local python setup.py install
cd ../..
rm -r tmp/
# We build using the system compilers which are recent enough for our needs
cmake -DCMAKE_BUILD_TYPE=Release -DKokkos_ENABLE_OPENMP=on /path/to/synergia/
Our general philosophy is to use Homebrew to install as many requirements as possible. See https://brew.sh for instructions on the installation and use of Homebrew.
# Note that the Homebrew installation of hdf5 does not (at the time of this writing)
# support MPI parallelism. There is also an hdf5-mpi package, which does support MPI
# parallelism. These packages conflict, so you must choose which you will install.
brew install gsl hdf5 fftw libomp ninja
# or
# brew install gsl hdf5-mpi fftw libomp ninja
# We recommend a python virtual environment for managing module versions.
python3 -m venv synergia-env
source synergia-env/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install cython matplotlib mpi4py numpy pyparsing pytest
# We do not install h5py using pip because the pip-installed version may not
# not use a version of HDF5 that matches what we have from the package manager.
# Instead, we build our own from source.
mkdir tmp
cd tmp
wget https://github.com/h5py/h5py/archive/refs/tags/3.7.0.tar.gz # check for the most recent version
tar xf 3.7.0.tar.gz
cd h5py-3.7.0
# For the version that does *not* use MPI
HDF5_DIR=/usr/local python setup.py install
# For the version that *does* use MPI
# CC="mpicc" HDF5_MPI="ON" HDF5_DIR=/usr/local python setup.py install
cd ../..
rm -r tmp/
# We do not recommend using /usr/local/ as your installation target. While this is the default, this will mix your Synergia installation
# with the tools installed using Homebrew -- but Homebrew will not know how to update Synergia.
cmake -DCMAKE_BUILD_TYPE=Release -DKokkos_ENABLE_OPENMP=on -DCMAKE_INSTALL_PREFIX=/path/to/install/target /path/to/synergia/
Note that updates of Homebrew-installed packages can invalidate an existing build.
Often rerunning the build (with make or ninja) will be sufficient.
Sometimes, however, this results in re-building or even re-configuration failures.
The solution in this case is a complete clean re-installation.
Delete all files (including hidden files) in the build directory, and re-run cmake.
We do not recommend the use of the the GNU compiler suite to build Synergia on macOS.
This can lead to incompatibilities between C++ libraries that are part of the OS or part of Homebrew on one hand, and C++ libraries built with g++ on the other.
# The current version of the Homebrew GCC formula at the time of this writing installs
# g++-11.
CC=gcc-11 CXX=g++-11 cmake -DCMAKE_BUILD_TYPE=Release -DKokkos_ENABLE_OPENMP=on -DBUILD_PYTHON_BINDINGS=on /path/to/synergia/