Merge branch 'ohms_law_runge_kutta' into hybrid_eta_deltaJ

roelof-groenewald · Dec 6, 2023 · 06839c9 · 06839c9
2 parents 22d9f6b + 10ef709
commit 06839c9
Show file tree

Hide file tree

Showing 160 changed files with 1,793 additions and 1,021 deletions.
diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml
@@ -115,7 +115,7 @@ jobs:
         which nvcc || echo "nvcc not in PATH!"
 
         git clone https://github.com/AMReX-Codes/amrex.git ../amrex
-        cd ../amrex && git checkout --detach 9b733ec45cd93a80234a7c98248b6eb4816589d5 && cd -
+        cd ../amrex && git checkout --detach 23.12 && cd -
         make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE USE_CCACHE=TRUE -j 2
 
         ccache -s

diff --git a/.github/workflows/macos.yml b/.github/workflows/macos.yml
@@ -18,22 +18,8 @@ jobs:
       #CMAKE_GENERATOR: Ninja
     steps:
     - uses: actions/checkout@v3
-    - name: Brew Cache
-      uses: actions/cache@v3
-      # - once stored under a key, they become immutable (even if local cache path content changes)
-      # - for a refresh the key has to change, e.g., hash of a tracked file in the key
-      with:
-        path: |
-          /usr/local/bin
-          /usr/local/lib
-          /usr/local/share
-          /Users/runner/Library/Caches/Homebrew
-        key: brew-macos-appleclang-${{ hashFiles('.github/workflows/macos.yml') }}
-        restore-keys: |
-          brew-macos-appleclang-
     - name: install dependencies
       run: |
-        brew --cache
         set +e
         brew unlink gcc
         brew update
@@ -69,6 +55,7 @@ jobs:
         export CCACHE_COMPRESS=1
         export CCACHE_COMPRESSLEVEL=10
         export CCACHE_MAXSIZE=100M
+        export CCACHE_DEPEND=1
         ccache -z
 
         source py-venv/bin/activate

diff --git a/.github/workflows/source/wrongFileNameInExamples b/.github/workflows/source/wrongFileNameInExamples
@@ -18,6 +18,7 @@ do
        [[ ${file:0:12} != PICMI_inputs ]] &&
        [[ ${file:0:8 } != analysis     ]] &&
        [[ ${file:  -4} != yaml         ]] &&
+       [[ ${file:0:4 } != plot         ]] &&
        [[ ${file:0:6 } != README       ]]
     then
         files+=($file)

diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -1,7 +1,7 @@
 # Preamble ####################################################################
 #
 cmake_minimum_required(VERSION 3.20.0)
-project(WarpX VERSION 23.11)
+project(WarpX VERSION 23.12)
 
 include(${WarpX_SOURCE_DIR}/cmake/WarpXFunctions.cmake)
 

diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
@@ -117,7 +117,7 @@ A typical format is:
 
    This is a short, 40-character title
 
-   After a newline, you can write arbitray paragraphs. You
+   After a newline, you can write arbitrary paragraphs. You
    usually limit the lines to 70 characters, but if you don't, then
    nothing bad will happen.
 

diff --git a/Docs/source/conf.py b/Docs/source/conf.py
@@ -79,9 +79,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = u'23.11'
+version = u'23.12'
 # The full version, including alpha/beta/rc tags.
-release = u'23.11'
+release = u'23.12'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

diff --git a/Docs/source/dataanalysis/ascent.rst b/Docs/source/dataanalysis/ascent.rst
@@ -274,7 +274,7 @@ Example Actions
 A visualization of the electric field component :math:`E_x` (variable: ``Ex``) with a contour plot and with added particles can be obtained with the following Ascent Action.
 This action can be used both in replay as well as in situ runs.
 
-.. literalinclude:: examples/Physics_applications/laser_acceleration/ascent_actions.yaml
+.. literalinclude:: ../../../Examples/Physics_applications/laser_acceleration/ascent_actions.yaml
    :language: yaml
 
 There are more `Ascent Actions examples available <https://ascent.readthedocs.io/en/latest/Actions/Examples.html#yaml-examples>`_ for you to play.

diff --git a/Docs/source/dataanalysis/examples b/Docs/source/dataanalysis/examples
diff --git a/Docs/source/developers/checksum.rst b/Docs/source/developers/checksum.rst
@@ -9,7 +9,7 @@ The checksum module is located in ``Regression/Checksum/``, and the benchmarks a
 
 For more details on the implementation, the Python files in ``Regression/Checksum/`` should be well documented.
 
-From a user point of view, you should only need to use ``checksumAPI.py``. It contains Python functions that can be imported and used from an analysis Python script. It can also be executed directly as a Python script. Here are recipies for the main tasks related to checksum regression tests in WarpX CI.
+From a user point of view, you should only need to use ``checksumAPI.py``. It contains Python functions that can be imported and used from an analysis Python script. It can also be executed directly as a Python script. Here are recipes for the main tasks related to checksum regression tests in WarpX CI.
 
 Include a checksum regression test in an analysis Python script
 ---------------------------------------------------------------

diff --git a/Docs/source/developers/fields.rst b/Docs/source/developers/fields.rst
@@ -112,9 +112,9 @@ Bilinear filter
 
 The multi-pass bilinear filter (applied on the current density) is implemented in ``Source/Filter/``, and class ``WarpX`` holds an instance of this class in member variable ``WarpX::bilinear_filter``. For performance reasons (to avoid creating too many guard cells), this filter is directly applied in communication routines, see ``WarpX::AddCurrentFromFineLevelandSumBoundary`` above and
 
-.. doxygenfunction:: WarpX::ApplyFilterJ(const amrex::Vector<std::array<std::unique_ptr<amrex::MultiFab>, 3>> &current, const int lev, const int idim)
+.. doxygenfunction:: WarpX::ApplyFilterJ(const amrex::Vector<std::array<std::unique_ptr<amrex::MultiFab>, 3>> &current, int lev, int idim)
 
-.. doxygenfunction:: WarpX::SumBoundaryJ(const amrex::Vector<std::array<std::unique_ptr<amrex::MultiFab>, 3>> &current, const int lev, const int idim, const amrex::Periodicity &period)
+.. doxygenfunction:: WarpX::SumBoundaryJ(const amrex::Vector<std::array<std::unique_ptr<amrex::MultiFab>, 3>> &current, int lev, int idim, const amrex::Periodicity &period)
 
 Godfrey's anti-NCI filter for FDTD simulations
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

diff --git a/Docs/source/developers/particles.rst b/Docs/source/developers/particles.rst
@@ -87,7 +87,7 @@ Main functions
 
 .. doxygenfunction:: PhysicalParticleContainer::PushPX
 
-.. doxygenfunction:: WarpXParticleContainer::DepositCurrent(amrex::Vector<std::array<std::unique_ptr<amrex::MultiFab>, 3>> &J, const amrex::Real dt, const amrex::Real relative_time)
+.. doxygenfunction:: WarpXParticleContainer::DepositCurrent(amrex::Vector<std::array<std::unique_ptr<amrex::MultiFab>, 3>> &J, amrex::Real dt, amrex::Real relative_time)
 
 .. note::
    The current deposition is used both by ``PhysicalParticleContainer`` and ``LaserParticleContainer``, so it is in the parent class ``WarpXParticleContainer``.

diff --git a/Docs/source/developers/profiling.rst b/Docs/source/developers/profiling.rst
@@ -247,7 +247,7 @@ node, or without a workload management system.
 .. note::
 
     To collect full statistics, Nsight-Compute reruns kernels,
-    temporarilly saving device memory in host memory. This makes it
+    temporarily saving device memory in host memory. This makes it
     slower than Nsight-Systems, so the provided script profiles only a single
     step of a single process. This is generally enough to extract relevant
     information.

diff --git a/Docs/source/developers/python.rst b/Docs/source/developers/python.rst
@@ -56,10 +56,10 @@ This is part of the standard - each of the PICMI classes call the method ``handl
 The main purpose is to process application specific keyword arguments (those that start with ``warpx_`` for example).
 These are then passed into the ``init`` methods.
 In the WarpX implementation, in the ``init``, each of the WarpX specific arguments are saved as attributes of the implementation
-class instancles.
+class instances.
 
 It is in the second method, ``initialize_inputs``, where the PICMI input parameters are translated into WarpX input parameters.
-This method is called later during the intialization.
+This method is called later during the initialization.
 The prefix instances described above are all accessible in the implementation classes (via the ``pywarpx`` module).
 For each PICMI input quantity, the appropriate WarpX input parameters are set in the prefix classes.
 As needed, for example in the ``Species`` class, the dynamic prefix instances are created and the attributes set.

diff --git a/Docs/source/developers/repo_organization.rst b/Docs/source/developers/repo_organization.rst
@@ -41,7 +41,7 @@ All WarpX header files need to be specified relative to the ``Source/`` director
 By default, in a ``MyName.cpp`` source file we do not include headers already included in ``MyName.H``. Besides this exception, if a function or a class
 is used in a source file, the header file containing its declaration must be included, unless the inclusion of a facade header is more appropriate. This is
 sometimes the case for AMReX headers. For instance ``AMReX_GpuLaunch.H`` is a façade header for ``AMReX_GpuLaunchFunctsC.H`` and ``AMReX_GpuLaunchFunctsG.H``, which
-contain respectively the CPU and the GPU implemetation of some methods, and which should not be included directly.
+contain respectively the CPU and the GPU implementation of some methods, and which should not be included directly.
 Whenever possible, forward declarations headers are included instead of the actual headers, in order to save compilation time (see dedicated section below). In WarpX forward
 declaration headers have the suffix ``*_fwd.H``, while in AMReX they have the suffix ``*Fwd.H``.
 The include order (see `PR #874 <https://github.com/ECP-WarpX/WarpX/pull/874#issuecomment-607038803>`__ and `PR #1947 <https://github.com/ECP-WarpX/WarpX/pull/1947>`__) and `proper quotation marks <https://gcc.gnu.org/onlinedocs/cpp/Include-Syntax.html>`__ are:

diff --git a/Docs/source/developers/testing.rst b/Docs/source/developers/testing.rst
@@ -25,7 +25,7 @@ For example, if you like to change the compiler to compilation to build on Nvidi
    branch = development
    cmakeSetupOpts = -DAMReX_ASSERTIONS=ON -DAMReX_TESTING=ON -DWarpX_COMPUTE=CUDA
 
-We also support changing compilation options via the usual :ref:`build enviroment variables <building-cmake-envvars>`.
+We also support changing compilation options via the usual :ref:`build environment variables <building-cmake-envvars>`.
 For instance, compiling with ``clang++ -Werror`` would be:
 
 .. code-block:: sh

diff --git a/Docs/source/developers/warning_logger.rst b/Docs/source/developers/warning_logger.rst
@@ -47,15 +47,15 @@ Each entry of warning list respects the following format:
 .. code-block:: sh
 
    * --> [PRIORITY] [TOPIC] [raised COUNTER]
-   *     MULTILINE MESSAGGE
-   *     MULTILINE MESSAGGE
+   *     MULTILINE MESSAGE
+   *     MULTILINE MESSAGE
    *     @ Raised by: WHICH_RANKS
 
 where:
 
 * ``[PRIORITY]`` can be ``[!  ]`` (low priority), ``[!! ]`` (medium priority) or ``[!!!]`` (high priority). It indicates the importance of the warning.
 * ``[TOPIC]`` indicates which part of the code is concerned by the warning (e.g., particles, laser, parallelization...)
-* ``MULTILINE MESSAGGE`` is an arbitrary text message. It can span multiple-lines. Text is wrapped automatically.
+* ``MULTILINE MESSAGE`` is an arbitrary text message. It can span multiple-lines. Text is wrapped automatically.
 * ``COUNTER`` indicates the number of times the warning was raised **across all the MPI ranks**. This means that if we run WarpX with 2048 MPI ranks and each rank raises the same warning once, the displayed message will be ``[raised 2048 times]``. Possible values are ``once``, ``twice``, ``XX times``
 * ``WHICH_RANKS`` can be either ``ALL`` or a sequence of rank IDs. It is the list of the MPI ranks which have raised the warning message.
 

diff --git a/Docs/source/glossary.rst b/Docs/source/glossary.rst
@@ -17,7 +17,7 @@ Abbreviations
 * **AMR:** adaptive mesh-refinement
 * **BC:** boundary condition (of a simulation)
 * **BCK:** `Benkler-Chavannes-Kuster <https://ieeexplore.ieee.org/document/1638381>`__ method, a stabilization technique for small cells in the electromagnetic solver
-* **BTD:** backtransformed diagnosics, a method to collect data for analysis from a *boosted frame* simulation
+* **BTD:** backtransformed diagnostics, a method to collect data for analysis from a *boosted frame* simulation
 * **CEX:** charge-exchange collisions
 * **CFL:** the Courant-Friedrichs-Lewy condition, a numerical parameter for the numerical convergence of PDE solvers
 * **CI:** continuous integration, automated tests that we perform before a proposed code-change is accepted; see PR

diff --git a/Docs/source/index.rst b/Docs/source/index.rst
@@ -75,11 +75,9 @@ Usage
    :hidden:
 
    usage/how_to_run
-   usage/domain_decomposition
-   usage/parameters
    usage/python
+   usage/parameters
    usage/examples
-   usage/pwfa
    usage/workflows
    usage/faq
 
@@ -126,10 +124,10 @@ Development
    :hidden:
 
    developers/contributing
-   developers/workflows
    developers/developers
    developers/doxygen
    developers/gnumake
+   developers/workflows
    developers/faq
 .. good to have in the future:
 ..   developers/repostructure

diff --git a/Docs/source/install/hpc/fugaku.rst b/Docs/source/install/hpc/fugaku.rst
@@ -24,7 +24,7 @@ Use the following commands to download the WarpX source code and switch to the c
    git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx
 
 
-Compiling WarpX on Fugaku is more pratical on a compute node. Use the following commands to acquire a compute node for one hour:
+Compiling WarpX on Fugaku is more practical on a compute node. Use the following commands to acquire a compute node for one hour:
 
 .. code-block:: bash
 

diff --git a/Docs/source/install/hpc/karolina.rst b/Docs/source/install/hpc/karolina.rst
@@ -17,7 +17,7 @@ If you are new to this system, **please see the following resources**:
 * `Filesystems <https://docs.it4i.cz/karolina/storage/>`__:
 
   * ``$HOME``: per-user directory, use only for inputs, source and scripts; backed up (25GB default quota)
-  * ``/scatch/``: `production directory <https://docs.it4i.cz/karolina/storage/#scratch-file-system>`__; very fast for parallel jobs (20TB default)
+  * ``/scratch/``: `production directory <https://docs.it4i.cz/karolina/storage/#scratch-file-system>`__; very fast for parallel jobs (20TB default)
 
 
 .. _building-karolina-preparation:
@@ -143,7 +143,7 @@ Use the following :ref:`cmake commands <building-cmake>` to compile the applicat
 
 Now, you can :ref:`submit Karolina compute jobs <running-cpp-karolina>` for WarpX :ref:`Python (PICMI) scripts <usage-picmi>` (:ref:`example scripts <usage-examples>`).
 Or, you can use the WarpX executables to submit Karolina jobs (:ref:`example inputs <usage-examples>`).
-For executables, you can reference their location in your :ref:`job script <running-cpp-karolina>` or copy them to a location in ``/scatch/``.
+For executables, you can reference their location in your :ref:`job script <running-cpp-karolina>` or copy them to a location in ``/scratch/``.
 
 
 .. _building-karolina-update:

diff --git a/Docs/source/install/hpc/lassen.rst b/Docs/source/install/hpc/lassen.rst
@@ -263,15 +263,15 @@ The batch script below can be used to run a WarpX simulation on 2 nodes on the s
 Replace descriptions between chevrons ``<>`` by relevant values, for instance ``<input file>`` could be ``plasma_mirror_inputs``.
 Note that the only option so far is to run with one MPI rank per GPU.
 
-.. literalinclude:: ../../../../Tools/machines/lassen-llnl/lassen.bsub
+.. literalinclude:: ../../../../Tools/machines/lassen-llnl/lassen_v100.bsub
    :language: bash
-   :caption: You can copy this file from ``Tools/machines/lassen-llnl/lassen.bsub``.
+   :caption: You can copy this file from ``Tools/machines/lassen-llnl/lassen_v100.bsub``.
 
-To run a simulation, copy the lines above to a file ``lassen.bsub`` and run
+To run a simulation, copy the lines above to a file ``lassen_v100.bsub`` and run
 
 .. code-block:: bash
 
-   bsub lassen.bsub
+   bsub lassen_v100.bsub
 
 to submit the job.
 

diff --git a/Docs/source/install/hpc/quartz.rst b/Docs/source/install/hpc/quartz.rst
@@ -37,14 +37,14 @@ Create it now:
 
 .. code-block:: bash
 
-   cp $HOME/src/warpx/Tools/machines/quartz-llnl/quartz/quartz_warpx.profile.example $HOME/quartz_warpx.profile
+   cp $HOME/src/warpx/Tools/machines/quartz-llnl/quartz_warpx.profile.example $HOME/quartz_warpx.profile
 
 .. dropdown:: Script Details
    :color: light
    :icon: info
    :animate: fade-in-slide-down
 
-   .. literalinclude:: ../../../../Tools/machines/quartz-llnl/quartz/quartz_warpx.profile.example
+   .. literalinclude:: ../../../../Tools/machines/quartz-llnl/quartz_warpx.profile.example
       :language: bash
 
 Edit the 2nd line of this script, which sets the ``export proj=""`` variable.

diff --git a/Docs/source/latex_theory/AMR/AMR.tex b/Docs/source/latex_theory/AMR/AMR.tex
@@ -53,7 +53,7 @@ \subsection{Electrostatic}
 %  \caption{Snapshot from a 3D self-consistent simulation of the injector in the High Current Experiment shows the beam emerging from the source at low energy (blue) and being accelerated (green-yellow-orange) and transported in a four quadrupole front end. The automatic layout of the mesh refinement patches from a 2D axisymmetric simulation of the source area shows 2 levels of refinement, concentrating the finer meshes around the emitter (white curve surface) and the beam edge (dark blue).}
 %  \label{fig:ESHCX}
 %\end{figure}
-%Automatic remeshing has been implemented in WarpX following the procedure described in \cite{Vaynim2005}, refining on criteria based on measures of local charge density magnitude and gradients. AMR WarpX simulations were applied to the modeling of the front end injector of the High Current Experiment (HCX) \cite{Prostprstab2005}, and provided the first numerically converged estimates of phase space beam distorsions, which directly affects beam quality \cite{Vaypop04}. Fig.~\ref{fig:ESHCX} shows snapshots from 2D axisymmetric simulation of the souce area illustrating the automatic placement of refined patches, and 3D simulation of the full injector showing the beam generation, acceleration and transport.
+%Automatic remeshing has been implemented in WarpX following the procedure described in \cite{Vaynim2005}, refining on criteria based on measures of local charge density magnitude and gradients. AMR WarpX simulations were applied to the modeling of the front end injector of the High Current Experiment (HCX) \cite{Prostprstab2005}, and provided the first numerically converged estimates of phase space beam distorsions, which directly affects beam quality \cite{Vaypop04}. Fig.~\ref{fig:ESHCX} shows snapshots from 2D axisymmetric simulation of the source area illustrating the automatic placement of refined patches, and 3D simulation of the full injector showing the beam generation, acceleration and transport.
 
 \subsection{Electromagnetic}
 The method that is used for electrostatic mesh refinement is not directly applicable to electromagnetic calculations. As was shown in section 3.4 of \cite{Vayjcp01}, refinement schemes relying solely on interpolation between coarse and fine patches lead to the reflection with amplification of the short wavelength modes that fall below the cutoff of the Nyquist frequency of the coarse grid. Unless these modes are damped heavily or prevented from occurring at their source, they may affect particle motion and their effect can escalate if trapped within a patch, via multiple successive reflections with amplification.