[Breaking Change][WIP] Make Mesh-Level Boundary Conditions Usable without "User" flag

CHANGELOG.md

@@ -9,6 +9,8 @@
 - [[PR 948]](https://github.com/parthenon-hpc-lab/parthenon/pull/948) Add solver interface and update Poisson geometric multi-grid example
 
 ### Changed (changing behavior/API/variables/...)
+- [[PR 989]](https://github.com/parthenon-hpc-lab/parthenon/pull/989) Update how mesh-level boundary conditions are registered and eliminate the need for the user flag
+- [[PR 974]](https://github.com/parthenon-hpc-lab/parthenon/pull/974) Change GetParentPointer to always return T*
 
 ### Fixed (not changing behavior/API/variables/...)
 - [[PR992]](https://github.com/parthenon-hpc-lab/parthenon/pull/992) Allow custom PR ops with sparse pools
@@ -22,6 +24,7 @@
 ### Removed (removing behavior/API/varaibles/...)
 
 ### Incompatibilities (i.e. breaking changes)
+- [[PR 989]](https://github.com/parthenon-hpc-lab/parthenon/pull/989) Update how mesh-level boundary conditions are registered and eliminate the need for the user flag
 - [[PR 974]](https://github.com/parthenon-hpc-lab/parthenon/pull/974) Change GetParentPointer to always return T*
 
 

README.md

@@ -8,6 +8,8 @@
 
 Parthenon -- a performance portable block-structured adaptive mesh refinement framework
 
+[[Code Paper]](https://doi.org/10.1177/10943420221143775))
+
 # Key features
 
 * High performance by

doc/sphinx/src/boundary_conditions.rst

@@ -40,7 +40,9 @@ for your ``parthenon_manager``. e.g.,
 
 .. code:: c++
 
-   pman.app_input->boundary_conditions[parthenon::BoundaryFace::inner_x1] = MyBoundaryInnerX1;
+   pman.app_input->RegisterBoundaryCondition(
+	  parthenon::BoundaryFace::inner_x1,
+	  "my_bc_name", MyBoundaryInnerX1);
 
 where ``BoundaryFace`` is an enum defined in ``defs.hpp`` as
 
@@ -58,13 +60,13 @@ where ``BoundaryFace`` is an enum defined in ``defs.hpp`` as
      outer_x3 = 5
    };
 
-You can then set this boundary condition via the ``user`` flag in the
-input file:
+You can then set this boundary condition by using the name you
+registered in the input file:
 
 ::
 
    <parthenon/mesh>
-   ix1_bc = user
+   ix1_bc = my_bc_name
 
 Boundary conditions so defined should look roughly like
 
@@ -133,4 +135,4 @@ for a more complete example):
     pkg->UserBoundaryFunctions[BF::inner_x1].push_back(GetMyBC<X1DIR, BCSide::Inner>());
     pkg->UserBoundaryFunctions[BF::inner_x2].push_back(GetMyBC<X2DIR, BCSide::Inner>());
     ...
-  }
+  }

doc/sphinx/src/particles.rst

@@ -154,13 +154,17 @@ Particle boundary conditions are not applied in separate kernel calls;
 instead, inherited classes containing boundary condition functions for
 updating particles or removing them when they are in boundary regions
 are allocated depending on the boundary flags specified in the input
-file. Currently, outflow and periodic boundaries are supported natively.
-User-specified boundary conditions must be set by specifying the “user”
-flag in the input parameter file and then updating the appropriate
-Swarm::bounds array entries to factory functions that allocate
-device-side boundary condition objects. An example is given in the
-``particles`` example when ix1 and ox1 are set to ``user`` in the input
-parameter file.
+file. Swarm boundary conditions are set via the ``swarm_*_bc``
+variable in the ``parthenon/mesh`` input block in the input file,
+where ``*`` is ``ix1``, ``ox2``, etc.
+
+Currently, outflow and periodic boundaries are supported natively.
+You can register a new boundary condition by calling
+``ApplicationInput::RegisterSwarmBoundaryCondition`` and assigning the
+function that allocates the device-side boundary condition
+objects. Then the ``parthenon/mesh/swarm_*_bv`` variable can be
+assigned to the name you registered. An example is given in the
+``particles`` example.
 
 Outputs
 --------

example/particles/main.cpp

@@ -1,5 +1,5 @@
 //========================================================================================
-// (C) (or copyright) 2020-2022. Triad National Security, LLC. All rights reserved.
+// (C) (or copyright) 2020-2023. Triad National Security, LLC. All rights reserved.
 //
 // This program was produced under U.S. Government contract 89233218CNA000001 for Los
 // Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC
@@ -34,26 +34,10 @@ int main(int argc, char *argv[]) {
   // Redefine parthenon defaults
   pman.app_input->ProcessPackages = particles_example::ProcessPackages;
   pman.app_input->ProblemGenerator = particles_example::ProblemGenerator;
-  if (pman.pinput->GetString("parthenon/mesh", "ix1_bc") == "user") {
-    // In this case, we are setting a custom swarm boundary condition while still using
-    // a default parthenon boundary condition for cell variables. In general, one can
-    // provide both custom cell variable and swarm boundary conditions. However, to use
-    // custom boundary conditions for either cell variables or swarms, the parthenon
-    // boundary must be set to "user" and both cell variable and swarm boundaries provided
-    // as here.
-    pman.app_input->boundary_conditions[parthenon::BoundaryFace::inner_x1] =
-        parthenon::BoundaryFunction::OutflowInnerX1;
-    pman.app_input->swarm_boundary_conditions[parthenon::BoundaryFace::inner_x1] =
-        particles_example::SetSwarmIX1UserBC;
-  }
-  if (pman.pinput->GetString("parthenon/mesh", "ox1_bc") == "user") {
-    // Again, we use a default parthenon boundary condition for cell variables but a
-    // custom swarm boundary condition.
-    pman.app_input->boundary_conditions[parthenon::BoundaryFace::outer_x1] =
-        parthenon::BoundaryFunction::OutflowOuterX1;
-    pman.app_input->swarm_boundary_conditions[parthenon::BoundaryFace::outer_x1] =
-        particles_example::SetSwarmOX1UserBC;
-  }
+  pman.app_input->RegisterSwarmBoundaryCondition(parthenon::BoundaryFace::inner_x1,
+                                                 &particles_example::SetSwarmIX1UserBC);
+  pman.app_input->RegisterSwarmBoundaryCondition(parthenon::BoundaryFace::outer_x1,
+                                                 &particles_example::SetSwarmOX1UserBC);
   pman.ParthenonInitPackagesAndMesh();
 
   // This needs to be scoped so that the driver object is destructed before Finalize

src/application_input.hpp

@@ -1,5 +1,5 @@
 //========================================================================================
-// (C) (or copyright) 2020-2022. Triad National Security, LLC. All rights reserved.
+// (C) (or copyright) 2020-2023. Triad National Security, LLC. All rights reserved.
 //
 // This program was produced under U.S. Government contract 89233218CNA000001 for Los
 // Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC
@@ -16,19 +16,77 @@
 #include <functional>
 #include <map>
 #include <memory>
+#include <sstream>
 #include <string>
+#include <unordered_map>
 #include <vector>
 
 #include "bvals/boundary_conditions.hpp"
 #include "defs.hpp"
 #include "interface/state_descriptor.hpp"
+#include "interface/swarm_boundaries.hpp"
+#include "kokkos_abstraction.hpp"
 #include "parameter_input.hpp"
 #include "parthenon_arrays.hpp"
+#include "utils/error_checking.hpp"
 
 namespace parthenon {
 
-struct ApplicationInput {
+class ApplicationInput {
  public:
+  ApplicationInput() {
+    RegisterBoundaryCondition(BoundaryFace::inner_x1, "outflow",
+                              &BoundaryFunction::OutflowInnerX1);
+    RegisterBoundaryCondition(BoundaryFace::outer_x1, "outflow",
+                              &BoundaryFunction::OutflowOuterX1);
+    RegisterBoundaryCondition(BoundaryFace::inner_x2, "outflow",
+                              &BoundaryFunction::OutflowInnerX2);
+    RegisterBoundaryCondition(BoundaryFace::outer_x2, "outflow",
+                              &BoundaryFunction::OutflowOuterX2);
+    RegisterBoundaryCondition(BoundaryFace::inner_x3, "outflow",
+                              &BoundaryFunction::OutflowInnerX3);
+    RegisterBoundaryCondition(BoundaryFace::outer_x3, "outflow",
+                              &BoundaryFunction::OutflowOuterX3);
+    RegisterBoundaryCondition(BoundaryFace::inner_x1, "reflecting",
+                              &BoundaryFunction::OutflowInnerX1);
+    RegisterBoundaryCondition(BoundaryFace::outer_x1, "reflecting",
+                              &BoundaryFunction::ReflectOuterX1);
+    RegisterBoundaryCondition(BoundaryFace::inner_x2, "reflecting",
+                              &BoundaryFunction::ReflectInnerX2);
+    RegisterBoundaryCondition(BoundaryFace::outer_x2, "reflecting",
+                              &BoundaryFunction::ReflectOuterX2);
+    RegisterBoundaryCondition(BoundaryFace::inner_x3, "reflecting",
+                              &BoundaryFunction::ReflectInnerX3);
+    RegisterBoundaryCondition(BoundaryFace::outer_x3, "reflecting",
+                              &BoundaryFunction::ReflectOuterX3);
+
+    // Currently outflow and periodic are the only particle BCs available
+    RegisterSwarmBoundaryCondition(BoundaryFace::inner_x1, "outflow",
+                                   &DeviceAllocate<ParticleBoundIX1Outflow>);
+    RegisterSwarmBoundaryCondition(BoundaryFace::outer_x1, "outflow",
+                                   &DeviceAllocate<ParticleBoundOX1Outflow>);
+    RegisterSwarmBoundaryCondition(BoundaryFace::inner_x2, "outflow",
+                                   &DeviceAllocate<ParticleBoundIX2Outflow>);
+    RegisterSwarmBoundaryCondition(BoundaryFace::outer_x2, "outflow",
+                                   &DeviceAllocate<ParticleBoundOX2Outflow>);
+    RegisterSwarmBoundaryCondition(BoundaryFace::inner_x3, "outflow",
+                                   &DeviceAllocate<ParticleBoundIX3Outflow>);
+    RegisterSwarmBoundaryCondition(BoundaryFace::outer_x3, "outflow",
+                                   &DeviceAllocate<ParticleBoundOX3Outflow>);
+    RegisterSwarmBoundaryCondition(BoundaryFace::inner_x1, "periodic",
+                                   &DeviceAllocate<ParticleBoundIX1Periodic>);
+    RegisterSwarmBoundaryCondition(BoundaryFace::outer_x1, "periodic",
+                                   &DeviceAllocate<ParticleBoundOX1Periodic>);
+    RegisterSwarmBoundaryCondition(BoundaryFace::inner_x2, "periodic",
+                                   &DeviceAllocate<ParticleBoundIX2Periodic>);
+    RegisterSwarmBoundaryCondition(BoundaryFace::outer_x2, "periodic",
+                                   &DeviceAllocate<ParticleBoundOX2Periodic>);
+    RegisterSwarmBoundaryCondition(BoundaryFace::inner_x3, "periodic",
+                                   &DeviceAllocate<ParticleBoundIX3Periodic>);
+    RegisterSwarmBoundaryCondition(BoundaryFace::outer_x3, "periodic",
+                                   &DeviceAllocate<ParticleBoundOX3Periodic>);
+  }
+
   // ParthenonManager functions
   std::function<Packages_t(std::unique_ptr<ParameterInput> &)> ProcessPackages = nullptr;
 
@@ -49,8 +107,6 @@ struct ApplicationInput {
 
   std::function<void(Mesh *, ParameterInput *, SimTime &)> UserWorkAfterLoop = nullptr;
   std::function<void(Mesh *, ParameterInput *, SimTime &)> UserWorkBeforeLoop = nullptr;
-  BValFunc boundary_conditions[BOUNDARY_NFACES] = {nullptr};
-  SBValFunc swarm_boundary_conditions[BOUNDARY_NFACES] = {nullptr};
 
   // MeshBlock functions
   std::function<std::unique_ptr<MeshBlockApplicationData>(MeshBlock *, ParameterInput *)>
@@ -59,8 +115,69 @@ struct ApplicationInput {
   std::function<void(MeshBlock *, ParameterInput *)> ProblemGenerator = nullptr;
   std::function<void(MeshBlock *, ParameterInput *)> MeshBlockUserWorkBeforeOutput =
       nullptr;
+
+  // Boundary conditions
+  void RegisterBoundaryCondition(BoundaryFace face, const std::string &name,
+                                 BValFunc condition) {
+    if (boundary_conditions_[face].count(name) > 0) {
+      PARTHENON_THROW("Boundary condition " + name + " at face " + std::to_string(face) +
+                      "already registered.");
+    }
+    boundary_conditions_[face][name] = condition;
+  }
+  void RegisterSwarmBoundaryCondition(BoundaryFace face, const std::string &name,
+                                      SBValFunc condition) {
+    if (swarm_boundary_conditions_[face].count(name) > 0) {
+      PARTHENON_THROW("Swarm boundary condition " + name + " at face " +
+                      std::to_string(face) + "already registered.");
+    }
+    swarm_boundary_conditions_[face][name] = condition;
+  }
+  template <typename T>
+  void RegisterSwarmBoundaryCondition(BoundaryFace face, const std::string &name) {
+    RegisterSwarmBoundaryCondition(face, name, &DeviceAllocate<T>);
+  }
+  void RegisterBoundaryCondition(BoundaryFace face, BValFunc condition) {
+    RegisterBoundaryCondition(face, "user", condition);
+  }
+  template <typename T>
+  void RegisterSwarmBoundaryCondition(BoundaryFace face) {
+    RegisterSwarmBoundaryCondition(face, "user", &DeviceAllocate<T>);
+  }
+  void RegisterSwarmBoundaryCondition(BoundaryFace face, SBValFunc condition) {
+    RegisterSwarmBoundaryCondition(face, "user", condition);
+  }
+  // Getters
+  BValFunc GetBoundaryCondition(BoundaryFace face, const std::string &name) const {
+    if (boundary_conditions_[face].count(name) == 0) {
+      std::stringstream msg;
+      msg << "Boundary condition " << name << " at face " << face << "not registered!\n"
+          << "Available conditions for this face are:\n";
+      for (const auto &[name, func] : boundary_conditions_[face]) {
+        msg << name << "\n";
+      }
+      PARTHENON_THROW(msg);
+    }
+    return boundary_conditions_[face].at(name);
+  }
+  SBValFunc GetSwarmBoundaryCondition(BoundaryFace face, const std::string &name) const {
+    if (swarm_boundary_conditions_[face].count(name) == 0) {
+      std::stringstream msg;
+      msg << "Swarm boundary condition " << name << " at face " << face
+          << "not registered!\n"
+          << "Available conditions for this face are:\n";
+      for (const auto &[name, func] : swarm_boundary_conditions_[face]) {
+        msg << name << "\n";
+      }
+      PARTHENON_THROW(msg);
+    }
+    return swarm_boundary_conditions_[face].at(name);
+  }
+
+ private:
+  Dictionary<BValFunc> boundary_conditions_[BOUNDARY_NFACES];
+  Dictionary<SBValFunc> swarm_boundary_conditions_[BOUNDARY_NFACES];
 };
 
 } // namespace parthenon
-
 #endif // APPLICATION_INPUT_HPP_