Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reflecting BC #79

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
add_executable(
athenaPK
main.cpp
bc.cpp
eos/adiabatic_glmmhd.cpp
units.hpp
eos/adiabatic_hydro.cpp
Expand Down
54 changes: 54 additions & 0 deletions src/bc.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
//========================================================================================
// AthenaPK - a performance portable block structured AMR astrophysical MHD code.
// Copyright (c) 2021, Athena-Parthenon Collaboration. All rights reserved.
// Licensed under the 3-clause BSD License, see LICENSE file for details
//========================================================================================
//! \file bc.cpp
// \brief Custom boundary conditions for AthenaPK
//
// Computes reflecting boundary conditions using AthenaPK's cons variable pack.
//========================================================================================

#include "bc.hpp"

// Parthenon headers
#include "main.hpp"
#include "mesh/mesh.hpp"

using parthenon::Real;

void ReflectingInnerX3(std::shared_ptr<parthenon::MeshBlockData<Real>> &mbd,
bool coarse) {
std::shared_ptr<parthenon::MeshBlock> pmb = mbd->GetBlockPointer();
auto cons_pack = mbd->PackVariables(std::vector<std::string>{"cons"}, coarse);

// loop over vars in cons_pack
const auto nvar = cons_pack.GetDim(4);
for (int n = 0; n < nvar; ++n) {
bool is_normal_dir = false;
if (n == IM3) {
is_normal_dir = true;
}
parthenon::IndexRange nv{n, n};
ApplyBC<parthenon::X3DIR, BCSide::Inner, BCType::Reflect>(pmb.get(), cons_pack, nv,
is_normal_dir, coarse);
}
}

void ReflectingOuterX3(std::shared_ptr<parthenon::MeshBlockData<Real>> &mbd,
bool coarse) {
std::shared_ptr<parthenon::MeshBlock> pmb = mbd->GetBlockPointer();
auto cons_pack = mbd->PackVariables(std::vector<std::string>{"cons"}, coarse);

// loop over vars in cons_pack
const auto nvar = cons_pack.GetDim(4);
for (int n = 0; n < nvar; ++n) {
bool is_normal_dir = false;
if (n == IM3) {
is_normal_dir = true;
}
parthenon::IndexRange nv{n, n};
ApplyBC<parthenon::X3DIR, BCSide::Outer, BCType::Reflect>(pmb.get(), cons_pack, nv,
is_normal_dir, coarse);
}
}
123 changes: 123 additions & 0 deletions src/bc.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
#ifndef BC_HPP_
#define BC_HPP_
//========================================================================================
// AthenaPK - a performance portable block structured AMR astrophysical MHD code.
// Copyright (c) 2021, Athena-Parthenon Collaboration. All rights reserved.
// Licensed under the 3-clause BSD License, see LICENSE file for details
//========================================================================================
//! \file bc.hpp
// \brief Custom boundary conditions for AthenaPK
//
// Computes reflecting boundary conditions using AthenaPK's cons variable pack.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is only for reflecting boundary conditions, should this file be renamed? Should we have a bc folder?

Or maybe this file can keep this name as a place holder for other commonly used boundary functions in AthenaPK

//========================================================================================

#include "bvals/bvals.hpp"
#include "mesh/meshblock.hpp"

using parthenon::Real;

void ReflectingInnerX3(std::shared_ptr<parthenon::MeshBlockData<Real>> &mbd, bool coarse);
void ReflectingOuterX3(std::shared_ptr<parthenon::MeshBlockData<Real>> &mbd, bool coarse);
Comment on lines +19 to +20
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are these only defined for X3, don't we need reflecting boundary conditions for all dimensions?

That would be an easy template to create.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where are these used? In a pgen in another branch?


// Function for checking boundary flags: is this a domain or internal bound?
//
inline bool IsDomainBound(parthenon::MeshBlock *pmb, parthenon::BoundaryFace face) {
return !(pmb->boundary_flag[face] == parthenon::BoundaryFlag::block ||
pmb->boundary_flag[face] == parthenon::BoundaryFlag::periodic);
}

// Get zones which are inside the physical domain, i.e. set by computation or MPI halo
// sync, not by problem boundary conditions.
//
inline auto GetPhysicalZones(parthenon::MeshBlock *pmb, parthenon::IndexShape &bounds)
-> std::tuple<parthenon::IndexRange, parthenon::IndexRange, parthenon::IndexRange> {
Comment on lines +32 to +33
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason not to use:

Suggested change
inline auto GetPhysicalZones(parthenon::MeshBlock *pmb, parthenon::IndexShape &bounds)
-> std::tuple<parthenon::IndexRange, parthenon::IndexRange, parthenon::IndexRange> {
inline std::tuple<parthenon::IndexRange, parthenon::IndexRange, parthenon::IndexRange>
GetPhysicalZones(parthenon::MeshBlock *pmb, parthenon::IndexShape &bounds) {

I'm not too used to this syntax, I only see it when the return type needs to be inferred with decltype in a template function

return std::tuple<parthenon::IndexRange, parthenon::IndexRange, parthenon::IndexRange>{
parthenon::IndexRange{IsDomainBound(pmb, parthenon::BoundaryFace::inner_x1)
? bounds.is(parthenon::IndexDomain::interior)
: bounds.is(parthenon::IndexDomain::entire),
IsDomainBound(pmb, parthenon::BoundaryFace::outer_x1)
? bounds.ie(parthenon::IndexDomain::interior)
: bounds.ie(parthenon::IndexDomain::entire)},
parthenon::IndexRange{IsDomainBound(pmb, parthenon::BoundaryFace::inner_x2)
? bounds.js(parthenon::IndexDomain::interior)
: bounds.js(parthenon::IndexDomain::entire),
IsDomainBound(pmb, parthenon::BoundaryFace::outer_x2)
? bounds.je(parthenon::IndexDomain::interior)
: bounds.je(parthenon::IndexDomain::entire)},
parthenon::IndexRange{IsDomainBound(pmb, parthenon::BoundaryFace::inner_x3)
? bounds.ks(parthenon::IndexDomain::interior)
: bounds.ks(parthenon::IndexDomain::entire),
IsDomainBound(pmb, parthenon::BoundaryFace::outer_x3)
? bounds.ke(parthenon::IndexDomain::interior)
: bounds.ke(parthenon::IndexDomain::entire)}};
Comment on lines +34 to +52
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This snippet of code seems like it could belong in Parthenon. I'm not sure about the name, GetZonesInDomain or something else referencing the problem domain might be better.

}

enum class BCSide { Inner, Outer };
enum class BCType { Outflow, Reflect };

template <parthenon::CoordinateDirection DIR, BCSide SIDE, BCType TYPE>
void ApplyBC(parthenon::MeshBlock *pmb, parthenon::VariablePack<Real> &q,
parthenon::IndexRange &nvar, const bool is_normal, const bool coarse) {
Comment on lines +58 to +60
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we instead template on:

Suggested change
template <parthenon::CoordinateDirection DIR, BCSide SIDE, BCType TYPE>
void ApplyBC(parthenon::MeshBlock *pmb, parthenon::VariablePack<Real> &q,
parthenon::IndexRange &nvar, const bool is_normal, const bool coarse) {
template <parthenon::BoundaryFace FACE, BCType TYPE>
void ApplyBC(parthenon::MeshBlock *pmb, parthenon::VariablePack<Real> &q,
parthenon::IndexRange &nvar, const bool is_normal, const bool coarse) {

and remove the BCSide enum? Would this work with your other code?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not quite sure how this function is used/should be used. In this PR it looks like it's only used to apply reflecting boundary conditions. Don't all the other pgen's have outflow boundary conditions applied as the default? Unless I've misunderstood something.

// convenient shorthands
constexpr bool X1 = (DIR == parthenon::X1DIR);
constexpr bool X2 = (DIR == parthenon::X2DIR);
constexpr bool X3 = (DIR == parthenon::X3DIR);
constexpr bool INNER = (SIDE == BCSide::Inner);

constexpr parthenon::BoundaryFace bface =
INNER ? (X1 ? parthenon::BoundaryFace::inner_x1
: (X2 ? parthenon::BoundaryFace::inner_x2
: parthenon::BoundaryFace::inner_x3))
: (X1 ? parthenon::BoundaryFace::outer_x1
: (X2 ? parthenon::BoundaryFace::outer_x2
: parthenon::BoundaryFace::outer_x3));

// check that we are actually on a physical boundary
if (!IsDomainBound(pmb, bface)) {
return;
}

const auto &bounds = coarse ? pmb->c_cellbounds : pmb->cellbounds;

const auto &range = X1 ? bounds.GetBoundsI(parthenon::IndexDomain::interior)
: (X2 ? bounds.GetBoundsJ(parthenon::IndexDomain::interior)
: bounds.GetBoundsK(parthenon::IndexDomain::interior));
const int ref = INNER ? range.s : range.e;

std::string label = (TYPE == BCType::Reflect ? "Reflect" : "Outflow");
label += (INNER ? "Inner" : "Outer");
label += "X" + std::to_string(DIR);

constexpr parthenon::IndexDomain domain =
INNER ? (X1 ? parthenon::IndexDomain::inner_x1
: (X2 ? parthenon::IndexDomain::inner_x2
: parthenon::IndexDomain::inner_x3))
: (X1 ? parthenon::IndexDomain::outer_x1
: (X2 ? parthenon::IndexDomain::outer_x2
: parthenon::IndexDomain::outer_x3));
Comment on lines +91 to +97
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise this could be in Parthenon. An is_inner function of IndexDomain or like


// used for reflections
const int offset = 2 * ref + (INNER ? -1 : 1);

pmb->par_for_bndry(
label, nvar, domain, coarse,
KOKKOS_LAMBDA(const int &l, const int &k, const int &j, const int &i) {
if (!q.IsAllocated(l)) return;
if (TYPE == BCType::Reflect) {
q(l, k, j, i) =
(is_normal ? -1.0 : 1.0) *
q(l, X3 ? offset - k : k, X2 ? offset - j : j, X1 ? offset - i : i);
} else {
q(l, k, j, i) = q(l, X3 ? ref : k, X2 ? ref : j, X1 ? ref : i);
Comment on lines +109 to +111
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might wonder about executing these ternary statements inside the kernel and wonder if the CUDA compiler is smart enough to compile them out.

However, this kernel is certainly memory bandwidth limited so the extra integer computes are irrelevant

}
});
}

template <parthenon::CoordinateDirection DIR, BCSide SIDE, BCType TYPE>
void ApplyBC(parthenon::MeshBlock *pmb, parthenon::VariablePack<Real> &q, bool is_normal,
bool coarse = false) {
auto nvar = parthenon::IndexRange{0, q.GetDim(4) - 1};
ApplyBC<DIR, SIDE, TYPE>(pmb, q, nvar, is_normal, coarse);
}

#endif // BC_HPP_
Loading