Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Embedded boundary edge lengths and Field arrays have different number of guard cells #5338

Open
clarkse opened this issue Sep 26, 2024 · 1 comment
Assignees
Labels
bug Something isn't working geometry: RZ axisymmetric 2D and quasi-3D

Comments

@clarkse
Copy link
Member

clarkse commented Sep 26, 2024

Allocation of the embedded boundary arrays uses ng_FieldSolver from the guard cell manager in WarpX, whereas the fields use the ng_alloc_EB value in teh guard cell manager. These values can be different, and when using the edge_lengths array in a grown field tilebox a segmentation fault occurs.

E.g. this code snippet yields a seg fault.

    amrex::MultiFab* mfx = field[0];
    amrex::MultiFab* mfy = field[1];
    amrex::MultiFab* mfz = field[2];

    const amrex::IntVect x_nodal_flag = mfx->ixType().toIntVect();
    const amrex::IntVect y_nodal_flag = mfy->ixType().toIntVect();
    const amrex::IntVect z_nodal_flag = mfz->ixType().toIntVect();

    // avoid implicit lambda capture
    auto x_external = expression[0];
    auto y_external = expression[1];
    auto z_external = expression[2];

    for ( MFIter mfi(*mfx, TilingIfNotGPU()); mfi.isValid(); ++mfi)
    {
        const amrex::Box& tbx = mfi.tilebox( x_nodal_flag, mfx->nGrowVect() );
        const amrex::Box& tby = mfi.tilebox( y_nodal_flag, mfy->nGrowVect() );
        const amrex::Box& tbz = mfi.tilebox( z_nodal_flag, mfz->nGrowVect() );

        auto const& mfxfab = mfx->array(mfi);
        auto const& mfyfab = mfy->array(mfi);
        auto const& mfzfab = mfz->array(mfi);

        amrex::Array4<amrex::Real> lx, ly, lz;
        if (EB::enabled()) {
            lx = warpx.m_fields.get(FieldType::edge_lengths, Direction{0}, lev)->array(mfi);
            ly = warpx.m_fields.get(FieldType::edge_lengths, Direction{1}, lev)->array(mfi);
            lz = warpx.m_fields.get(FieldType::edge_lengths, Direction{2}, lev)->array(mfi);
        }

        amrex::ParallelFor (tbx, tby, tbz,
            [=] AMREX_GPU_DEVICE (int i, int j, int k) {
                // skip if node is covered by an embedded boundary
                if (lx && lx(i, j, k) <= 0) { return; }
            }
        );
    }

This was compiled and run in RZ. I am not sure if it does the same in 3D.

@ax3l ax3l added bug Something isn't working geometry: RZ axisymmetric 2D and quasi-3D labels Sep 30, 2024
@ax3l

This comment was marked as outdated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working geometry: RZ axisymmetric 2D and quasi-3D
Projects
None yet
Development

No branches or pull requests

3 participants