Executing post-processing tools

[rybchuk]

Hello! I am trying to debug checkpoint data from an AMR-Wind simulation, and I am trying to use some of the Post-Processing tools described here. Specifically, I am trying to use WritePlotfileToASCII and fnan.

I compiled a WritePlotfileToASCII executable by editing GNUmakefile and then ran it on a restart file: ./WritePlotfileToASCII3d.gnu.ex infile=/scratch/orybchuk/wakedynamics/test_lidar_placement/unstable_precursor_native/chk12600 However, I got the following error

AMReX (22.04) initialized

Error: you are using a 3D build to open a 1831826288D plotfile

amrex::Abort::SIGABRT
See Backtrace.0 file for details

and Backtrace.0 read:

=== If no file names and line numbers are shown below, one can run
            addr2line -Cpfie my_exefile my_line_address
    to convert `my_line_address` (e.g., 0x4a6b) into file name and line number.
    Or one can use amrex/Tools/Backtrace/parse_bt.py.

=== Please note that the line number reported by addr2line may not be accurate.
    One can use
            readelf -wl my_exefile | grep my_line_address'
    to find out the offset for that line.

 0: ./WritePlotfileToASCII3d.gnu.ex() [0x51a976]
    amrex::BLBackTrace::print_backtrace_info(_IO_FILE*)
../../../Src/Base/AMReX_BLBackTrace.cpp:179:39

 1: ./WritePlotfileToASCII3d.gnu.ex() [0x51c792]
    amrex::BLBackTrace::handler(int)
../../../Src/Base/AMReX_BLBackTrace.cpp:85:15

 2: ./WritePlotfileToASCII3d.gnu.ex() [0x41344e]
    main
/projects/wakedynamics/orybchuk/amrex/amrex/Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp:91:32

 3: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f383e75a555]

 4: ./WritePlotfileToASCII3d.gnu.ex() [0x418132]
    _start at ??:?

---

I also ran into issues with fnan. I built the executible by running make, and then I ran ./fnan.gnu.ex /scratch/orybchuk/wakedynamics/test_lidar_placement/unstable_precursor_native/chk12600, but I got the following error:

amrex::Error::0::operator>>(istream&,Box&): expected '(' !!!
SIGABRT
See Backtrace.0 file for details

and Backtrace.0 read

=== If no file names and line numbers are shown below, one can run
            addr2line -Cpfie my_exefile my_line_address
    to convert `my_line_address` (e.g., 0x4a6b) into file name and line number.
    Or one can use amrex/Tools/Backtrace/parse_bt.py.

=== Please note that the line number reported by addr2line may not be accurate.
    One can use
            readelf -wl my_exefile | grep my_line_address'
    to find out the offset for that line.

 0: ./fnan.gnu.ex() [0x519d86]
    amrex::BLBackTrace::print_backtrace_info(_IO_FILE*)
../../Src/Base/AMReX_BLBackTrace.cpp:179:39

 1: ./fnan.gnu.ex() [0x51bba2]
    amrex::BLBackTrace::handler(int)
../../Src/Base/AMReX_BLBackTrace.cpp:85:15

 2: ./fnan.gnu.ex() [0x484160]
    amrex::IndexType::IndexType(amrex::IntVect const&) inlined at ../../Src/Base/AMReX_Box.H:79:20 in amrex::operator>>(std::istream&, amrex::Box&)
../../Src/Base/AMReX_IndexType.H:36:17
amrex::Box::Box(amrex::IntVect const&, amrex::IntVect const&, amrex::IntVect const&)
../../Src/Base/AMReX_Box.H:79:20
amrex::operator>>(std::istream&, amrex::Box&)
../../Src/Base/AMReX_Box.cpp:78:9

 3: ./fnan.gnu.ex() [0x517169]
    amrex::PlotFileDataImpl::PlotFileDataImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
../../Src/Base/AMReX_PlotFileDataImpl.cpp:51:25

 4: ./fnan.gnu.ex() [0x4158fe]
    amrex::PlotFileData::finestLevel() const inlined at /projects/wakedynamics/orybchuk/amrex/amrex/Tools/Plotfile/fnan.cpp:29:45 in main_main()
../../Src/Base/AMReX_PlotFileUtil.H:161:71
main_main()
/projects/wakedynamics/orybchuk/amrex/amrex/Tools/Plotfile/fnan.cpp:29:45

 5: ./fnan.gnu.ex() [0x412e2f]
    main
/projects/wakedynamics/orybchuk/amrex/amrex/Tools/Plotfile/fnan.cpp:65:20

 6: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7efc5664d555]

 7: ./fnan.gnu.ex() [0x4157f2]
    _start at ??:?

I'm fairly new to C++, so I might be missing something simple. If it helps, the structure of the restart file is

orybchuk@ed2:/scratch/orybchuk/wakedynamics/test_lidar_placement/unstable_precursor_native/chk12600$ ls
> amr_wind_info  Header  Level_0/
orybchuk@ed2:/scratch/orybchuk/wakedynamics/test_lidar_placement/unstable_precursor_native/chk12600$ ls Level_0/
> density_D_00000, density_D_00001, ..., density_D_00035, density_H

Any guidance would be appreciated, thanks!

[WeiqunZhang]

The issue might be your files are checkpoint files, not plot files. How to read a checkpoint file will depend on how it's written in AMR-Wind.

[rybchuk]

Thanks for the input Weiqun! AMR-Wind uses amrex::VisMF::Write to output the checkpoint files, so I think the data might be formatted correctly, but I'm not certain.

void IOManager::write_checkpoint_file(const int start_level)
{
    BL_PROFILE("amr-wind::IOManager::write_checkpoint_file");
    const std::string level_prefix = "Level_";
    const std::string chkname =
        amrex::Concatenate(m_chk_prefix, m_sim.time().time_index());

    amrex::Print() << "Writing checkpoint file " << chkname << " at time "
                   << m_sim.time().new_time() << std::endl;
    const auto& mesh = m_sim.mesh();
    amrex::PreBuildDirectorHierarchy(
        chkname, level_prefix, mesh.finestLevel() + 1 - start_level, true);
    write_header(chkname, start_level);
    write_info_file(chkname);

    for (int lev = start_level; lev < mesh.finestLevel() + 1; ++lev) {
        for (auto* fld : m_chk_fields) {
            auto& field = *fld;
            amrex::VisMF::Write(
                field(lev),
                amrex::MultiFabFileFullPrefix(
                    lev - start_level, chkname, level_prefix, field.name()));
        }
    }
}

Is there any chance you know of any public AMReX plot file data that I could test my issue on? I checked the Tests directory but didn't see any in there.

[rybchuk]

Hi Weiqun, I didn't realize that AMR-Wind plot files and checkpoint files are formatted differently. I was able to to run these scripts successfully on AMR-Wind plot files! I will look into the specifics of the checkpoint files and try to read them accordingly. Thanks for the help.