Eyoo/multicuts #1

ejyoo921 · 2024-07-02T23:58:17Z

Summary

Additional background

Checklist

The proposed changes:

fix a bug or incorrect behavior in AMReX
add new capabilities to AMReX
changes answers in the test suite to more than roundoff level
are likely to significantly affect the results of downstream AMReX users
include documentation in the code and/or rst files, if appropriate

…d a feature to capture the location of multicuts. EB/AMReX_EB2_3D_C.cpp will be the main file to edit since the function 'build_faces' is here.

…h dx, dy, and dz: This is given so that one can consider refining grid to avoid multicuts

…ns: Please find 'EY' tags for any changes

baperry2 · 2024-07-03T20:14:49Z

Suggestions based on our chat today:

Add parmparse option that turns on/off visualization/printing of multicuts
When visualizing/printing multicuts, move the amrex::Abort outside of the MFIter, such that all multicuts in all boxes are captured (somewhere around here:

amrex/Src/EB/AMReX_EB2_Level.H

Line 481 in 39f46ae

ParallelAllReduce::Sum<int>({nsmallcells,nmulticuts}, ParallelContext::CommunicatorSub());

)
Print out multicut x/y/z or i/j/k location instead of (or in addition to) making a plt file
If continuing to plot multicuts, refactor so all multicuts are plotted in a single plt file for each type of face (where the data is the number of cuts). This would mean creating a multifab before the MFIter loop starts in AMReX_EB2_Level.H and then passing additional Array4s into build_faces, then saving after the MFIter loop ends.
Add visualization/printing pathway for both 2D and 3D code

…lt and you may choose 'true' in your input file if you want to see the multicut locations. -> eb2.plt_multiple_cuts = 'true' 2) If (plt_multiple_cuts), we delay aborting the code until finishing up the loop, including the build_cells part. 3) I just made the int total_multicuts to get the final number of multicuts -- to make sure in case it is not the same number as nmulticuts.

ejyoo921 · 2024-07-12T22:44:29Z

sorry I accidentally clicked closing the pull request... 🤦🏻‍♀️

ejyoo921 · 2024-07-12T23:00:25Z

Somethings done

Add parmparse option that turns on/off visualization/printing of multicuts
When visualizing/printing multicuts, move the amrex::Abort outside of the MFIter, such that all multicuts in all boxes are captured (somewhere around here:

amrex/Src/EB/AMReX_EB2_Level.H

Line 481 in 39f46ae

ParallelAllReduce::Sum<int>({nsmallcells,nmulticuts}, ParallelContext::CommunicatorSub());

Src/EB/AMReX_EB2_Level.H

…multicuts. We can now print out the index of multicut boxes but this current output seems to be no good use. Need to adjust more.

…ticuts.0 that includes physical location of the multicut cells: The coordinates indicate the cell center.

ejyoo921 · 2024-07-19T20:41:53Z

Print out multicut x/y/z or i/j/k location instead of (or in addition to) making a plt file
loc_multicuts.txt

…H. Next step would be to create a multifab to store locations of the multicut in build_faces function and bring it out to AMReX_EB2_Leve.H. Also, try to output one plot file.

2) This is used as an input for build_faces. 3) Simply set value - multicut_arr(i,j,k) = 10.0 - whenever we detect multicut. 4) In AMReX_EB2_Level.H (line 494 - ), we first check if multicut_arr stored the multicut locations (value 10.0). - this will be removed once we get right plot file. 5) The hope is that we get "volume" of multicut location when we read the plt file. 6) We probably do not need anything in this line, except the line 508 (WriteSingleLevelPlotfile). - The question is: Is this enought to see the multicuts?

ejyoo921 · 2024-07-29T16:06:10Z

Looking at "Volume" for multicut.
Adjusted "Scalar Opacity Unit Distance" to highlight the multicut locations.
See the bunny.

Src/EB/AMReX_EB2_Level.H

Src/EB/AMReX_EB2_3D_C.cpp

baperry2 · 2024-08-07T18:00:42Z

Src/EB/AMReX_EB2_Level.H

@@ -476,6 +490,12 @@ GShopLevel<G>::define_fine (G const& gshop, const Geometry& geom,
 nsmallcells += nsm;
 nmulticuts += nmc;
 }
+ // EY: Here -> make plot file
+ if (plt_multiple_cuts)


Move after the ParallelAllReduce for nmulticuts, then add nmulticuts > 0 && !cover_multiple_cuts to the conditional

Src/EB/AMReX_EB2_3D_C.cpp

baperry2 · 2024-08-07T18:02:38Z

Src/EB/AMReX_EB2_3D_C.cpp

+ if (plt_multiple_cuts){
+ multicut_arr(i,j,k) = 10.0;
+ // Not GPU friendly
+ amrex::PrintToFile("loc_multicuts") << "fz: (x,y,z) = (" << problo[0]+(i)*dx[0] << ","<< problo[1]+(j)*dx[1] << "," << problo[2]+(k)*dx[2] << ") \n";


surround the PrintToFile bit with an #ifndef AMREX_USE_GPU so this can still compile on GPU.

let me try to run things.

…or stl on Kestrel.

… xbx, ybx, and zbx: next step will be plotting face-center multifabs since they are multicut faces we are plotting now.

ejyoo921 · 2024-08-15T22:51:36Z

I guess Visit won.

…locations in using PrintToFile in AMReX_EB2_3D_C.cpp file. Also, most importantly, I fixed a stupid if-condition mistake in AMReX_EB2_Level.H: line 502. We should not abort the code when no multicuts present but plt_multiple_cuts = true

We now create separate plot files containing multi-cut locations from xbx, ybx, and zbx: next step will be plotting face-center multifabs since they are multicut faces we are plotting now.

…ile with data multicuts. See AMReX_EB2_3D_C.cpp

ejyoo921 · 2024-08-28T18:37:02Z

All the failings come from the same place.
/home/runner/work/amrex/amrex/Src/EB/AMReX_EB2_3D_C.cpp:474:22: error: embedding a directive within macro arguments has undefined behavior [-Werror,-Wembedded-directive] 474 | #ifndef AMREX_USE_GPU | ^

baperry2 · 2024-08-28T20:09:13Z

This is the error I was looking for before!

A couple options:

simply remove the print statements as all the info should be accessible via the plot files
Dig in and see if you can find a place in Pele, AMReX, etc that do something similar and how they do it. I feel like there must be a way. Perhaps it would work to wrap the print in a function.

A next step is to see how this works for multilevel cases

…rintToFile, it seems like we don't need this condition for GPU

baperry2 · 2024-08-28T21:09:19Z

Src/EB/AMReX_EB2_Level.H

+
+ multicut_fcx.define(m_grids, m_dmap, 1, ng, mf_info); 
+ multicut_fcy.define(m_grids, m_dmap, 1, ng, mf_info); 
+ multicut_fcz.define(m_grids, m_dmap, 1, ng, mf_info); 


These should be defined similar to m_areafrac below, so that they have face data rather than cell-centered data

…g ok?

…bring it back if we find out a better solution to deal with GPU runs.

… also implemented

2) added if-conditions for plotting only when we have multicuts for each direction 3) face-centered multicut

baperry2

You should also add Documentation of the new input parameter here: https://github.com/AMReX-Codes/amrex/blob/87de52ef071811629120df6247c53e0e6840c4ec/Docs/sphinx_documentation/source/RuntimeParameters.rst?plain=1#L780

Src/EB/AMReX_EB2_3D_C.cpp

Src/EB/AMReX_EB2_2D_C.cpp

Src/EB/AMReX_EB2_3D_C.cpp

Src/EB/AMReX_EB2_Level.H

Src/EB/AMReX_EB2_3D_C.cpp

Merge branch 'eyoo/multicuts' of github.com:ejyoo921/amrex into eyoo/multicuts

baperry2 · 2024-09-09T21:13:43Z

Src/EB/AMReX_EB2_2D_C.cpp

@@ -310,6 +311,9 @@ int build_faces (Box const& bx, Array4<EBCellFlag> const& cell,
 if (fy(i ,j+1,0) == Type::irregular) { ++ncuts; }
 if (ncuts > 2) {
 Gpu::Atomic::Add(dp,1);
+ if (plt_multiple_cuts){


bring this outside of the if (ncuts > 2) scope. That way ncuts = 2 will show regular cut cells, ncuts > 2 multicuts, and ncuts = 0 cells that are either completely covered or completely fluid.

…(or very tiny) is covered, and some cell values > 2 means multicuts

ejyoo921 and others added 4 commits June 28, 2024 15:08

This branch is for visulization of multicut faces/cells. We mainly ad…

df6b6d0

…d a feature to capture the location of multicuts. EB/AMReX_EB2_3D_C.cpp will be the main file to edit since the function 'build_faces' is here.

As a baby step, we now can see how many multicuts detected, along wit…

734f7aa

…h dx, dy, and dz: This is given so that one can consider refining grid to avoid multicuts

All the parts are implemented to write plot files of multicut locatio…

fb5c787

…ns: Please find 'EY' tags for any changes

Merge branch 'AMReX-Codes:development' into eyoo/multicuts

ac65ec6

ejyoo921 closed this Jul 12, 2024

ejyoo921 reopened this Jul 12, 2024

baperry2 reviewed Jul 12, 2024

View reviewed changes

Src/EB/AMReX_EB2_Level.H Outdated Show resolved Hide resolved

Src/EB/AMReX_EB2_Level.H Outdated Show resolved Hide resolved

ejyoo921 added 2 commits July 18, 2024 16:26

Edited the parmparse option to turn on/off visualization/printing of …

b35cba5

…multicuts. We can now print out the index of multicut boxes but this current output seems to be no good use. Need to adjust more.

Erased everything for plotting; We now generate one text file loc_mul…

5c7561e

…ticuts.0 that includes physical location of the multicut cells: The coordinates indicate the cell center.

ejyoo921 added 4 commits July 26, 2024 14:46

We found a 'right' place to produce the plot file in AMReX_EB2_Level.…

92c65fa

…H. Next step would be to create a multifab to store locations of the multicut in build_faces function and bring it out to AMReX_EB2_Leve.H. Also, try to output one plot file.

Deleted non-necessary printings.

823e97f

Deleted total_multicuts value

4007d81

baperry2 requested changes Aug 7, 2024

View reviewed changes

ejyoo921 added 2 commits August 7, 2024 14:54

minor changed after Bruce's comments. Run a tutorial input with a rot…

8e67650

…or stl on Kestrel.

We now create separate plot files containing multi-cut locations from…

1a7edb3

… xbx, ybx, and zbx: next step will be plotting face-center multifabs since they are multicut faces we are plotting now.

ejyoo921 and others added 4 commits August 26, 2024 17:39

<Cherry-pick due to detached head>

2fd9f4d

We now create separate plot files containing multi-cut locations from xbx, ybx, and zbx: next step will be plotting face-center multifabs since they are multicut faces we are plotting now.

Merge branch 'AMReX-Codes:development' into eyoo/multicuts

11e7ddd

Cleanup more no useful comments && using now std::ofstream for text f…

1b1a970

…ile with data multicuts. See AMReX_EB2_3D_C.cpp

Removed #ifndef condition for GPU use since we no longer use amrex::P…

cc0af83

…rintToFile, it seems like we don't need this condition for GPU

baperry2 reviewed Aug 28, 2024

View reviewed changes

ejyoo921 and others added 15 commits August 28, 2024 15:57

Return to PrintToFile since ofstream is not compatible with CUDA

436bb3f

Forgot to add #ifndef back

0536861

added AMREX_USE_GPU back -> check with Kestrel. is this even compilin…

9b63533

…g ok?

I just deleted the print txt file part in AMReX_EB2_3D_C.cpp. We may …

427d166

…bring it back if we find out a better solution to deal with GPU runs.

2D version is added -> how can we chekc?

df35481

Whitespace edit

bb08807

Fixed 2D case error -

3c0b9ce

Fixed whitespace things

7db7054

RE-generate the MF for face-centered one. called multicut_face. 2D is…

3aeff0e

… also implemented

RE-generate the MF for face-centered one. called multicut_face. 2D is…

b8179f1

… also implemented

1) fixed 2D bug

0023869

2) added if-conditions for plotting only when we have multicuts for each direction 3) face-centered multicut

Whitespace sytling

0f19915

changed the location to call mcx,y,z arrays -> 2D is checked as well

fb47a1a

fixed notations in .H files

a0dc9c6

Merge branch 'AMReX-Codes:development' into eyoo/multicuts

048d390

baperry2 reviewed Sep 6, 2024

View reviewed changes

ejyoo921 added 2 commits September 9, 2024 10:51

Edited based on Bruce's comments on Sep 6.

8b4c582

I don't understand this pull -

75b419a

Merge branch 'eyoo/multicuts' of github.com:ejyoo921/amrex into eyoo/multicuts

baperry2 reviewed Sep 9, 2024

View reviewed changes

changed the way we fill the plot file data: Value of 2 is regular, 0 …

766e8a9

…(or very tiny) is covered, and some cell values > 2 means multicuts

baperry2 approved these changes Sep 10, 2024

View reviewed changes

ejyoo921 and others added 3 commits September 10, 2024 12:27

Merge branch 'AMReX-Codes:development' into eyoo/multicuts

0ba88b9

Merge branch 'AMReX-Codes:development' into eyoo/multicuts

2304763

make chage to cell-centered MF

8eb29d8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Eyoo/multicuts #1

Eyoo/multicuts #1

ejyoo921 commented Jul 2, 2024 •

edited

Loading

baperry2 commented Jul 3, 2024

ejyoo921 commented Jul 12, 2024

ejyoo921 commented Jul 12, 2024 •

edited

Loading

ejyoo921 commented Jul 19, 2024

ejyoo921 commented Jul 29, 2024

baperry2 Aug 7, 2024

baperry2 Aug 7, 2024

ejyoo921 Aug 7, 2024

ejyoo921 commented Aug 15, 2024

ejyoo921 commented Aug 28, 2024

baperry2 commented Aug 28, 2024

baperry2 Aug 28, 2024

baperry2 left a comment

baperry2 Sep 9, 2024

Eyoo/multicuts #1

Are you sure you want to change the base?

Eyoo/multicuts #1

Conversation

ejyoo921 commented Jul 2, 2024 • edited Loading

Summary

Additional background

Checklist

baperry2 commented Jul 3, 2024

ejyoo921 commented Jul 12, 2024

ejyoo921 commented Jul 12, 2024 • edited Loading

ejyoo921 commented Jul 19, 2024

ejyoo921 commented Jul 29, 2024

baperry2 Aug 7, 2024

Choose a reason for hiding this comment

baperry2 Aug 7, 2024

Choose a reason for hiding this comment

ejyoo921 Aug 7, 2024

Choose a reason for hiding this comment

ejyoo921 commented Aug 15, 2024

ejyoo921 commented Aug 28, 2024

baperry2 commented Aug 28, 2024

baperry2 Aug 28, 2024

Choose a reason for hiding this comment

baperry2 left a comment

Choose a reason for hiding this comment

baperry2 Sep 9, 2024

Choose a reason for hiding this comment

ejyoo921 commented Jul 2, 2024 •

edited

Loading

ejyoo921 commented Jul 12, 2024 •

edited

Loading