coastal_ian_atlantic_datm2sch2ww3 #124
Comments
|
Hi @uturuncoglu @saeed-moghimi-noaa @janahaddad @pvelissariou1 For the high-resolution subset mesh, I have finished the ATM+WW3 at However, for the ATM+SCH+WW3 ( The error message is as follows: Hi @uturuncoglu , do you know if there is a way for ESMF to locate the problematic triangulation and write the log file? Thank you! |
|
@yunfangsun It seems that there is an issue with the mesh. So, I think it is an issue with WW3 mesh. I think there is no way in WW3 side to write the mesh but CMEPs mediator could dump the mesh in VTK format. Please set |
|
Hi @uturuncoglu , I have used And I didn't see the vtk files in the folder If there anything wrong in the ufs.configure |
|
Hi @sbanihash, Thank you for helping me check the log files, ATM+SCH+WW3: |
|
@yunfangsun you could also try to set it in the code. Goto |
|
To test the forcing issue, in the ATM+SCH+WW3 case, the forcing flags are turned off The run can't go through To test the memory issue, for the ATM+SCH+WW3, the WW3 parts are using the same number of cores (5000 cores), ATM+SCH+WW3 is still failed. |
|
The mesh quality of the subset mesh is evaluated: 
There are two points are almost identical, the modification is done |
|
Per Yunfang's request I reran the mesh subseting/merging processes using OCSMesh latest functions. This is the code I used: @yunfangsun you will probably have to interpolate the DEM back to the mesh again. The final mesh was uploaded here Please let me know if there is anything else I can help with. Also, let know if you encounter any problems with this mesh. |
|
I have manually corrected the mesh by using SMS. And now the worst quality mesh calculated from MATLAB is as follows:
This mesh is acceptable. |
|
Based on the above mesh, the new ATM+SCHISM, ATM+WW3, ATM+SCHISM+WW3 configuration is developed for the current version of UFS-Coastal. The ATM+SCHISM is located at /work2/noaa/nosofs/yunfangs/hurricane_ian/atm_sch_new/coastal_ian_subset_atm2sch_intel_1, this configuration is working for the Hurricane Ian case, and tested by @mansurjisan . The ATM+WW3 is located at /work2/noaa/nosofs/yunfangs/hurricane_ian/atm_ww_new/coastal_ian_subset_atm2ww3_intel_1, this configuration is also working, finished the simulation. The ATM+SCHISM+WW3 is located at /work2/noaa/nosofs/yunfangs/hurricane_ian/atm_sch_ww3_new/coastal_ian_subset_atm2sch2ww3_intel_1, however, the same error occurs. |
|
Since now the mesh quality is validated, a few of ATM+SCHISM+WW3 configurations have been tested: Since this mesh has the maximum numbers of elements for all of the existing ufs-coastal applications, The may be possible memory issue, I have used different number of cores for this case: In the folder of /work2/noaa/nosofs/yunfangs/hurricane_ian/atm_sch_ww3_new/coastal_ian_subset_atm2sch2ww3_intel_1_1, 8000 cores are used, the job dropped at the processors of 6163, 6675. /work2/noaa/nosofs/yunfangs/hurricane_ian/atm_sch_ww3_new/coastal_ian_subset_atm2sch2ww3_intel_1_2, 10000 cores are used, the job dropped at the processors of 8866, 977. /work2/noaa/nosofs/yunfangs/hurricane_ian/atm_sch_ww3_new/coastal_ian_subset_atm2sch2ww3_intel_1_3, 12000 cores are used, the job dropped at the processors of 4494, 9876, 9875. |
|
To test the exact location of the broken run. The subset mesh The option And for ww3 |
|
By comparing the up two cases: The subset mesh configuration is stopped in the cmeps Before the following: The subset mesh is broken at Infrastructure/Mesh/src/ESMCI_MeshDual.C for the ghost mesh part |
|
Hi @uturuncoglu , The configuration with mesh problem is located at The configuration without problem is located at Thank you! |
|
@yunfangsun I could able to create VTK files but since they are very high resolution it is hard to fine the problematic region. So, I ask one of my colleague for help. I'll update you when I have some update about it. In the mean time, I plot those vtk files using Preview (a simile section of it can be seen in the following figure) and I wonder if they are exactly same or not. From plot, it seems that meshes are not same. Maybe the difference is coming because SCHSIM is creating mesh using ESMF API and WW3 is creating by reading SCRIP grid definition (netcdf file).
|
|
Hi @uturuncoglu , The SCHISM and WW3 are using exactly the same mesh: ww3 mesh SCHISM hgrid file: |
|
@yunfangsun It seems that is an issue with the triangulation. I need to run the case with a specific version of ESMf that will output the region that causes issue. I think I could use your directory |
|
@yunfangsun They seems they are same but actually not. I think creating mesh with SCRIP file might use same coordinates but could end up different number of elements. So, maybe in WW3 side, ESMf call needs to get extra argument to have exact mesh but not sure. |
|
Hi @uturuncoglu , Yes, the folder is /work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/ufuk/coastal_ian_subset_atm2sch2ww3_intel_1_9_1 |
|
Hi @uturuncoglu , To produce the scrip_ww3_esmf.nc file, I firstly ran the WW3 stand-alone case and get a scrip.nc file, and then use |
|
@yunfangsun That is great. Yes, that is the preferred way. Once I have ESMF with debug feature, I'll try to run your case and we see the issue. After that we could look at WW3 side for its mesh generation. |
|
@yunfangsun BTW, I am getting permission error for following files, Are those used by the case? If not it is not important. |
|
Hi @uturuncoglu , |
|
@yunfangsun Do you remember your command that you used to convert scrip file to esmf mesh file. It seems that ESMf created dual mesh for WW3. The command should be something like |
|
Hi @uturuncoglu , I am using |
|
@yunfangsun Okay but it does not mean both side has same mesh in the coarse case. It is just running. Do you still have scrip.nc file around? |
|
Hi @uturuncoglu , Yes, the scrip.nc is located at /work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/ufuk/coastal_ian_subset_atm2sch2ww3_intel_1_9_1 |
|
Hi @uturuncoglu Yes, And I have copied the |
|
@yunfangsun @janahaddad Just to confirm mesh resolution difference, I run
As next step, I'll try to run SCHSIM with node based decomposition to see it has same issue or not. This case would be easier to debug than Ian case since it uses less processor but it also shows the same issue which is great. I'll keep you updated about it. PS: I am also working on interpolation issue. I need to run the case again to get more debug output. That requires updating debug version of ESMF and run the case agains it. I'll keep updating about it too. |
|
@yunfangsun @janahaddad Okay. I run same configuration but this time SCHSIM uses node based decomposition and no CMEPS mediator. The result is same. See the following plot (left is SCHSIM is using element based decomposition and right one is using node based decomposition).
I think there is something going on mesh generation of SCHSIM and it creates nodes on element centers. Since we implemented element based decomposition approach using node based on as reference to be compatible with CMEPS, I think we also inherit this issue. @josephzhang8 @platipodium I am not sure you check this before or not but it seems that SCHSIM mesh generation has issue and it does not creating exactly same mesh with WW3 and this is same in both element and node based decomposition algorithm. I'll check both and try to solve the issue. In the meantime, please let me know if you have some information or suggestion. |
@uturuncoglu We certainly did not check this before in the detail you did. I may have been too naive implementing the decomposition and it is quite possible that we thought we told it to decompose on nodes but in fact let it decompose on element centers. I would look at the arguments of the MeshCreate call; I believe we gave it node coords (need to check) but may have provided some information such that the actual decomposition is on the element centers (indirectly calculated from node coords). Please go ahead making necessary changes. |
|
Hi Ufuk, Thanks |
|
The native decomp inside SCHISM is elem based, not node based. I think this is the root cause for confusion. Once the decomp is done, node/side/elem inside SCHISM are assigned to each MPI process, so resident elements (non-ghost) are exclusive in each process, but resident nodes/sides can be present in >1 process, as a result of the decomp. I don't know how exactly ESMF distribute the info among processes; I was told we only need resident, not ghost. But how about the case where nodes/sides are resident on >1 proc? The same issue is there with node based decomp; in that case, elem will be resident on >1 proc. |
|
I suggest 3 of us (Ufuk, Carsten and I) have a meeting to discuss this asap. I'm available tomorrow between 10 and noon ET. Thx. |
|
@saeed-moghimi-noaa Yes, I could checkADCIRC configuration since we have no WW3 component in there I am not sure it would be helpful or not. Anyway, let me try. |
|
@platipodium I am not sure this is simply decomposition issue or not. Maybe it is just the way of creating mesh with API. Anyway, I am plaining to get some suggestion internally from ESMF team. I'll update you when I have more information. @josephzhang8 Yes. We could meet about this issue but let's wait until I discuss it with the team tomorrow (we have seeking core team call every Wednesday). |
|
@saeed-moghimi-noaa I am seeing same problem in ADCIRC model. Here is the same plot but in this case last column is ADCIRC grid. It is same with SCHSIM.
I think this is a result of the way of generating grid in the ocean model side. Anyway, I'll try to debug and if we solve with SCHSIM we could also port same way to ADCIRC too. |
|
@yunfangsun @janahaddad BTW, i checked |
|
@yunfangsun If you don't mind, could you share some information about the mesh creation process. How are you creating @mansurjisan @yunfangsun @janahaddad I am checking a single element from I could also do similar check for Ian case but lets focus this low resolution first since it would be easier for me to debug. |
|
Hi @uturuncoglu, The SCHISM mesh files And for the RT case of However, the ww3 esmf mesh generates 6496 nodes and 3070 cells. When I developed the Hurricane Ian case with the three meshes, I also used the same mesh for both schism and WW3, For the low resolution SCHISM mesh at /work2/noaa/nos-surge/yunfangs/hurricane_ian/ufs_atm2sch2ww3_crs:SCHISM is using For the middle resolution HSOFS mesh at /work2/noaa/nos-surge/yunfangs/hurricane_ian/ufs_atm2sch2ww3_hsofsSCHISM is using 3564104 cells and 1813443 nodes, WW3's esmf mesh is as follows: For the high resolution of combined mesh at /work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/ufuk/coastal_ian_subset_atm2sch2ww3_intel_1_1_new_msh09182024SCHISM is using 4885987 cells and 2470026 nodes, ww3's esmf mesh is as follows: |
|
@yunfangsun If I am following correctly, the source mesh is created by external mesh tool and the result mesh is used to create both WW3 and SCHSIM meshes. Right? Is this tool creating hgrid.gr3 file directly? Are you using any other tool to go from mesh generation tool to SCHSIM mesh? If not, I am surprised since I could not find center node in the SCHSIM mesh file. Maybe WW3 is doing something special and adding those center nodes. How can be sure about the mesh file produced by the mesh tool and the hgrid.gr3 are same. If we could plot them and they are same and similar with the VTK files that I am producing then WW3 is doing something special that I don't know. |
Are you still on daylight saving time? 10-12 ET would then be 16-18 CEST, I am traveling but could find some time in that slot as well. |
Does this call for a quick Hgrid2Scrip conversion tool? ... SCHISM output is Ugrid so internally we kind of do this. |
|
@platipodium At this point I am not sure about the source of the problem and I am trying to narrow down. The first thing that we can do is to be sure that the source grid (created by some meshing tool) is converted correctly to both model input format (scrip file for WW3 and hgrid.gr3 for SCHSIM). My initial impression is that we do not have same information in SCHSIM side like we have in WW3. So, I just want to be sure that SCHSIM is reading correct information and has all the nodes that is available to WW3. This can be done by creating mesh plot just after having output of meshing tool and compare with the information found in the hgrid.gr3. Of course I am assuming the meshing tool has additional step to convert internal mesh representation to hgrid.gr3 format. If they are same, then I am plaining to look at SCHSIM cap (especially mesh creation part) to identify the source (I did not see anything obvious in my initial pass). anyway, I could make a call on Friday morning (around 9-9:30 MT) if it works for everyone. I also suggest to include @yunfangsun and @mansurjisan. Since they have experience of creating those configurations. |
|
Friday 9am MT works for me. Thx @uturuncoglu. |
|
As a reference, the plot below shows the elem-based decomposition used in SCHISM:
|
|
Hi @uturuncoglu , I’ve primarily been working on the atm2sch configuration, so I haven’t had a chance to dive into the mesh issue with the atm2sch2ww3 setup yet. I’ll join the Friday 9 am MT meeting and look forward to learning more about it then. -Mansur |
|
From meeting on 10/16/2024
|
|
@saeed-moghimi-noaa @josephzhang8 @danishyo @aliabdolali please note I'm moving the comment from our meeting today (above) to oceanmodeling/WW3#11 , where Dan was originally updating on this topic. |
Just saying that despite the fact that Joseph and I are pretty confident (with your help) about our part of the Mesh Generation there might always the possibility of problem due to lack of testing. |
|
@yunfangsun @josephzhang8 @platipodium @janahaddad @saeed-moghimi-noaa I have some update about the issues that we faced in here;
|
|
@uturuncoglu : thank you so much for working on these issues! |
|
@janahaddad @yunfangsun @aliabdolali @saeed-moghimi-noaa @josephzhang8 JFYI, I created an issue in NOAA-EMC side to track this issue, NOAA-EMC/WW3#1319 |
No description provided.
The text was updated successfully, but these errors were encountered: