Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use MPI_Bcast instead of multiple p2p messages to update nest from parent #2059

Merged
merged 34 commits into from
Jan 22, 2024
Merged

Use MPI_Bcast instead of multiple p2p messages to update nest from parent #2059

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
5f8771c
point to dkokron's atmos_cubed_sphere
May 5, 2023
28d0ac1
Modifications to replace numerous p2p transfers in fill_nested_grid_c…
May 13, 2023
ef442cf
Modifications to replace numerous p2p transfers in fill_nested_grid_c…
May 13, 2023
394aee0
point to dkokron's atmos_cubed_sphere
May 13, 2023
1567bac
Add reason for including ESMF in fv_control_init. Add code to cleanup…
May 15, 2023
4695fbf
New test log from acorn and wcoss2
May 15, 2023
54fd5d9
Get rid of unneeded mpp_sync_self
May 17, 2023
323a95a
Use mpp_broadcast instead of MPI_Bcast at the request of NOAA-GFDL
May 23, 2023
46462a9
Use mpp_broadcast instead of MPI_Bcast at the request of NOAA-GFDL
May 23, 2023
341fde4
Get rid of mpp pelist scoping calls in fill_nested_grid_cpl(). This …
Jun 9, 2023
4f52857
Add alogin03 to recognised acorn login nodes
Jun 9, 2023
8f462c0
Point to scratch area I have access to
Jun 9, 2023
aaffbdb
Keeping pace with upstream.
Jun 9, 2023
9df4a32
Bcast_comm and sending_proc variables not needed anymore
Jun 9, 2023
ce44dfa
Merge branch 'hafs-BcastFillNestedGridCpl' of github.com:dkokron/ufs-…
Jun 10, 2023
dd86b46
New member of the fv_atmos_type type called BcastMember allows more e…
Jun 23, 2023
ea37353
Cleanup unused variables
Jun 23, 2023
cd44a4a
Port to FMS-2023.02
Aug 14, 2023
66c60b6
Sync
Jan 17, 2024
f3d8709
FV3 feature branch hash update
jkbk2004 Jan 18, 2024
ec0ab6e
Merge remote-tracking branch 'upstream/develop' into hafs-BcastFillNe…
jkbk2004 Jan 18, 2024
3ed4a09
FV3 feature branch hash update
jkbk2004 Jan 19, 2024
27c417b
Merge branch 'ufs-community:develop' into hafs-BcastFillNestedGridCpl
jkbk2004 Jan 19, 2024
696f8f4
FV3 feature branch hash update
jkbk2004 Jan 19, 2024
b48477e
add hera RT log passed
FernandoAndrade-NOAA Jan 21, 2024
bee72b5
add jet RT log passed
FernandoAndrade-NOAA Jan 21, 2024
6ba9f6b
add hercules RT logs: passed
zach1221 Jan 21, 2024
b67e1d2
add derecho RT logs: passed
zach1221 Jan 21, 2024
ec9bc24
add orion RT logs: passed
zach1221 Jan 22, 2024
26a0fb0
Acorn RT Log
BrianCurtis-NOAA Jan 22, 2024
f9ae710
add control_p8_gnu ORT logs: passed
zach1221 Jan 22, 2024
b021702
add regional_control_gnu ORT logs: passed
zach1221 Jan 22, 2024
5f73724
add cpld_control_gnu ORT logs: passed
zach1221 Jan 22, 2024
a5e3366
update fv3 hash and revert change in gitmodules
jkbk2004 Jan 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion FV3
Open in desktop
Submodule FV3 updated 8 files
+15 −9 .github/workflows/GCC.yml
+14 −1 CMakeLists.txt
+1 −1 atmos_cubed_sphere
+17 −0 docs/CMakeLists.txt
+2,665 −0 docs/Doxyfile.in
+5 −0 docs/user_guide.md
+118 −28 io/module_fv3_io_def.F90
+36 −16 module_fv3_config.F90
96 changes: 75 additions & 21 deletions tests/logs/OpnReqTests_control_p8_hera.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Thu Jan 18 20:56:52 UTC 2024
Sun Jan 21 16:22:33 UTC 2024
Start Operation Requirement Test


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_bit_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_289728/control_p8_gnu_bit_base
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_294052/control_p8_gnu_bit_base
Checking test bit_base control_p8_gnu results ....
Moving baseline bit_base control_p8_gnu files ....
Moving sfcf000.nc .........OK
Expand Down Expand Up @@ -51,14 +51,14 @@ Moving baseline bit_base control_p8_gnu files ....
Moving RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Moving RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 281.497456
0: The maximum resident set size (KB) = 1302428
0: The total amount of wall time = 278.810775
0: The maximum resident set size (KB) = 1306868

Test bit_base control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_dbg_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_289728/control_p8_gnu_dbg_base
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_294052/control_p8_gnu_dbg_base
Checking test dbg_base control_p8_gnu results ....
Moving baseline dbg_base control_p8_gnu files ....
Moving sfcf000.nc .........OK
Expand Down Expand Up @@ -106,14 +106,14 @@ Moving baseline dbg_base control_p8_gnu files ....
Moving RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Moving RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 916.105846
0: The maximum resident set size (KB) = 1288088
0: The total amount of wall time = 923.177982
0: The maximum resident set size (KB) = 1288004

Test dbg_base control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_289728/control_p8_gnu_dcp
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_294052/control_p8_gnu_dcp
Checking test dcp control_p8_gnu results ....
Comparing sfcf000.nc .........OK
Comparing sfcf021.nc .........OK
Expand Down Expand Up @@ -160,14 +160,14 @@ Checking test dcp control_p8_gnu results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 249.392791
0: The maximum resident set size (KB) = 1285764
0: The total amount of wall time = 247.584509
0: The maximum resident set size (KB) = 1276892

Test dcp control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_289728/control_p8_gnu_mpi
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_294052/control_p8_gnu_mpi
Checking test mpi control_p8_gnu results ....
Comparing sfcf000.nc .........OK
Comparing sfcf021.nc .........OK
Expand Down Expand Up @@ -214,14 +214,14 @@ Checking test mpi control_p8_gnu results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 251.492784
0: The maximum resident set size (KB) = 1280936
0: The total amount of wall time = 251.539685
0: The maximum resident set size (KB) = 1278948

Test mpi control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_289728/control_p8_gnu_rst
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_294052/control_p8_gnu_rst
Checking test rst control_p8_gnu results ....
Comparing sfcf000.nc .........OK
Comparing sfcf021.nc .........OK
Expand Down Expand Up @@ -268,14 +268,14 @@ Checking test rst control_p8_gnu results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 250.611367
0: The maximum resident set size (KB) = 1281936
0: The total amount of wall time = 248.860720
0: The maximum resident set size (KB) = 1278980

Test rst control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_289728/control_p8_gnu_std_base
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_294052/control_p8_gnu_std_base
Checking test std_base control_p8_gnu results ....
Moving baseline std_base control_p8_gnu files ....
Moving sfcf000.nc .........OK
Expand Down Expand Up @@ -323,11 +323,65 @@ Moving baseline std_base control_p8_gnu files ....
Moving RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Moving RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 254.679817
0: The maximum resident set size (KB) = 1277036
0: The total amount of wall time = 254.890624
0: The maximum resident set size (KB) = 1282588

Test std_base control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_294052/control_p8_gnu_thr
Checking test thr control_p8_gnu results ....
Comparing sfcf000.nc .........OK
Comparing sfcf021.nc .........OK
Comparing sfcf024.nc .........OK
Comparing atmf000.nc .........OK
Comparing atmf021.nc .........OK
Comparing atmf024.nc .........OK
Comparing GFSFLX.GrbF00 .........OK
Comparing GFSFLX.GrbF21 .........OK
Comparing GFSFLX.GrbF24 .........OK
Comparing GFSPRS.GrbF00 .........OK
Comparing GFSPRS.GrbF21 .........OK
Comparing GFSPRS.GrbF24 .........OK
Comparing RESTART/20210323.060000.coupler.res .........OK
Comparing RESTART/20210323.060000.fv_core.res.nc .........OK
Comparing RESTART/20210323.060000.fv_core.res.tile1.nc .........OK
Comparing RESTART/20210323.060000.fv_core.res.tile2.nc .........OK
Comparing RESTART/20210323.060000.fv_core.res.tile3.nc .........OK
Comparing RESTART/20210323.060000.fv_core.res.tile4.nc .........OK
Comparing RESTART/20210323.060000.fv_core.res.tile5.nc .........OK
Comparing RESTART/20210323.060000.fv_core.res.tile6.nc .........OK
Comparing RESTART/20210323.060000.fv_srf_wnd.res.tile1.nc .........OK
Comparing RESTART/20210323.060000.fv_srf_wnd.res.tile2.nc .........OK
Comparing RESTART/20210323.060000.fv_srf_wnd.res.tile3.nc .........OK
Comparing RESTART/20210323.060000.fv_srf_wnd.res.tile4.nc .........OK
Comparing RESTART/20210323.060000.fv_srf_wnd.res.tile5.nc .........OK
Comparing RESTART/20210323.060000.fv_srf_wnd.res.tile6.nc .........OK
Comparing RESTART/20210323.060000.fv_tracer.res.tile1.nc .........OK
Comparing RESTART/20210323.060000.fv_tracer.res.tile2.nc .........OK
Comparing RESTART/20210323.060000.fv_tracer.res.tile3.nc .........OK
Comparing RESTART/20210323.060000.fv_tracer.res.tile4.nc .........OK
Comparing RESTART/20210323.060000.fv_tracer.res.tile5.nc .........OK
Comparing RESTART/20210323.060000.fv_tracer.res.tile6.nc .........OK
Comparing RESTART/20210323.060000.phy_data.tile1.nc .........OK
Comparing RESTART/20210323.060000.phy_data.tile2.nc .........OK
Comparing RESTART/20210323.060000.phy_data.tile3.nc .........OK
Comparing RESTART/20210323.060000.phy_data.tile4.nc .........OK
Comparing RESTART/20210323.060000.phy_data.tile5.nc .........OK
Comparing RESTART/20210323.060000.phy_data.tile6.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile1.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile2.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile3.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile4.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 251.852498
0: The maximum resident set size (KB) = 1279220

Test thr control_p8_gnu PASS

OPERATION REQUIREMENT TEST WAS SUCCESSFUL
Thu Jan 18 22:06:08 UTC 2024
Elapsed time: 01h:09m:16s. Have a nice day!
Sun Jan 21 17:30:30 UTC 2024
Elapsed time: 01h:07m:57s. Have a nice day!
24 changes: 12 additions & 12 deletions tests/logs/OpnReqTests_cpld_control_nowave_noaero_p8_hera.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Thu Jan 18 22:33:35 UTC 2024
Mon Jan 22 14:14:01 UTC 2024
Start Operation Requirement Test


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/cpld_control_c96_noaero_p8_dbg_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_201234/cpld_control_nowave_noaero_p8_gnu_dbg_base
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_239905/cpld_control_nowave_noaero_p8_gnu_dbg_base
Checking test dbg_base cpld_control_nowave_noaero_p8_gnu results ....
Moving baseline dbg_base cpld_control_nowave_noaero_p8_gnu files ....
Moving sfcf021.tile1.nc .........OK
Expand Down Expand Up @@ -66,14 +66,14 @@ Moving baseline dbg_base cpld_control_nowave_noaero_p8_gnu files ....
Moving RESTART/iced.2021-03-23-21600.nc .........OK
Moving RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc .........OK

0: The total amount of wall time = 1248.222948
0: The maximum resident set size (KB) = 1379648
0: The total amount of wall time = 1268.844986
0: The maximum resident set size (KB) = 1409776

Test dbg_base cpld_control_nowave_noaero_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/cpld_control_c96_noaero_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_201234/cpld_control_nowave_noaero_p8_gnu_rst
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_239905/cpld_control_nowave_noaero_p8_gnu_rst
Checking test rst cpld_control_nowave_noaero_p8_gnu results ....
Comparing sfcf021.tile1.nc .........OK
Comparing sfcf021.tile2.nc .........OK
Expand Down Expand Up @@ -135,14 +135,14 @@ Checking test rst cpld_control_nowave_noaero_p8_gnu results ....
Comparing RESTART/iced.2021-03-23-21600.nc .........OK
Comparing RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc .........OK

0: The total amount of wall time = 388.716059
0: The maximum resident set size (KB) = 1381872
0: The total amount of wall time = 379.623302
0: The maximum resident set size (KB) = 1398708

Test rst cpld_control_nowave_noaero_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/cpld_control_c96_noaero_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_201234/cpld_control_nowave_noaero_p8_gnu_std_base
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_239905/cpld_control_nowave_noaero_p8_gnu_std_base
Checking test std_base cpld_control_nowave_noaero_p8_gnu results ....
Moving baseline std_base cpld_control_nowave_noaero_p8_gnu files ....
Moving sfcf021.tile1.nc .........OK
Expand Down Expand Up @@ -205,11 +205,11 @@ Moving baseline std_base cpld_control_nowave_noaero_p8_gnu files ....
Moving RESTART/iced.2021-03-23-21600.nc .........OK
Moving RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc .........OK

0: The total amount of wall time = 389.788849
0: The maximum resident set size (KB) = 1386320
0: The total amount of wall time = 387.075066
0: The maximum resident set size (KB) = 1403652

Test std_base cpld_control_nowave_noaero_p8_gnu PASS

OPERATION REQUIREMENT TEST WAS SUCCESSFUL
Thu Jan 18 23:35:28 UTC 2024
Elapsed time: 01h:01m:54s. Have a nice day!
Mon Jan 22 15:12:57 UTC 2024
Elapsed time: 00h:58m:57s. Have a nice day!
24 changes: 12 additions & 12 deletions tests/logs/OpnReqTests_regional_control_hera.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Thu Jan 18 23:40:06 UTC 2024
Sun Jan 21 23:30:59 UTC 2024
Start Operation Requirement Test


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_153057/regional_control_gnu_dcp
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_219650/regional_control_gnu_dcp
Checking test dcp regional_control_gnu results ....
Comparing dynf000.nc .........OK
Comparing dynf006.nc .........OK
Expand All @@ -14,14 +14,14 @@ Checking test dcp regional_control_gnu results ....
Comparing NATLEV.GrbF00 .........OK
Comparing NATLEV.GrbF06 .........OK

0: The total amount of wall time = 522.233732
0: The maximum resident set size (KB) = 577632
0: The total amount of wall time = 518.322638
0: The maximum resident set size (KB) = 589984

Test dcp regional_control_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_153057/regional_control_gnu_std_base
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_219650/regional_control_gnu_std_base
Checking test std_base regional_control_gnu results ....
Moving baseline std_base regional_control_gnu files ....
Moving dynf000.nc .........OK
Expand All @@ -33,14 +33,14 @@ Moving baseline std_base regional_control_gnu files ....
Moving NATLEV.GrbF00 .........OK
Moving NATLEV.GrbF06 .........OK

0: The total amount of wall time = 535.159778
0: The maximum resident set size (KB) = 574824
0: The total amount of wall time = 517.001768
0: The maximum resident set size (KB) = 587296

Test std_base regional_control_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Zachary.Shrader/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_153057/regional_control_gnu_thr
working dir = /scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_OPNREQ_TEST/opnReqTest_219650/regional_control_gnu_thr
Checking test thr regional_control_gnu results ....
Comparing dynf000.nc .........OK
Comparing dynf006.nc .........OK
Expand All @@ -51,11 +51,11 @@ Checking test thr regional_control_gnu results ....
Comparing NATLEV.GrbF00 .........OK
Comparing NATLEV.GrbF06 .........OK

0: The total amount of wall time = 517.618309
0: The maximum resident set size (KB) = 577192
0: The total amount of wall time = 524.960283
0: The maximum resident set size (KB) = 587216

Test thr regional_control_gnu PASS

OPERATION REQUIREMENT TEST WAS SUCCESSFUL
Fri Jan 19 00:22:28 UTC 2024
Elapsed time: 00h:42m:22s. Have a nice day!
Mon Jan 22 00:08:06 UTC 2024
Elapsed time: 00h:37m:08s. Have a nice day!
Loading