Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SCHISM integration #2

Closed
uturuncoglu opened this issue Mar 30, 2023 · 317 comments
Closed

SCHISM integration #2

uturuncoglu opened this issue Mar 30, 2023 · 317 comments
Assignees
@uturuncoglu
Labels
old_integration

Comments

@uturuncoglu
Copy link
Collaborator

@platipodium @saeed-moghimi-noaa @pvelissariou1
I am creating this issue to communicate the SCHISM related developments. I am planing to do the same for each component. So, we could focus specific component in each issue. Anyway here is my update about the ongoing development regarding to SCHISM:

  • I created initial version of SCHISM interface to build it under UFS. It is in feature/coastal_app branch.
  • At this point, I could able to build CDEPS+CMEPS+SCHISM configuration without any issue. You could compile it following commands,
git clone -b feature/coastal_app --recursive https://github.com/oceanmodeling/ufs-coastal.git
cd ufs-coastal/tests
./compile.sh "cheyenne.intel" "-DAPP=COASTALS -DNO_PARMETIS=ON -DOLDIO=ON" coastal YES NO
  • The build does not include Utilities yet since it seems there are some issue in the SCHISM CMake interface and build fails with some NetCDF error. I'll look at it later.
  • I slightly modified the SCHISM/src/CMakeLists.txt file and add new option like BLD_STANDALONE. This is ON by default and will not change anything interns of existing functionality but when I build SCHISM under UFS I am setting it to OFF since I don't want to build Driver and finding libraries (they are provided by UFS). I tried to use the exiting build system as much as possible to create SCHISM-Interface under UFS but we could change the design in the later stage of the project. You could see the changes in the following link,
    https://github.com/schism-dev/schism/compare/master...uturuncoglu:schism:feature/coastal_app?expand=1
  • I did not test the build yet and I need a test case for it. There are several test cases under CoastalApp-testsuite *shinnecock* but I could not use them without having ESMF Mesh file for the atmospheric forcing and also 2d forcing fields (CDEPS requires those). As a workaround, I could go with the CFS data that are provided by @pvelissariou1 for the florence_hsofs.atm2adc and used by new CDEPS mode (ATMMESH) but I am not sure that will work with the SCHISM configuration or not. So, I need some guidance to create test case to test SCHISM build.
@uturuncoglu uturuncoglu added the enhancement New feature or request label Mar 30, 2023
@uturuncoglu uturuncoglu self-assigned this Mar 30, 2023
@saeed-moghimi-noaa
Copy link

Thanks @uturuncoglu . For schism I also tag @josephzhang8 .

@uturuncoglu
Copy link
Collaborator Author

Maybe I could go with ike_shinnecock.sch. I think this is SCHISM forced by data atmosphere that requires only u and v wind components. I need to find data for 2008-08-23 and maybe could use CDEPS ATMMESH data mode but I need CFS data for it. @pvelissariou1 Can you also send me CFS data for those dates. It seems forecast length is 120 hours for this case.

@uturuncoglu
Copy link
Collaborator Author

ike_shinnecock.sch seems standalone test. So maybe it is good to start with it without worrying data atmosphere etc. Let me start with it and maybe I could define it as RT under UFS.

@uturuncoglu
Copy link
Collaborator Author

@platipodium @saeed-moghimi-noaa @pvelissariou1 @josephzhang8 ike_shinnecock.sch seems working now. Is there any simple script or tool that I could use to create 2d maps etc. to check the results.

@platipodium
Copy link
Member

platipodium commented Mar 31, 2023 •
edited
Loading

Thanks @uturuncoglu ... quite a beast to download :=).

On my M1 the command should be "./compile.sh macosx.gnu-DAPP=COASTALS -DNO_PARMETIS=ON -DOLDIO=ON" coastal YES NO. I got as far as missing PIO...

@platipodium
Copy link
Member

Thanks @uturuncoglu ... quite a beast to download :=).

On my M1 the command should be "./compile.sh macosx.gnu-DAPP=COASTALS -DNO_PARMETIS=ON -DOLDIO=ON" coastal YES NO. I got as far as missing PIO...

New issue #3

The CMAKE_FIND package does not locate my installed PIO, I'll make a new issue

@josephzhang8
Copy link
Collaborator

josephzhang8 commented Mar 31, 2023 via email

@uturuncoglu
Copy link
Collaborator Author

@josephzhang8 The exiting CoastalApp was using OLDIO (-DOLDIO=ON) and I used that tooin my case but it is easy to test it by turning it off. Let me try.

@uturuncoglu
Copy link
Collaborator Author

uturuncoglu commented Mar 31, 2023 •
edited
Loading

@josephzhang8 I tried to compile without OLDIO but I am getting following error when I run the model,

3: 3: ABORT: INIT: Too few scribes (0). Please specify atleast equal to number of output variables (1)

So, maybe the configuration needs to be adjusted to run in this case. Let me know if I need to change any option in the ike_shinnecock.sch case.

@josephzhang8
Copy link
Collaborator

josephzhang8 commented Mar 31, 2023 via email

@josephzhang8
Copy link
Collaborator

josephzhang8 commented Mar 31, 2023 via email

@uturuncoglu
Copy link
Collaborator Author

@josephzhang8 Okay. I'll check the documentation. Thanks for your help.

@josephzhang8
Copy link
Collaborator

josephzhang8 commented Mar 31, 2023 via email

@uturuncoglu
Copy link
Collaborator Author

Hi All,

I added new RT under tests/rt_coastal.conf (in the same branch)

./rt.sh -l rt_coastal.conf -k -n coastal_ike_shinnecock_sch

It is running in my side (still uses old I/O, once I test new one I'll fix the configuration). To run ufs_coastal related RT, you need to copy input directories and make minor change in the rt.sh to point your new input directory. Let me know if you need to access them. They are on Cheyenne.

What is next? I am plaining to continue to work and run one coupled schism configuration. It seems ike_shinnecock.atm2sch uses data atmosphere. So, running first with NUOPC connector and then with CMEPS would be good to see if we have issue with CMPES like we had in ADCIRC component. It would be also nice to sync with UFS Weather Model since lots of development done after we fork it. So, this would be a good test for syncing ufs-coastal.

@pvelissariou1 If you don't mind, could you send me CFS data for 2008-08-23? I am not sure this configuration also covers the same region like florence_hsofs.atm2adc (maybe that test name needs to be florence_cfs.atm2adc since we are facing with CFS) but if you provide a file for me I could create ESMF mesh file for it if it is required.

@saeed-moghimi-noaa
Copy link

@uturuncoglu @pvelissariou1 @josephzhang8 @platipodium

Very good progress indeed. I agree with staying with schism to include more of the tests. For the pull request I have no idea when would be the best time. I feel like it make sense if we start a regular weekly tag-up (perhaps just 30 min) for now only with schism developers. We are still waiting to hear back from adcirc folks. FVCOM also just finished their tests for nems-fvcom and nems-ww3-fvcom. They are also actively working with us right now. As another avenue to take if adcirc folks stayed unresponsive.

@pvelissariou1
Copy link
Collaborator

pvelissariou1 commented Mar 31, 2023 via email

@josephzhang8
Copy link
Collaborator

josephzhang8 commented Mar 31, 2023 via email

@uturuncoglu
Copy link
Collaborator Author

@josephzhang8 let me test in my side. If there is some issue with .gitmodules file that could create issue.

@uturuncoglu
Copy link
Collaborator Author

@saeed-moghimi-noaa Just to clarify. I will not have PR in UFS Weather Model side. I am just planing to bring their updates to our fork (and to branch). That is all. So we could keep up-to-date along with the development in UFS Weather Model side. I think we decided not to push anything to UFS Weather Model authoritative repository yet in our last call.

@saeed-moghimi-noaa
Copy link

@uturuncoglu Thanks for the clarification. Understood.

@uturuncoglu
Copy link
Collaborator Author

@josephzhang8 Following command works for me (I named the directory as TEST).

git clone --recursive -b feature/coastal_app https://github.com/oceanmodeling/ufs-coastal.git TEST

BTW, since ADCIRC is private repository, they need to add you (maybe you are already in that group) and enter password when git asks.

@josephzhang8
Copy link
Collaborator

josephzhang8 commented Mar 31, 2023 via email

@uturuncoglu
Copy link
Collaborator Author

@josephzhang8 Yes that is the plan but we will keep it up-to-date with authoritative repository. I could also make this branch default so you don't need to specify branch in the clone but we could decide it later. I am plaining to keep develop up-to-date with authoritative repository. Anyway, we could discuss these details in the next call.

@josephzhang8
Copy link
Collaborator

josephzhang8 commented Mar 31, 2023 via email

@uturuncoglu
Copy link
Collaborator Author

uturuncoglu commented Mar 31, 2023 •
edited
Loading

@josephzhang8 @platipodium I set <# scribes> as following in my job submission script,

mpiexec_mpt -p %g: -np 4 omplace ./fv3.exe 1

but still getting same error. I think this is because of not building src/Driver. The UFS driver does not have similar code like following,

https://github.com/uturuncoglu/schism/blob/639d57a17f22d33d82d422b494e41c1cd99cd29e/src/Driver/schism_driver.F90#L91

I think we make this as an option in ESMF config file (under OCN_attributes:: group) and pass to the model with this way. I could modify the schism cap but maybe you might want to follow different approach. I think this was not used with the exiting CoastalApp since it was building with OLDIO. Anyway, let me know what you think. We could also open an another ticket for this issue.

@josephzhang8
Copy link
Collaborator

josephzhang8 commented Mar 31, 2023 via email

@uturuncoglu
Copy link
Collaborator Author

@josephzhang8 Sure. That sounds fine.

@platipodium
Copy link
Member

Go ahead with making that an option. We did not so far because in principle none of the models should do I/O, but rather a dedicated I/O server....

@uturuncoglu
Copy link
Collaborator Author

@josephzhang8 There are some information in the following link to port UFS coastal,

https://oceanmodeling.github.io/ufs-coastal-app/versions/main/html/porting.html#docker-container

and more information about porting to another platform is very similar to the ufs-weather-model,

https://spack-stack.readthedocs.io/en/latest/

Please refer to this both document and try to port ufs-coastal. If you could able to take notes and share your experiences, we could add them to the app level documentation and it would be helpful for other users.

@josephzhang8
Copy link
Collaborator

Thanks @uturuncoglu . Since our college system is shared I'll need to ask admin to install some packages.

Meanwhile I was able to compile on hercules but could u tell me the binary location/name? Thx.

@pvelissariou1
Copy link
Collaborator

@josephzhang8 I am compiling on hercules and orion right now. On hercules it fails to compile SCHISM due to the missing python executable (usually a link to python3). Changed that in the Cmake file for schism and it compiled. Anyway the executable is at: build_fv3_coastal/ufs_model and the command: nm -a ufs_model | grep -i schism
will produce all schism related compiled elements.

@pvelissariou1
Copy link
Collaborator

@uturuncoglu spack-python/3.9.14 doe not provide the link python->python3 as it points to system's installed python. Need to fix this.

@josephzhang8
Copy link
Collaborator

Thx @pvelissariou1 . It's strange b/c I did not see any errors when building on hercules but I did a global search and failed to find ufs_model

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 we need to open issue in spack-stack if it is not related with the component. I know 1.5.1 will be available soon and maybe they fixed before it. Btw, the UFS moved to 1.5.0 recently. So, maybe it is fixed already. We need to sync UFS-Coastal to get those changes.

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 I sync the model with upstream. Testing now. I'll update you soon. I'll also test on hercules If I could to see the issue. BTW, which case are you trying to install.

@pvelissariou1
Copy link
Collaborator

@uturuncoglu I just tried to compile using:
./compile.sh "hercules" "-DAPP=CSTLSW -DUSE_ATMOS=ON -DUSE_WW3=ON -DNO_PARMETIS=OFF -DOLDIO=ON -DPDLIB=ON" coastal intel NO NO

The python command is hardcoded into SCHISM's CMake file

@josephzhang8
Copy link
Collaborator

Maybe my python env is set up differently on hercules, and I did not encounter any issue there - or maybe that's why I did not see the binary?

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 I tested ROMS, SCHISM and WW3 tests and fixed them (there was some changes in the RT framework and our tests need to be adjusted). They are running without any issue. I'll also fix ADCIRC and FVCOM tests. The synced model has spack stack 1.5. I'll test the updated model on Hera to see I could reproduce the Python issue. If I could, I will fix and let you know.

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 I could compile SCHISM datm configuarion on Hercules without any issue after syncing ufs-coastal with automative repository (not using spack-stack 1.5). Please try in your end and let me know if you have any issue.

@pvelissariou1
Copy link
Collaborator

@uturuncoglu Thank you so much Ufuk. Upon my return I will check UFS-Coastal on the different platforms and finalize the RTs for SCHISM/PAHM/WW3. I thought we had a good meeting last Friday with Joseph and Carsten. Let's keep the plan we discussed about SCHISM in UFS-Coastal. We also need to start thinking about 3D support for SCHISM in UFS-Coastal (SCHISM+WW3, CICE and sediments/water quality). Again, thank you for all your efforts.

@uturuncoglu
Copy link
Collaborator Author

@pvelissariou1 Sure. No worries. It is not urgent. Best,

@yunfangsun
Copy link
Collaborator

@pvelissariou1

@uturuncoglu It is not -DUSE_WW3=ON, but the -DUSE_WWM=ON (internal wave model), please see #24 Reposting here: Yunfang, in the current version of UFS-Coastal building the wave model (WWM) inside SCHISM is not supported and also SCHISM fails to compile if the USE_WWM option is active (it has beed fixed in the latest version of SCHISM). To compile SCHISM+WWM please follow the steps:

  1. Download the latest version of SCHISM:
    git clone https://github.com/schism-dev/schism.git SCHISM
  2. Download UFS-Coastal (if you haven't done so):
    git clone --recurse-submodules https://github.com/oceanmodeling/ufs-coastal.git -b feature/coastal_app ufs-coastal-coastal_app

Update your ufs-coastal-coastal_app to use the latest version of SCHISM a) Remove the SCHISM source from ufs-coastal-coastal_app rm -rf ufs-coastal-coastal_app/SCHISM-interface/SCHISM b) Update SCHISM cp -rp SCHISM ufs-coastal-coastal_app/SCHISM-interface/ c) Modify the SCHISM-interface/CMakeLists.txt as follows: LINES FROM (lines 97-98): # SCHISM add_library(schism STATIC schismnuopcsrcfiles<TARGET_OBJECTS:core> $&lt;TARGET_OBJECTS:hydro&gt;) LINES TO: # SCHISM if(USE_WWM) add_library(schism STATIC schismnuopcsrcfiles&lt;TARGET_OBJECTS:core&gt;$&lt;TARGET_OBJECTS:hydro> $&lt;TARGET_OBJECTS:wwmIII&gt;) else(USE_WWM) add_library(schism STATIC schismnuopcsrcfiles&lt;TARGET_OBJECTS:core&gt;$&lt;TARGET_OBJECTS:hydro>) endif(USE_WWM)

Now to compile SCHISM standalone with WWM ON or SCHISM within UFS-Coastal do the following (I am working on hera):

  1. Load the necessary modules from spack-stack:
    module use PATH_TO_ufs-coastal-coastal_app/modulefiles
    module load module load ufs_hera.intel
  2. Compile SCHISM stanalone (outside UFS-Coastal)
    cd SCHISM
    mkdir build
    cd build
    cmake ../src -DUSE_WWM=ON -DNO_PARMETIS=OFF -DOLDIO=ON
    This will build SCHISM and its utilities. Executables are located in: build/bin
  3. Compile SCHISM within UFS-Coastal
    cd ufs-coastal-coastal_app/tests, then
    3a) compile without using the atmospheric forcing through ESMF:
    ./compile.sh hera "-DAPP=CSTLS -DUSE_WWM=ON -DNO_PARMETIS=OFF -DOLDIO=ON" "coastalSWWM" intel YES NO
    3b) compile using the atmospheric forcing through ESMF:
    ./compile.sh hera "-DAPP=CSTLS -DUSE_ATMOS=ON -DUSE_WWM=ON -DNO_PARMETIS=OFF -DOLDIO=ON" "coastalSWWM" intel YES NO
    The binaries are located in tests/fv3_coastalSWWM.exe tests/build_fv3_coastalSWWM/SCHISM-interface/SCHISM/src/bin
    You may also set/configure your test case (for ian) by adding the cmake flag -DUSE_WWM=ON and proceed as usual
    with your simulation

Hi Takis,

I have following your steps and compile UFS-coastal for SCHISIM with internal WWM on Hera,

it gives me the following mistakes for

./compile.sh hera "-DAPP=CSTLS -DUSE_WWM=ON -DNO_PARMETIS=OFF -DOLDIO=ON" "coastalSWWM" intel YES NO

.so;/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.5.0/envs/unified-env-noavx512/install/intel/2021.5.0/netcdf-fortran-4.6.0-wplqtff/lib/libnetcdff.so;/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.5.0/envs/unified-env-noavx512/install/intel/2021.5.0/netcdf-c-4.9.2-pc2lepn/lib/libnetcdf.so;/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.5.0/envs/unified-env-noavx512/install/intel/2021.5.0/netcdf-fortran-4.6.0-wplqtff/lib/libnetcdff.so;/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.5.0/envs/unified-env-noavx512/install/intel/2021.5.0/netcdf-c-4.9.2-pc2lepn/lib/libnetcdf.so;/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.5.0/envs/unified-env-noavx512/install/intel/2021.5.0/netcdf-fortran-4.6.0-wplqtff/lib/libnetcdff.so
-- In Utility/Gen_Hotstart
-- In Utility/Grid_Scripts
-- Configuring done
CMake Error at SCHISM-interface/CMakeLists.txt:99 (add_library):
  Cannot find source file:

    /scratch2/STI/coastal/Yunfang.Sun/run_schism/schism_wwm/ufs-coastal-coastal_app/tests/build_fv3_coastalSWWM/SCHISM-interface/SCHISM/src/Core/CMakeFiles/core.dir/schism_version.F90.o/scratch2/STI/coastal/Yunfang.Sun/run_schism/schism_wwm/ufs-coastal-coastal_app/tests/build_fv3_coastalSWWM/SCHISM-interface/SCHISM/src/Hydro/CMakeFiles/hydro.dir/schism_init.F90.o

  Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm .h
  .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .hip .ispc


CMake Error at SCHISM-interface/CMakeLists.txt:99 (add_library):
  No SOURCES given to target: schism

for ./compile.sh hera "-DAPP=CSTLS -DUSE_WWM=ON -DNO_PARMETIS=OFF -DOLDIO=ON" "coastalSWWM" intel YES NO

It gives :

stack-1.5.0/envs/unified-env-noavx512/install/intel/2021.5.0/netcdf-c-4.9.2-pc2lepn/lib/libnetcdf.so;/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.5.0/envs/unified-env-noavx512/install/intel/2021.5.0/netcdf-fortran-4.6.0-wplqtff/lib/libnetcdff.so
-- In Utility/Gen_Hotstart
-- In Utility/Grid_Scripts
-- Configuring done
CMake Error at SCHISM-interface/CMakeLists.txt:99 (add_library):
  Cannot find source file:

    /scratch2/STI/coastal/Yunfang.Sun/run_schism/schism_wwm/ufs-coastal-coastal_app/tests/build_fv3_coastalSWWM/SCHISM-interface/SCHISM/src/Core/CMakeFiles/core.dir/schism_version.F90.o/scratch2/STI/coastal/Yunfang.Sun/run_schism/schism_wwm/ufs-coastal-coastal_app/tests/build_fv3_coastalSWWM/SCHISM-interface/SCHISM/src/Hydro/CMakeFiles/hydro.dir/schism_init.F90.o

  Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm .h
  .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .hip .ispc


CMake Error at SCHISM-interface/CMakeLists.txt:99 (add_library):
  No SOURCES given to target: schism


CMake Generate step failed.  Build files cannot be regenerated correctly.

And the line 99 looks like:

# SCHISM
if(USE_WWM)
  add_library(schism STATIC ${schism_nuopc_src_files} $<TARGET_OBJECTS:core>$<TARGET_OBJECTS:hydro> $<TARGET_OBJECTS:wwmIII>)
else(USE_WWM)
  add_library(schism STATIC ${schism_nuopc_src_files} $<TARGET_OBJECTS:core>$<TARGET_OBJECTS:hydro>)
endif(USE_WWM)

My work folder is located at /scratch2/STI/coastal/Yunfang.Sun/run_schism/schism_wwm/ufs-coastal-coastal_app/ on Hera.

Thank you!

@pvelissariou1
Copy link
Collaborator

@yunfangsun Lets talk about this in the meeting. I'll try it again on hera

@pvelissariou1 pvelissariou1 mentioned this issue Mar 7, 2024
Open
@danishyo
Copy link

Hi, I am doing schism RT on frontera with the following command.
./rt.sh -a OCE22003 -l rt_coastal.conf -k -n "coastal_ike_shinnecock_atm2sch intel"

It's a bit odd the script won't select the specified test.
Instead, it always choose the first test "atm2adc_intel" in rt_coastal.conf.
(with -v option, ${COMPILE_ID}]=atm2adc_intel )
Is there anything I missed?

@uturuncoglu
Copy link
Collaborator Author

@danishyo This is known as issue in UFS weather model and once we sync the UFS coastal we also inherited the bug. It will be fixed soon. In the meantime time you could edit conf file and just keep the test that you want to run and run rt.sh without -n option. I hope it helps

@danishyo
Copy link

Thanks, @uturuncoglu !
I actually do RT with the way you just suggested.

@danishyo
Copy link

Hi

I try to run RT coastal_ike_shinnecock_atm2sch2ww3 with latest code on Frontera without a success.
(coastal_ike_shinnecock_atm2sch is OK)
Compile is OK, but model stops right before the stepping.
Is there anything I'm missed?

I also notice when I do RT coastal_ike_shinnecock_atm2sch on Hercules, DATM_ATMMESH_input_data is missing under input-data-20221101.
Is the folder changed?

@uturuncoglu
Copy link
Collaborator Author

@danishyo Let me try in my end. As I know it was working fine but it is always nice to double check. If I remember correctly, we might need to run ww3_grid again to recreate some of the wave files wince new version of WW3 was not happy with old version of the code. Maybe I could sync the input directory again to solve the issue. I'll update you.

@danishyo
Copy link

Thanks @uturuncoglu
By the way, on Frontera, I need to temporary comment out EMPTY_CHECK in rt.sh to let RT find the input under "input-data-20221101".
Since "develop-20240426" is not existed on Frontera, maybe it is the reason?

@uturuncoglu
Copy link
Collaborator Author

@danishyo Okay. I fixed the files. Now, it is running in my end. The baseline check is failing but I think we need to create baseline again since there was some change in the SCHSIM side. I'll do that too. But, this will not prevent you to run the case.

@danishyo
Copy link

Thanks @uturuncoglu
Now I can run the cases on Frontera without problem, model can step and produce the outputs for both cases.

@janahaddad janahaddad moved this from In Progress to Backlog in ufs-coastal project Sep 23, 2024
@janahaddad janahaddad moved this from Backlog to In Progress in ufs-coastal project Sep 23, 2024
@janahaddad janahaddad added old_integration and removed enhancement New feature or request labels Sep 23, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in ufs-coastal project Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@uturuncoglu uturuncoglu

Labels
old_integration
Projects
ufs-coastal project
Status: Done
Milestone
No milestone
Development

No branches or pull requests
8 participants
@uturuncoglu @platipodium @janahaddad @josephzhang8 @danishyo @pvelissariou1 @saeed-moghimi-noaa @yunfangsun