Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ufs-coastal ww3-schism test case #24

Closed
saeed-moghimi-noaa opened this issue Dec 11, 2023 · 145 comments
Closed

ufs-coastal ww3-schism test case #24

saeed-moghimi-noaa opened this issue Dec 11, 2023 · 145 comments
Assignees
@pvelissariou1 @AliS-Noaa @yunfangsun

Comments

@saeed-moghimi-noaa
Copy link

No description provided.

@saeed-moghimi-noaa saeed-moghimi-noaa converted this from a draft issue Dec 11, 2023
@saeed-moghimi-noaa
Copy link
Author

@yunfangsun

@yunfangsun
Copy link
Collaborator

Hi Saeideh and Ali,@sbanihash @AliS-Noaa

I have already been granted to access the coastal project on HPSS from here, could you please let me know how to get the GFS data and the script to process it?

Thank you very much!

@AliS-Noaa
Copy link

Hello Yunfang,

Here is the path on HERA where you can find the bash script:
/scratch2/NCEPDEV/marine/alisalimi/SWELL-WESTCOAST/GFS-WAVE-2022/dl-hpss.sh

The path you can find the gfs data is also in the bash script.
Cheers,

@yunfangsun
Copy link
Collaborator

@AliS-Noaa
Hi Ali,

Thank you very much! I have got the script, it seems that I also need to apply access to the folder /NCEPPROD/hpssprod/, as I can only get access to the /STI/coastal at this moment.

@yunfangsun
Copy link
Collaborator

Hi Ali @AliS-Noaa ,

I have retrieved the GFS spectra data from HPSS, those files are ASCII files, do you have any scripts to process those files to make netcdf B.C. file?

Thank you very much!

@AliS-Noaa
Copy link

Hi Yunfang,

There are grib2 files which have gridded data. You can export the netcdf files using wgrib2 for your usage. Please let me know if you need me to show you how to do it. Also, the ww3 can process ascii boundary files as well. ww3_bound will process ascii boundary data while ww3_bounc will process netcdf files.

Cheers,
Ali

@pvelissariou1
Copy link
Collaborator

@AliS-Noaa Hi Ali. Do you have any scripts at your disposal to read the ASCII spec files to generate a single spec file (ASCII or NetCDF) that contain the hourly data per day? Maybe python/matlab scripts? This will be very helpful to us.

@AliS-Noaa
Copy link

Yunfang,

You should be able to use some of these scripts that @aliabdolali created to handle spectral data.
https://github.com/NOAA-EMC/WW3-tools/tree/develop/matlab_tools

There are scripts to read and write spectral ww3 data.
Ali

@yunfangsun
Copy link
Collaborator

Hi Ali,

The files I downloaded are located at /scratch2/STI/coastal/Yunfang.Sun/HPSS on Hera, there are no grib2 files, only ASCII files. Could you please help me to check if I am using the correct files in that folder? I was using /scratch2/STI/coastal/Yunfang.Sun/HPSS/dl-hpss.sh to retrieve the files.

Thank you very much!

Best,

Yunfang

@AliS-Noaa
Copy link

AliS-Noaa commented Dec 18, 2023 via email

@yunfangsun
Copy link
Collaborator

Hi Ali,

Which module should I load to use "htar -tvf /NCEPPROD/hpssprod/runhistory/rh2022/202212/20221212/com_gfs_v16.3_gfs.20221212_18.gfs.tar"?

As I got the following error:

[hpsscore1]/NCEPPROD->htar -tvf /NCEPPROD/hpssprod/runhistory/rh2022/202212/20221212/com_gfs_v16.3_gfs.20221212_18.gfs.tar
*** unrecognized command: 'htar -tvf /NCEPPROD/hpssprod/runhistory/rh2022/202212/20221212/com_gfs_v16.3_gfs.20221212_18.gfs.tar'
[hpsscore1]/NCEPPROD->

Thank you!

Yunfang

@AliS-Noaa
Copy link

AliS-Noaa commented Dec 18, 2023 via email

@yunfangsun
Copy link
Collaborator

Hi Ali,

I have loaded hpss by using module load hpss, and usedhsito enter the HPSS system, and it gives me the following error

[hpsscore1]/NCEPPROD->htar -tvf /NCEPPROD/hpssprod/runhistory/rh2022/202212/20221212/com_gfs_v16.3_gfs.20221212_18.gfs.tar
*** unrecognized command: 'htar -tvf /NCEPPROD/hpssprod/runhistory/rh2022/202212/20221212/com_gfs_v16.3_gfs.20221212_18.gfs.tar'
[hpsscore1]/NCEPPROD->

Do you have a clue, or do I need to contact the HPSS help?

Thank you!

Yunfang

@saeed-moghimi-noaa
Copy link
Author

Yunfang, Please use: module load hpss Cheers, Ali -------------------------------------------------------- Ali Salimi-Tarazouj, Ph.D. Physical Scientist, Coastal Engineer Lynker at NOAA/NWS/NCEP/EMC 5830 University Research Court College Park, Maryland, 20740 Office: (202) 964-0965 Mobile: (302) 588-5505
…
On Mon, Dec 18, 2023 at 11:29 AM Yunfang Sun @.> wrote: Hi Ali, Which module should I load to use "htar -tvf /NCEPPROD/hpssprod/runhistory/rh2022/202212/20221212/com_gfs_v16.3_gfs.20221212_18.gfs.tar"? As I got the following error: [hpsscore1]/NCEPPROD->htar -tvf /NCEPPROD/hpssprod/runhistory/rh2022/202212/20221212/com_gfs_v16.3_gfs.20221212_18.gfs.tar *** unrecognized command: 'htar -tvf /NCEPPROD/hpssprod/runhistory/rh2022/202212/20221212/com_gfs_v16.3_gfs.20221212_18.gfs.tar' [hpsscore1]/NCEPPROD-> Thank you! Yunfang — Reply to this email directly, view it on GitHub <#24 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4J7D7U4WG52OVKWCFXQMA3YKBVOZAVCNFSM6AAAAABAQG6ODGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRQHE3TCNZYHE . You are receiving this because you were mentioned.Message ID: @.>

@AliS-Noaa Thanks for your help.

@aliabdolali
Copy link

From what i recall, you need to have hpss access from your FOM otherwise you cannot access hpss

@pvelissariou1
Copy link
Collaborator

pvelissariou1 commented Dec 18, 2023 via email

@yunfangsun
Copy link
Collaborator

Hi @aliabdolali ,

Thank you for your reply, I figured out that the command should be used on Hera, not inside the HPSS.

Best,

Yunfang

@yunfangsun
Copy link
Collaborator

Hi @aliabdolali @AliS-Noaa,
I want to set the WW3 with only surface wind forcing and without open boundaries for the Atlantic domain, could I know which case in the regression tests could be an example for me to make my case?
Thank you very much!
Best,
Yunfang

@aliabdolali
Copy link

Hi @aliabdolali @AliS-Noaa, I want to set the WW3 with only surface wind forcing and without open boundaries for the Atlantic domain, could I know which case in the regression tests could be an example for me to make my case? Thank you very much! Best, Yunfang

if you do not execute ww3_bound or ww3_bounc, it does not generate nest.ww3 file and therefore the model does not use boundary condition.

@yunfangsun
Copy link
Collaborator

Hi Ali @aliabdolali ,

Thank you for your reply.

When I try to use the Global ERA5 wind forcing, I can't produce the wind.ww3, in my ww3_prnc.inp, the setting is as follows:

$
  'WND' 'LL' T T
$
$ Name of dimensions ------------------------------------------------- $
$
$ longitude latitude time
 time, latitude, longitude
$
$ Variables to use --------------------------------------------------- $
$
  u10 v10
$  uwnd vwnd
$zeta
$u-vel v-vel
$
$ Additional time input ---------------------------------------------- $
$ If time flag is .FALSE., give time of field in yyyymmdd hhmmss format.
$
$   19680606 053000
$
$ Define data files -------------------------------------------------- $
$ The input line identifies the filename using for the forcing field.
download_inv.nc
$
$ -------------------------------------------------------------------- $
$ End of input file                                                    $
$ -------------------------------------------------------------------- $

And the download_inv.nc file looks like this:

 ncdump -h download_inv.nc
netcdf download_inv {
dimensions:
	time = UNLIMITED ; // (1464 currently)
	longitude = 1440 ;
	latitude = 721 ;
variables:
	int time(time) ;
		time:standard_name = "time" ;
		time:long_name = "time" ;
		time:units = "hours since 1900-01-01 00:00:00.0" ;
		time:calendar = "gregorian" ;
		time:axis = "T" ;
	float longitude(longitude) ;
		longitude:standard_name = "longitude" ;
		longitude:long_name = "longitude" ;
		longitude:units = "degrees_east" ;
		longitude:axis = "X" ;
	float latitude(latitude) ;
		latitude:standard_name = "latitude" ;
		latitude:long_name = "latitude" ;
		latitude:units = "degrees_north" ;
		latitude:axis = "Y" ;
	short u10(time, latitude, longitude) ;
		u10:long_name = "10 metre U wind component" ;
		u10:units = "m s**-1" ;
		u10:add_offset = -0.692859959977449 ;
		u10:scale_factor = 0.00110688284552248 ;
		u10:_FillValue = -32767s ;
		u10:missing_value = -32767s ;
	short v10(time, latitude, longitude) ;
		v10:long_name = "10 metre V wind component" ;
		v10:units = "m s**-1" ;
		v10:add_offset = 3.74782109577442 ;
		v10:scale_factor = 0.00106191001366916 ;
		v10:_FillValue = -32767s ;
		v10:missing_value = -32767s ;
	short d2m(time, latitude, longitude) ;
		d2m:long_name = "2 metre dewpoint temperature" ;
		d2m:units = "K" ;
		d2m:add_offset = 247.993123435418 ;
		d2m:scale_factor = 0.00177497974951253 ;
		d2m:_FillValue = -32767s ;
		d2m:missing_value = -32767s ;
	short msl(time, latitude, longitude) ;
		msl:standard_name = "air_pressure_at_mean_sea_level" ;
		msl:long_name = "Mean sea level pressure" ;
		msl:units = "Pa" ;
		msl:add_offset = 98586.6813332596 ;
		msl:scale_factor = 0.262333480841713 ;
		msl:_FillValue = -32767s ;
		msl:missing_value = -32767s ;

// global attributes:
		:CDI = "Climate Data Interface version 1.9.10 (https://mpimet.mpg.de/cdi)" ;
		:Conventions = "CF-1.6" ;
		:history = "Mon Sep 18 10:02:45 2023: cdo invertlat download.nc download_inv.nc\n",
			"2023-09-18 13:34:14 GMT by grib_to_netcdf-2.25.1: /opt/ecmwf/mars-client/bin/grib_to_netcdf.bin -S param -o /cache/data6/adaptor.mars.internal-1695043598.8457577-27378-3-fe079c82-8c62-4e80-8ae1-47406b4a76ff.nc /cache/tmp/fe079c82-8c62-4e80-8ae1-47406b4a76ff-adaptor.mars.internal-1695043353.1076667-27378-3-tmp.grib" ;
		:CDO = "Climate Data Operators version 1.9.10 (https://mpimet.mpg.de/cdo)" ;
}

When I run the ww3_prnc, it gives me the following error:

Abort(1090575) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(143):
MPID_Init(1221)......:
MPIR_pmi_init(168)...: PMI2_Job_GetId returned 14
Abort(1090575) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(143):
MPID_Init(1221)......:
MPIR_pmi_init(168)...: PMI2_Job_GetId returned 14
Abort(1090575) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(143):
MPID_Init(1221)......:
MPIR_pmi_init(168)...: PMI2_Job_GetId returned 14
Abort(1090575) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(143):
MPID_Init(1221)......:
MPIR_pmi_init(168)...: PMI2_Job_GetId returned 14
Abort(1090575) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
MPIR_Init_thread(143):
MPID_Init(1221)......:
MPIR_pmi_init(168)...: PMI2_Job_GetId returned 14

The switch I am using is as follows: NCO PDLIB SCOTCH NOGRB DIST MPI PR3 UQ FLX0 SEED ST4 STAB0 NL1 BT1 DB1 MLIM FLD2 TR0 BS0 RWND WNX1 WNT1 CRX1 CRT1 O0 O1 O2 O3 O4 O5 O6 O7 O14 O15 IC0 IS0 REF0 SCRIP SCRIPNC , and the files are located at /scratch2/STI/coastal/Yunfang.Sun/ww3_hera/ian on Hera.

Could you please give me any suggestions to fix it?

Thank you again!

Best,

Yunfang

@pvelissariou1
Copy link
Collaborator

@yunfangsun You need to run ww3_prnc (and other stanalone ww3 programs) through slurm scheduler and srun. It is the same issue we discussed a few days ago when generating rge ESMFMesh files. Please check any of the setup jobs in the run folder of the ww3 tests in CoastalApp-testsuite. Could you please try this?

@yunfangsun
Copy link
Collaborator

Hi Takis, @pvelissariou1,

I was using slurm and srun, and it seems that this error started from reading the netcdf file


  Description of inputs
 --------------------------------------------------
       Input type        : winds
       Format type       : long.-lat. grid
          Field conserves velocity.

           File name         : download_inv.nc
           Dimension along x : time
           Dimension along y : latitude
           Field component 1 : u10
           Field component 2 : v10
Abort(59) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
Abort(59) on node 3 (rank 3 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3
Abort(59) on node 5 (rank 5 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 5

It mixed the dimension of time and longitude and caused the error

@pvelissariou1
Copy link
Collaborator

@yunfangsun I see that the NetCDF file might have an issue. Is it on hera?

@yunfangsun
Copy link
Collaborator

Hi Takis,

It is /scratch2/STI/coastal/Yunfang.Sun/ww3_hera/ian/download_inv.nc on Hera.

Thank you!

@AliS-Noaa
Copy link

@yunfangsun you forgot to put time on the Name of the dimensions

@uturuncoglu
Copy link
Collaborator

uturuncoglu commented Jan 25, 2024 •
edited
Loading

@yunfangsun Thanks it is very helpful to learn the details. The only think that I would like to say that It is hard for me to keep track all the configuration that you are testing/experimenting in your side since we are working on multiple configurations with multiple model components. I think some how we need to sit and talk about the current development process after AMS since I think it is not working smoothly. We are testing various configurations without knowing which one is running and which one is not. So, my opinion this is a problem since we are wasting our time to tackle issues rather than having robust progress in specified direction. Anyway, I'll modify your configuration to use internal WW3. If this what we want at this point? I am just asking for the confirmation before trying to run it. Thanks again for your effort to make this case available. I hope we could run it.

@uturuncoglu
Copy link
Collaborator

@pvelissariou1 I changed the build like following,

diff --git a/SCHISM-interface/CMakeLists.txt b/SCHISM-interface/CMakeLists.txt
index c5ade4c0..f2713f17 100644
--- a/SCHISM-interface/CMakeLists.txt
+++ b/SCHISM-interface/CMakeLists.txt
@@ -95,7 +95,11 @@ list(APPEND schism_nuopc_src_files
 ###############################################################################
 
 # SCHISM
-add_library(schism STATIC ${schism_nuopc_src_files} $<TARGET_OBJECTS:core> $<TARGET_OBJECTS:hydro>)
+if(USE_WWM)
+  add_library(schism STATIC ${schism_nuopc_src_files} $<TARGET_OBJECTS:core> $<TARGET_OBJECTS:hydro> $<TARGET_OBJECTS:wwmIII>)
+else(USE_WWM)
+  add_library(schism STATIC ${schism_nuopc_src_files} $<TARGET_OBJECTS:core> $<TARGET_OBJECTS:hydro>)
+endif(USE_WWM)
 set_target_properties(schism PROPERTIES Fortran_MODULE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/mod)
 target_include_directories(schism PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}/mod> $<INSTALL_INTERFACE:mod>)
 list(APPEND schism_libs esmf)

but I am getting following,

/work/noaa/nems/tufuk/COASTAL/ufs-coastal/SCHISM-interface/SCHISM/src/WWMIII/wwm_parall_solver.F90(2214): error #6580: Name in only-list does not exist or is not accessible.   [EXCHANGE_P4D_WWM]
      USE datapool, only : exchange_p4d_wwm, myrank
---------------------------^
/work/noaa/nems/tufuk/COASTAL/ufs-coastal/SCHISM-interface/SCHISM/src/WWMIII/wwm_parall_solver.F90(2632): error #6580: Name in only-list does not exist or is not accessible.   [EXCHANGE_P4D_WWM]
      USE datapool, only : exchange_p4d_wwm, myrank
---------------------------^
/work/noaa/nems/tufuk/COASTAL/ufs-coastal/SCHISM-interface/SCHISM/src/WWMIII/wwm_parall_solver.F90(3315): error #6580: Name in only-list does not exist or is not accessible.   [EXCHANGE_P4D_WWM]
      USE datapool, only : myrank, exchange_p4d_wwm, WRITESTATFLAG
-----------------------------------^

Where this datapool is defined? Do I need any other change?

@pvelissariou1
Copy link
Collaborator

@uturuncoglu You need to upgrade SCHISM to its latest version. Have you done this?

@yunfangsun
Copy link
Collaborator

@uturuncoglu

@yunfangsun Thanks it is very helpful to learn the details. The only think that I would like to say that It is hard for me to keep track all the configuration that you are testing/experimenting in your side since we are working on multiple configurations with multiple model components. I think some how we need to sit and talk about the current development process after AMS since I think it is not working smoothly. We are testing various configurations without knowing which one is running and which one is not. So, my opinion this is a problem since we are wasting our time to tackle issues rather than having robust progress in specified direction. Anyway, I'll modify your configuration to use internal WW3. If this what we want at this point? I am just asking for the confirmation before trying to run it. Thanks again for your effort to make this case available. I hope we could run it.

We are now using the external WW3 to run the coupling test, please don't modify the configuration to use internal WW3. It is because they are using completely different namelist and inputfiles for internal and external wave cases.

@pvelissariou1
Copy link
Collaborator

@uturuncoglu Please just update SCHISM and modify the cmake files. The configurations (RT cases and such) remain as they are. We need to support these options (internal/external wave model) when compiling SCHISM. @yunfangsun needs to run SCHISM using both internal and external wave models. He is working on the configurations for both options.

@yunfangsun yunfangsun mentioned this issue Jan 25, 2024
Closed
@josephzhang8
Copy link
Collaborator

@saeed-moghimi-noaa
There are quite a few variables that SCHISM needs from WW3 for 3D vortex formalism, depending on how simple/complex options are. We followed Bennis-Ardhuin formulation which can be seen in this paper:

http://ccrm.vims.edu/yinglong/SVN_large_files/Bennis_etal_OM2011-Wave-Current.pdf

I'll compile a list for the simplest case soon. Thx

@saeed-moghimi-noaa
Copy link
Author

@josephzhang8

Please start a new issue with more specific title for the request from ww3 folks.

Thanks,
-Saeed

@uturuncoglu
Copy link
Collaborator

@saeed-moghimi-noaa @josephzhang8 We have WW3 fork. So you could create issue in there. https://github.com/oceanmodeling/WW3. I am also plaining to create fork for SCHISM. So, once we have those. You could also create issue in SCHISM fork. Let me know your preference. At this point we are aiming to create specific issue for each development/support request.

@josephzhang8
Copy link
Collaborator

@saeed-moghimi-noaa Aron tested WW3 with SECOFS mesh and found an error due to single precision used there. Hopefully the new project from EMC will fix that.

Meanwhile I suggest we send him the 'nicer' mesh if you still have problem with that.

@saeed-moghimi-noaa
Copy link
Author

@yunfangsun

Please share your subset mesh with Joseph.

Thanks

@saeed-moghimi-noaa
Copy link
Author

I mean the one you subset from 120 m mesh.

@saeed-moghimi-noaa
Copy link
Author

@josephzhang8

@yunfangsun already run this mesh with WW3 and worked even with checks on. I think we really hoping to get the SCHISM original mesh working. Is there a way to turn on/off single precision flag!?

@yunfang please chime in!

@yunfangsun
Copy link
Collaborator

Hi @josephzhang8 ,

My subset mesh from 120 m ADCIRC mesh works for SCHISM stand-alone, WW3 stand-alone cases, and UFS-Coastal ATM+WW3, could you please help me check this mesh?

It is located at Orion/Hercules: /work2/noaa/nos-surge/yunfangs/stmp/yunfangs/FV3_RT/rt_11601_atm2sch2ww3/coastal_ian_atm2sch2ww3_intel_2_adcirc01242024/hgrid.gr3

Thank you!

@josephzhang8
Copy link
Collaborator

@yunfangsun if it's already working, then what do you want me/Aron to check?
@saeed: double precision is not an option in WW3 at the moment.

@yunfangsun
Copy link
Collaborator

@josephzhang8 This mesh is not working for the ATM+SCHISM+WW3 case

@josephzhang8
Copy link
Collaborator

Thx @yunfangsun. Since it's working with WW3 alone, I'm not sure Aron can help. I guess this is in the court of @uturuncoglu

@saeed-moghimi-noaa
Copy link
Author

saeed-moghimi-noaa commented Feb 12, 2024 via email

@josephzhang8
Copy link
Collaborator

Understood @saeed-moghimi-noaa

He told me it'd take him some time to fix that.

@saeed-moghimi-noaa
Copy link
Author

@josephzhang8
Thanks for following up on this important feature.

@janahaddad janahaddad moved this to In Progress in ufs-coastal project Mar 21, 2024
@janahaddad janahaddad moved this from In Progress to Backlog in ufs-coastal project Apr 1, 2024
@yunfangsun yunfangsun mentioned this issue May 16, 2024
Open
29 tasks
@janahaddad
Copy link
Collaborator

We've "reset" this general ww3+schism coupling task with #92, so closing here.

@github-project-automation github-project-automation bot moved this from Backlog to Done in ufs-coastal project May 21, 2024
@josephzhang8
Copy link
Collaborator

Updates from me: I found the WW3's esmf caps and most of variables SCHISM needs for 3D coupling. However, I still do not know how to add missing ones (Hs, wave direction etc); I think this requires updating the dictionary.

@pvelissariou1
Copy link
Collaborator

@josephzhang8 Joseph hi, I have the impression (I might be wrong) that in addition to the esmf cap the WW3 code itself needs to be modified as well so all these variables can be exported. The dictionary might need to be modified as well.

@platipodium
Copy link
Member

The dictionary might need to be modified as well.

In the SCHISM cap we have the function

 src/schism/schism_nuopc_util.F90:  NUOPC_FieldDictionaryAddIfNeeded()

Is this what you're looking for?

@josephzhang8
Copy link
Collaborator

Thx @platipodium. I'm not sure how to add the info from WW3 side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pvelissariou1 pvelissariou1

@AliS-Noaa AliS-Noaa

@yunfangsun yunfangsun

Labels
None yet
Projects
ufs-coastal project
Status: Done
Milestone
No milestone
Development

No branches or pull requests