Follow-up to prepare for NOAA AWS S3 data bucket: ufs-coastal baseline maintenance and storage

[jkbk2004]

Description

• Hercules baseline file location: /work2/noaa/nems/tufuk/RT/NEMSfv3gfs
• To check procedures to prepare for the paperwork and requirement of the NOAA Big Data Program (BDP) to create the S3 bucket

Solution

Alternatives

Related to

[jkbk2004]

@janahaddad @pvelissariou1 @uturuncoglu EPIC will go ahead to submit a request to NOAA-BDP to create a UFS-Coastal S3 bucket. It will take a few weeks, I think. BDP will require a landing page as well. We will coordinate to meet the requirement.

[janahaddad]

@jkbk2004 thank you for the update, this seems like a great development.

Re: landing page, we have plans to use https://github.com/janahaddad/ufs-coastal-app as the primary landing page with dedicated readme, etc. This is on our roadmap to get done in the next month or so.

cc'ing @saeed-moghimi-noaa for his awareness

[jkbk2004]

@janahaddad just to touch base regarding the landing page. do you want to move on for draft to finalize? As you suggested, we can use the readme from https://github.com/janahaddad/ufs-coastal-app, right? @CSyl @ankimball FYI

[janahaddad]

@jkbk2004 we need to take some time to collect and restructure the RTs and Test Cases on our side so we can point @CSyl to a good folder structure on one of the platforms to copy/port to the S3. Unfortunately we won't have the bandwidth to do that until probably mid-late July.

But for just setting up the S3 landing page, yes the readme & license from ufs-coastal-app would work.

[zach1221]

Hi, @janahaddad, I wanted to follow up to see when you plan on finalizing landing page ( https://github.com/janahaddad/ufs-coastal-app)? I touched base with @CSyl and she said it needs to be finalized before an s3 bucket can be established.

[CSyl]

Hi @janahaddad ,
I took a look at the readme file you have, but it will not suffice to be used as a RODA landing page for a new S3 bucket establish with NODD. The RODA "landing page" for a newly established S3 bucket seen publicly must be formatted & structured with the following sections:

• Summary Description
• Update Frequency
• License
• Documentation
• Manage By
• Contact

Here is an example/template of a RODA: https://registry.opendata.aws/noaa-ufs-regtests/

Please let me know when there is a final write-up of the RODA so, we can begin establishing a new bucket for the UFS coastal app.
cc @zach1221

[janahaddad]

@uturuncoglu @saeed-moghimi-noaa @CSyl @zach1221 @jkbk2004 , I'm working on a draft RODA landing page here and will share once reviewed/finalized by UFS Coastal team

[janahaddad]

@CSyl @zach1221 @jkbk2004 we're nearly there with this draft RODA, but have a question that may impact the text in the RODA landing page and what RTs we include here: will our team be able to update the S3 fairly easily, or will you need to do that for us? Our RT system is evolving, so ideally we could push changes/updates to the S3 ourselves. Will that be possible?

[jkbk2004]

@gspetro-NOAA @selbigmtnwx23 FYI: data management item across communities.

[selbigmtnwx23]

@janahaddad @gspetro-NOAA it sounds like management on our end is fine with you being able to push the data to the bucket yourself. So I will setup an email thread with you and NODD to get you the IAM credentials you need. And then when the RODA draft is done we will just need to give that to them as well to setup the bucket. Once we've done those steps we can set up a meeting and I can help walk you through the data transfer process assuming you don't already know.

[janahaddad]

@selbigmtnwx23 @gspetro-NOAA thanks so much for this and your email. Just set up a meeting for next week, please add anyone I missed from your team who should be there. Noting the initial questions I had here just for reference:

• What will be the process for initially populating the S3 bucket?
• What will be the process if we want to update/modify the bucket over time?
• Will this bucket be somehow linked to our Jenkins pipeline, so that regression test results/artifacts are uploaded automatically?

Thanks and talk soon