Add eNATL recipe #75
Open
Add eNATL recipe #75
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
pre-commit.ci autofix |
for more information, see https://pre-commit.ci
|
@cisaacstern this failed with the same error I encountered earlier. Could you take a look at this? |
|
Trying out pangeo-forge/deploy-recipe-action#27 |
|
Ok I was able to deploy this using the super hacky changes made in pangeo-forge/deploy-recipe-action#27. I will check later if the dataflow job successfully ran But maybe more important, we need to wait here how the discussion over at pangeo-forge/deploy-recipe-action#27 goes. Sorry for the delay. |
|
pre-commit.ci autofix |
|
Yay, this worked! 
@auraoupa do you have access to the leap hub? You can inspect the dataset with the following snippet: import xarray as xr
path = 'gs://leap-persistent-ro/data-library/enatl60-blbt02-595733423-7175544257-1/eNATL60_BLBT02.zarr'
ds = xr.open_dataset(path, engine='zarr', chunks={})
dsTwo things I noticed:
|
|
There is something even weirder going on around the halo land values: ds = ds.where(abs(ds)<1e20)
ds['vosaline'].isel(time_counter=0)gives me:
What is the best way to get rid of those low values on land in a reliable way? |
|
Finally we should think about the chunking of the final product. These are all things we can/should discuss before we have the deployment figured out. |
|
Than you @jbusecke for advancing so quickly on this !
|
|
Thanks for the quick response @auraoupa. Lets deal with the most challenging issue first:
I think ideally each file would contain the masks as coordinates, then we could apply the masking on each file, and also retain the masks in the final output (this might be very important for budget analysis etc). I have raised pangeo-forge/pangeo-forge-recipes#663 to discuss this more broadly. Just as a heads up, this will probably not move before next week earliest, since folks are at AGU.
I already renamed to
That seems fairly small to me. I would aim for chunksizes in the 100-200MB range, but this is a detail we can discuss at the end. |
|
Hi @jbusecke, I hope you had a nice end of 2023 and wish you the best for 2024 ! |
|
Thanks for doing this @auraoupa. This might unblock us here. I will keep track of it over at pgf. |
* Update eNATL60.py new name and zenodo records for homogeneously masked data * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
|
Seems like we are getting an error that |
|
Ok can confirm the dimension is now named |
|
@auraoupa should |
|
pre-commit.ci autofix |
for more information, see https://pre-commit.ci
|
Well thats a new one (cc @cisaacstern ): Let me change the target_chunks to see if this goes away |
|
Ok now I am getting yet another error that I cannot quite grok: @cisaacstern could we dig into this in the coming days? Sorry this will still be blocked for now @auraoupa. |
Yes forgot that, it is time now.
No it is not dependent on time indeed, sorry I missed it Thanks and good luck with the unusual errors ... |
|
Fixing:
|
|
@SammyAgrawal can you move any further discussion to https://github.com/leap-stc/eNATL_feedstock and close this? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Towards #73
DO NOT MERGE AS IS. HIGHLY EXPERIMENTAL