-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
starting surf trip EDS model notebook #116
base: main
Are you sure you want to change the base?
Conversation
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
@ocefpaf might need your help here. Check out the last cell. I am trying to iterate through each model, extract a time-series for a specific variable ( First, only the last timeseries stays on the plot (probably because I'm overwriting Second, it takes forever to run the cell (see below for
|
Can you run |
That is not the problem. What you are doing there is correct. I cannot confirm for all models b/c I cannot access them but, for the |
I'm getting errors during the write step. See the updated notebook and broken netCDF file included in the most recent commit, and below: ---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
File :39
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\dataset.py:1903, in Dataset.to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf)
1900 encoding = {}
1901 from ..backends.api import to_netcdf
-> 1903 return to_netcdf( # type: ignore # mypy cannot resolve the overloads:(
1904 self,
1905 path,
1906 mode=mode,
1907 format=format,
1908 group=group,
1909 engine=engine,
1910 encoding=encoding,
1911 unlimited_dims=unlimited_dims,
1912 compute=compute,
1913 multifile=False,
1914 invalid_netcdf=invalid_netcdf,
1915 )
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\backends\api.py:1230, in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf)
1225 # TODO: figure out how to refactor this logic (here and in save_mfdataset)
1226 # to avoid this mess of conditionals
1227 try:
1228 # TODO: allow this work (setting up the file for writing array data)
1229 # to be parallelized with dask
-> 1230 dump_to_store(
1231 dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims
1232 )
1233 if autoclose:
1234 store.close()
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\backends\api.py:1277, in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
1274 if encoder:
1275 variables, attrs = encoder(variables, attrs)
-> 1277 store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\backends\common.py:266, in AbstractWritableDataStore.store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
263 if writer is None:
264 writer = ArrayWriter()
--> 266 variables, attributes = self.encode(variables, attributes)
268 self.set_attributes(attributes)
269 self.set_dimensions(variables, unlimited_dims=unlimited_dims)
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\backends\common.py:355, in WritableCFDataStore.encode(self, variables, attributes)
352 def encode(self, variables, attributes):
353 # All NetCDF files get CF encoded by default, without this attempting
354 # to write times, for example, would fail.
--> 355 variables, attributes = cf_encoder(variables, attributes)
356 variables = {k: self.encode_variable(v) for k, v in variables.items()}
357 attributes = {k: self.encode_attribute(v) for k, v in attributes.items()}
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\conventions.py:868, in cf_encoder(variables, attributes)
865 # add encoding for time bounds variables if present.
866 _update_bounds_encoding(variables)
--> 868 new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}
870 # Remove attrs from bounds variables (issue #2921)
871 for var in new_vars.values():
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\conventions.py:868, in (.0)
865 # add encoding for time bounds variables if present.
866 _update_bounds_encoding(variables)
--> 868 new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}
870 # Remove attrs from bounds variables (issue #2921)
871 for var in new_vars.values():
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\conventions.py:273, in encode_cf_variable(var, needs_copy, name)
264 ensure_not_multiindex(var, name=name)
266 for coder in [
267 times.CFDatetimeCoder(),
268 times.CFTimedeltaCoder(),
(...)
271 variables.UnsignedIntegerCoder(),
272 ]:
--> 273 var = coder.encode(var, name=name)
275 # TODO(shoyer): convert all of these to use coders, too:
276 var = maybe_encode_nonstring_dtype(var, name=name)
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\coding\times.py:680, in CFDatetimeCoder.encode(self, variable, name)
679 def encode(self, variable, name=None):
--> 680 dims, data, attrs, encoding = unpack_for_encoding(variable)
681 if np.issubdtype(data.dtype, np.datetime64) or contains_cftime_datetimes(
682 variable
683 ):
684 (data, units, calendar) = encode_cf_datetime(
685 data, encoding.pop("units", None), encoding.pop("calendar", None)
686 )
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\coding\variables.py:104, in unpack_for_encoding(var)
103 def unpack_for_encoding(var):
--> 104 return var.dims, var.data, var.attrs.copy(), var.encoding.copy()
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\variable.py:435, in Variable.data(self)
433 return self._data
434 else:
--> 435 return self.values
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\variable.py:608, in Variable.values(self)
605 @property
606 def values(self):
607 """The variable's data as a numpy.ndarray"""
--> 608 return _as_array_or_item(self._data)
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\variable.py:314, in _as_array_or_item(data)
300 def _as_array_or_item(data):
301 """Return the given values as a numpy array, or as an individual item if
302 it's a 0d datetime64 or timedelta64 array.
303
(...)
312 TODO: remove this (replace with np.asarray) once these issues are fixed
313 """
--> 314 data = np.asarray(data)
315 if data.ndim == 0:
316 if data.dtype.kind == "M":
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\indexing.py:653, in MemoryCachedArray.__array__(self, dtype)
652 def __array__(self, dtype=None):
--> 653 self._ensure_cached()
654 return np.asarray(self.array, dtype=dtype)
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\indexing.py:650, in MemoryCachedArray._ensure_cached(self)
648 def _ensure_cached(self):
649 if not isinstance(self.array, NumpyIndexingAdapter):
--> 650 self.array = NumpyIndexingAdapter(np.asarray(self.array))
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\indexing.py:623, in CopyOnWriteArray.__array__(self, dtype)
622 def __array__(self, dtype=None):
--> 623 return np.asarray(self.array, dtype=dtype)
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\indexing.py:524, in LazilyIndexedArray.__array__(self, dtype)
522 def __array__(self, dtype=None):
523 array = as_indexable(self.array)
--> 524 return np.asarray(array[self.key], dtype=None)
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\backends\netCDF4_.py:85, in NetCDF4ArrayWrapper.__getitem__(self, key)
84 def __getitem__(self, key):
---> 85 return indexing.explicit_indexing_adapter(
86 key, self.shape, indexing.IndexingSupport.OUTER, self._getitem
87 )
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\core\indexing.py:815, in explicit_indexing_adapter(key, shape, indexing_support, raw_indexing_method)
793 """Support explicit indexing by delegating to a raw indexing method.
794
795 Outer and/or vectorized indexers are supported by indexing a second time
(...)
812 Indexing result, in the form of a duck numpy-array.
813 """
814 raw_key, numpy_indices = decompose_indexer(key, shape, indexing_support)
--> 815 result = raw_indexing_method(raw_key.tuple)
816 if numpy_indices.tuple:
817 # index the loaded np.ndarray
818 result = NumpyIndexingAdapter(np.asarray(result))[numpy_indices]
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\backends\netCDF4_.py:98, in NetCDF4ArrayWrapper._getitem(self, key)
96 with self.datastore.lock:
97 original_array = self.get_array(needs_lock=False)
---> 98 array = getitem(original_array, key)
99 except IndexError:
100 # Catch IndexError in netCDF4 and return a more informative
101 # error message. This is most often called when an unsorted
102 # indexer is used before the data is loaded from disk.
103 msg = (
104 "The indexing operation you are attempting to perform "
105 "is not valid on netCDF4.Variable object. Try loading "
106 "your data into memory first by calling .load()."
107 )
File ~\programs\Miniforge\envs\IOOS\Lib\site-packages\xarray\backends\common.py:69, in robust_getitem(array, key, catch, max_retries, initial_delay)
67 for n in range(max_retries + 1):
68 try:
---> 69 return array[key]
70 except catch:
71 if n == max_retries:
File src\netCDF4\_netCDF4.pyx:4973, in netCDF4._netCDF4.Variable.__getitem__()
File src\netCDF4\_netCDF4.pyx:5930, in netCDF4._netCDF4.Variable._get()
File src\netCDF4\_netCDF4.pyx:2028, in netCDF4._netCDF4._ensure_nc_success()
RuntimeError: NetCDF: DAP failure |
Add Caribbean SWAN Height @ CARICOOS SWAN Wave Height: https://edsdata.oceansmap.com/thredds/catalog/EDS/CARIB_SWAN/catalog.html |
Merge branch 'main' into surf_trip
adding erddap calls for sensor realtime data
@ocefpaf things are looking better in this notebook. I think TDS is behaving well and we have a good start. Marking this as ready for review, knowing that work needs to be done on the story telling side. |
I have some minor comments but it is good to go with or without them.
|
closes #99