From fb5fb10d941899eeb2f8955c90458b4ded199446 Mon Sep 17 00:00:00 2001 From: jklenzing Date: Thu, 7 Sep 2023 11:56:58 -0400 Subject: [PATCH 01/10] DOC: add examples --- docs/examples/ex_archival.rst | 64 +++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 docs/examples/ex_archival.rst diff --git a/docs/examples/ex_archival.rst b/docs/examples/ex_archival.rst new file mode 100644 index 00000000..e2fbab02 --- /dev/null +++ b/docs/examples/ex_archival.rst @@ -0,0 +1,64 @@ +Building data files for archival at NASA SPDF +============================================= + +The codes and routines at pysatNASA are designed for end-users of NASA data +products. However, pysat in general has also been used to build operational +instruments for generating archival data to be uploaded to the Space Physics +Data Facility (SPDF) at NASA. + +In general, such instruments should include separate naming conventions. An +example of this is the REACH data, where netCDF4 files are generated for +archival purposes as part of the `ops_reach` package, but can be accessed by +the end user through pysatNASA. + +In general, a ``pysat.Instrument`` object can be constructed for any dataset. In +case of the REACH data, the operational code reads in a series of csv files and +updates the metadata according to user specifications. Once the file is loaded, +it can be exported to a netCDF4 file via pysat. In the simplest case, this is + +:: + + reach = pysat.Instrument(inst_module=aero_reach, tag='l1b', inst_id=inst_id) + pysat.utils.io.inst_to_netcdf(reach, 'output_file.nc', epoch_name='Epoch') + + +However, there are additional options when translating pysat metadata to SPDF +preferred formats. An example of this is + +:: + + # Use meta translation table to include SPDF preferred format. + # Note that multiple names are output for compliance with pysat. + # Using the most generalized form for labels for future compatibility. + meta_dict = {reach.meta.labels.min_val: ['VALIDMIN'], + reach.meta.labels.max_val: ['VALIDMAX'], + reach.meta.labels.units: ['UNITS'], + reach.meta.labels.name: ['CATDESC', 'LABLAXIS', 'FIELDNAM'], + reach.meta.labels.notes: ['VAR_NOTES'], + reach.meta.labels.fill_val: ['_FillValue'], + 'Depend_0': ['DEPEND_0'], + 'Format': ['FORMAT'], + 'Monoton': ['MONOTON'], + 'Var_Type': ['VAR_TYPE']} + + pysat.utils.io.inst_to_netcdf(reach, 'output_file.nc', epoch_name='Epoch', + meta_translation=meta_dict, + export_pysat_info=False) + + +In this case, note that the pysat 'name' label is output to three different +metadata values required by the ITSP standards. Additionally, the +``export_pysat_info`` option is set to false here. This drops several internal +pysat metadata values before writing to file. + +Other best practices for archival include adding the oprational software version +to the metadata header before writing. The pysat version will be automatically +written to the metadata. + +:: + + reach.meta.header.Software_version = ops_reach.__version__ + + +A full example script to generate output files can be found at +https://github.com/jklenzing/ops_reach/blob/main/scripts/netcdf_gen.py From a3ff1adce863f4f927034cc274d3b63012c79b26 Mon Sep 17 00:00:00 2001 From: jklenzing Date: Thu, 7 Sep 2023 11:57:50 -0400 Subject: [PATCH 02/10] DOC: update changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2091a7c1..65a48291 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,8 @@ All notable changes to this project will be documented in this file. This project adheres to [Semantic Versioning](https://semver.org/). ## [0.X.X] - 2023-XX-XX +* Documentation + * Added example of how to export data for archival * Maintenance * Implemented unit tests for cleaning warnings * Use pip install for readthedocs From 24afa8b6c4e1ee819d7b7d0ff89cdc2ae9b56164 Mon Sep 17 00:00:00 2001 From: jklenzing Date: Thu, 7 Sep 2023 12:00:21 -0400 Subject: [PATCH 03/10] DOC: add link to pysat conventions --- docs/examples/ex_archival.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/examples/ex_archival.rst b/docs/examples/ex_archival.rst index e2fbab02..f7ee7151 100644 --- a/docs/examples/ex_archival.rst +++ b/docs/examples/ex_archival.rst @@ -11,7 +11,9 @@ example of this is the REACH data, where netCDF4 files are generated for archival purposes as part of the `ops_reach` package, but can be accessed by the end user through pysatNASA. -In general, a ``pysat.Instrument`` object can be constructed for any dataset. In +In general, a ``pysat.Instrument`` object can be constructed for any dataset. +Full instructions and conventions can be found +`here `_. In the case of the REACH data, the operational code reads in a series of csv files and updates the metadata according to user specifications. Once the file is loaded, it can be exported to a netCDF4 file via pysat. In the simplest case, this is From 7c8c5ae084d2d0755aad585cbc4a640e7fb07ec0 Mon Sep 17 00:00:00 2001 From: jklenzing Date: Thu, 7 Sep 2023 12:06:27 -0400 Subject: [PATCH 04/10] BUG: fix link --- docs/examples.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/examples.rst b/docs/examples.rst index a2062741..c18593ed 100644 --- a/docs/examples.rst +++ b/docs/examples.rst @@ -6,3 +6,4 @@ tools .. toctree:: examples/ex_init.rst + examples/ex_archival.rst From 183b4aa98ae42e8501403711c6ac1a12e7a4aab6 Mon Sep 17 00:00:00 2001 From: jklenzing Date: Thu, 7 Sep 2023 12:08:41 -0400 Subject: [PATCH 05/10] STY: top level --- docs/{examples/ex_archival.rst => archival.rst} | 0 docs/examples.rst | 2 +- docs/index.rst | 1 + 3 files changed, 2 insertions(+), 1 deletion(-) rename docs/{examples/ex_archival.rst => archival.rst} (100%) diff --git a/docs/examples/ex_archival.rst b/docs/archival.rst similarity index 100% rename from docs/examples/ex_archival.rst rename to docs/archival.rst diff --git a/docs/examples.rst b/docs/examples.rst index c18593ed..df897a6b 100644 --- a/docs/examples.rst +++ b/docs/examples.rst @@ -6,4 +6,4 @@ tools .. toctree:: examples/ex_init.rst - examples/ex_archival.rst + diff --git a/docs/index.rst b/docs/index.rst index cda7cf4f..926c4f76 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -18,6 +18,7 @@ CDAWeb interface. supported_constellations.rst examples.rst develop_guide.rst + archival.rst migration_guide.rst history.rst From 64cb003313fe027ad6f690f4571347eb539fcfb8 Mon Sep 17 00:00:00 2001 From: jklenzing Date: Thu, 7 Sep 2023 12:19:40 -0400 Subject: [PATCH 06/10] BIG: spelling --- docs/archival.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/archival.rst b/docs/archival.rst index f7ee7151..61efffb1 100644 --- a/docs/archival.rst +++ b/docs/archival.rst @@ -53,7 +53,7 @@ metadata values required by the ITSP standards. Additionally, the ``export_pysat_info`` option is set to false here. This drops several internal pysat metadata values before writing to file. -Other best practices for archival include adding the oprational software version +Other best practices for archival include adding the operational software version to the metadata header before writing. The pysat version will be automatically written to the metadata. From 7660162c572d61147ff826b58b34ef7cb8d7e59f Mon Sep 17 00:00:00 2001 From: Jeff Klenzing <19592220+jklenzing@users.noreply.github.com> Date: Thu, 7 Sep 2023 12:44:02 -0400 Subject: [PATCH 07/10] Update docs/archival.rst --- docs/archival.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/archival.rst b/docs/archival.rst index 61efffb1..72b710d6 100644 --- a/docs/archival.rst +++ b/docs/archival.rst @@ -53,6 +53,7 @@ metadata values required by the ITSP standards. Additionally, the ``export_pysat_info`` option is set to false here. This drops several internal pysat metadata values before writing to file. +A full guide to SPDF metadata standards can be found `here `_. Other best practices for archival include adding the operational software version to the metadata header before writing. The pysat version will be automatically written to the metadata. From 30d4cee099e11ad21f8faa9976eb3942a513a69c Mon Sep 17 00:00:00 2001 From: Jeff Klenzing <19592220+jklenzing@users.noreply.github.com> Date: Thu, 7 Sep 2023 12:44:21 -0400 Subject: [PATCH 08/10] Update docs/archival.rst --- docs/archival.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/archival.rst b/docs/archival.rst index 72b710d6..9c02a932 100644 --- a/docs/archival.rst +++ b/docs/archival.rst @@ -53,7 +53,9 @@ metadata values required by the ITSP standards. Additionally, the ``export_pysat_info`` option is set to false here. This drops several internal pysat metadata values before writing to file. -A full guide to SPDF metadata standards can be found `here `_. +A full guide to SPDF metadata standards can be found +`here `_. + Other best practices for archival include adding the operational software version to the metadata header before writing. The pysat version will be automatically written to the metadata. From 47448103147bfdb51e78652e97ba46684488fd02 Mon Sep 17 00:00:00 2001 From: Jeff Klenzing <19592220+jklenzing@users.noreply.github.com> Date: Thu, 7 Sep 2023 14:51:48 -0400 Subject: [PATCH 09/10] Apply suggestions from code review Co-authored-by: Angeline Burrell --- docs/archival.rst | 2 +- docs/examples.rst | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/archival.rst b/docs/archival.rst index 9c02a932..46de4910 100644 --- a/docs/archival.rst +++ b/docs/archival.rst @@ -1,7 +1,7 @@ Building data files for archival at NASA SPDF ============================================= -The codes and routines at pysatNASA are designed for end-users of NASA data +The codes and routines at :py:mod:`pysatNASA` are designed for end-users of NASA data products. However, pysat in general has also been used to build operational instruments for generating archival data to be uploaded to the Space Physics Data Facility (SPDF) at NASA. diff --git a/docs/examples.rst b/docs/examples.rst index df897a6b..a2062741 100644 --- a/docs/examples.rst +++ b/docs/examples.rst @@ -6,4 +6,3 @@ tools .. toctree:: examples/ex_init.rst - From 57be5e1f3eef94b1e93c991740c9749e33ec3709 Mon Sep 17 00:00:00 2001 From: Jeff Klenzing <19592220+jklenzing@users.noreply.github.com> Date: Thu, 7 Sep 2023 18:48:45 -0400 Subject: [PATCH 10/10] Apply suggestions from code review --- docs/archival.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/archival.rst b/docs/archival.rst index 46de4910..ad1b1d5e 100644 --- a/docs/archival.rst +++ b/docs/archival.rst @@ -8,11 +8,11 @@ Data Facility (SPDF) at NASA. In general, such instruments should include separate naming conventions. An example of this is the REACH data, where netCDF4 files are generated for -archival purposes as part of the `ops_reach` package, but can be accessed by -the end user through pysatNASA. +archival purposes as part of the :py:mod:`ops_reach` package, but can be accessed by +the end user through :py:mod:`pysatNASA`. -In general, a ``pysat.Instrument`` object can be constructed for any dataset. -Full instructions and conventions can be found +In general, a :py:class:`pysat.Instrument` object can be constructed for any +dataset. Full instructions and conventions can be found `here `_. In the case of the REACH data, the operational code reads in a series of csv files and updates the metadata according to user specifications. Once the file is loaded, @@ -50,8 +50,8 @@ preferred formats. An example of this is In this case, note that the pysat 'name' label is output to three different metadata values required by the ITSP standards. Additionally, the -``export_pysat_info`` option is set to false here. This drops several internal -pysat metadata values before writing to file. +:py:attr:`export_pysat_info` option is set to false here. This drops several +internal :py:mod:`pysat` metadata values before writing to file. A full guide to SPDF metadata standards can be found `here `_.