Commented FDB.h and restructuring

ecmwf · Feb 20, 2024 · fdbf783 · fdbf783
1 parent aaf55e9
commit fdbf783
Show file tree

Hide file tree

Showing 48 changed files with 133 additions and 2,840 deletions.
diff --git a/.gitignore b/.gitignore
@@ -6,4 +6,5 @@ cmake
 **/_build
 build
 install.sh
-.DS_Store
+**/.DS_Store
+**/Doxyfile.in.bak
diff --git a/docs/.DS_Store b/docs/.DS_Store
diff --git a/docs/Doxyfile.in.bak b/docs/Doxyfile.in.bak
diff --git a/docs/content/.DS_Store b/docs/content/.DS_Store
diff --git a/docs/content/architectural-introduction.rst b/docs/content/architectural-introduction.rst
@@ -1,7 +1,7 @@
 .. _architectural-introduction-label:
 
 Architectural Introduction
-##########################
+~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 |Licence|
 

diff --git a/docs/content/architecture/.DS_Store b/docs/content/architecture/.DS_Store
diff --git a/docs/content/architecture/img/.DS_Store b/docs/content/architecture/img/.DS_Store
diff --git a/docs/content/operational-introduction.rst b/docs/content/operational-introduction.rst
@@ -1,4 +1,10 @@
 .. _operational-introduction-label:
 
 Operational Introduction
-########################
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. toctree::
+   :maxdepth: 2
+
+   operational/tools
+
diff --git a/docs/content/technical/DebugTools.rst → docs/content/operational/DebugTools.rst b/docs/content/technical/DebugTools.rst → docs/content/operational/DebugTools.rst
@@ -1,5 +1,5 @@
 Debug Tools
-===========
+"""""""""""
 
 The debugging tools are found here:
 

diff --git a/...ntent/technical/DebugTools/dump-index.rst → ...ent/operational/DebugTools/dump-index.rst b/...ntent/technical/DebugTools/dump-index.rst → ...ent/operational/DebugTools/dump-index.rst
@@ -1,5 +1,5 @@
 fdb dump-index
-==============
+**************
 
 Dump the contents of a particular index file for debugging purposes.
 
@@ -40,4 +40,4 @@ Dump the contents of an index file.
     Contents of index:
       Fingerprint: 0:1000:130, location: FieldRefLocation(pathid=0,offset=0)
       Fingerprint: 0:300:130, location: FieldRefLocation(pathid=0,offset=3280398)
-      ...
+      ...
diff --git a/...content/technical/DebugTools/dump-toc.rst → ...ntent/operational/DebugTools/dump-toc.rst b/...content/technical/DebugTools/dump-toc.rst → ...ntent/operational/DebugTools/dump-toc.rst
@@ -1,5 +1,5 @@
 fdb dump-toc
-============
+************
 
 Description
 -----------
@@ -45,4 +45,4 @@ Dump the contents of a toc walking the subtocs logically. Note that subtocs that
   TOC_INIT  2019-06-07 14:17:01.032360, version:2, fdb: 50308, uid: <id>, pid 31508, host: <host>  Key: {class=rd,expver=xxxx,stream=oper,date=20160907,time=1200,domain=g}, sub-toc: no
   TOC_INDEX 2019-06-07 14:28:31.850438, version:2, fdb: 50308, uid: <id>, pid 644  , host: <host>  Path: an:pl.20190607.142831.<host>.2765958938625.index, offset: 0, type: BTreeIndex  Prefix: an:pl, key: {type=an,levtype=pl}
   TOC_INDEX 2019-06-07 14:17:01.943474, version:2, fdb: 50308, uid: <id>, pid 31508, host: <host>  Path: an:pl.20190607.141701.<host>.135325829562374.index, offset: 0, type: BTreeIndex  Prefix: an:pl, key: {type=an,levtype=pl}
-  ...
+  ...
diff --git a/docs/content/technical/DebugTools/dump.rst → docs/content/operational/DebugTools/dump.rst b/docs/content/technical/DebugTools/dump.rst → docs/content/operational/DebugTools/dump.rst
@@ -1,5 +1,5 @@
 fdb dump
-========
+********
 
 Dump the structural contents of the FDB. In particular, in the TOC formulation, enumerate the different entries in the Table of Contents (including INIT and CLEAR entries).
 

diff --git a/docs/content/technical/DebugTools/hammer.rst → ...content/operational/DebugTools/hammer.rst b/docs/content/technical/DebugTools/hammer.rst → ...content/operational/DebugTools/hammer.rst
@@ -1,5 +1,5 @@
 fdb hammer
-==========
+**********
 
 Description
 -----------

diff --git a/docs/content/technical/DebugTools/patch.rst → .../content/operational/DebugTools/patch.rst b/docs/content/technical/DebugTools/patch.rst → .../content/operational/DebugTools/patch.rst
@@ -1,5 +1,5 @@
 fdb patch
-=========
+*********
 
 Description
 -----------
@@ -53,4 +53,4 @@ Note that this is a global search through all the databases of the FDB that matc
   
   12 fields (37.5412 Mbytes) copied to {expver=xxxz}
   Rates: 114.971 Mbytes per second, 36.7503 fields/s
-  fdb patch: 0.946881 second elapsed, 0.159061 second cpu
+  fdb patch: 0.946881 second elapsed, 0.159061 second cpu
diff --git a/docs/content/technical/DebugTools/read.rst → docs/content/operational/DebugTools/read.rst b/docs/content/technical/DebugTools/read.rst → docs/content/operational/DebugTools/read.rst
@@ -1,5 +1,5 @@
 fdb read
-========
+********
 
 Read data from the FDB and write this data into a specified target file. This may involve visiting multiple databases if required by the request.
 

diff --git a/...echnical/DebugTools/reconsolidate-toc.rst → ...rational/DebugTools/reconsolidate-toc.rst b/...echnical/DebugTools/reconsolidate-toc.rst → ...rational/DebugTools/reconsolidate-toc.rst
@@ -1,5 +1,5 @@
 fdb reconsolidate-toc
-=====================
+*********************
 
 This is an advanced tool that exists to assist cleanup in a situation where databases have been poorly written. This can occur in a number of contexts such as:
 
@@ -12,4 +12,4 @@ In general it is preferable to wipe such databases, and rerun with correct exper
 
 Usage
 -----
-``fdb reconsolidate-toc [database path]``
+``fdb reconsolidate-toc [database path]``
diff --git a/docs/content/technical/DebugTools/write.rst → .../content/operational/DebugTools/write.rst b/docs/content/technical/DebugTools/write.rst → .../content/operational/DebugTools/write.rst
@@ -1,5 +1,5 @@
 fdb write
-=========
+*********
 
 Inserts data into the FDB, creating a new databases if needed.  
 The data is copied into the FDB, and the tool reports the location where it was inserted.  

diff --git a/...content/technical/GeneralPurposeTools.rst → ...ntent/operational/GeneralPurposeTools.rst b/...content/technical/GeneralPurposeTools.rst → ...ntent/operational/GeneralPurposeTools.rst
@@ -1,10 +1,10 @@
 General Purpose Tools
-=====================
+"""""""""""""""""""""
 
 The General Purpose tools are found here:
 
 .. toctree::
    :maxdepth: 1
    :glob:
 
-   GeneralPurposeTools/*
+   GeneralPurposeTools/*
diff --git a/...chnical/GeneralPurposeTools/fdb-stats.rst → ...ational/GeneralPurposeTools/fdb-stats.rst b/...chnical/GeneralPurposeTools/fdb-stats.rst → ...ational/GeneralPurposeTools/fdb-stats.rst
@@ -1,5 +1,5 @@
 fdb stats
-=========
+*********
 
 Prints information about FDB databases, aggregating the information over all the databases visited into a final summary.
 
@@ -33,7 +33,7 @@ Example 1
 
 You may pass a partial request (as a key) that will print information on all FDB databases that match that key.
 
-::
+.. code-block:: bash
 
   % fdb stats class=od,expver=0001,stream=oper,date=20151001
   ...
@@ -67,7 +67,7 @@ Example 2
 
 The --details flag prints a report per database that is visited, as well as the overall summary
 
-::
+.. code-block:: bash
   
   % fdb stats class=od,expver=0001
   ...
@@ -98,4 +98,4 @@ The --details flag prints a report per database that is visited, as well as the
   ========
 
   Number of databases             : 4
-  ...
+  ...
diff --git a/...ent/technical/GeneralPurposeTools/fdb.rst → ...t/operational/GeneralPurposeTools/fdb.rst b/...ent/technical/GeneralPurposeTools/fdb.rst → ...t/operational/GeneralPurposeTools/fdb.rst
@@ -1,5 +1,5 @@
 fdb
-===
+***
 
 Description
 -----------
@@ -38,4 +38,4 @@ fdb list example
   {class=od,expver=0001,stream=oper,date=20151004,time=1200,domain=g}{type=an,levtype=pl}{step=0,levelist=700,param=155}
   {class=od,expver=0001,stream=oper,date=20151004,time=1200,domain=g}{type=an,levtype=pl}{step=0,levelist=850,param=129}
   {class=od,expver=0001,stream=oper,date=20151004,time=1200,domain=g}{type=an,levtype=pl}{step=0,levelist=850,param=130}
-  ...
+  ...
diff --git a/...echnical/GeneralPurposeTools/grib2fdb.rst → ...rational/GeneralPurposeTools/grib2fdb.rst b/...echnical/GeneralPurposeTools/grib2fdb.rst → ...rational/GeneralPurposeTools/grib2fdb.rst
@@ -1,5 +1,5 @@
 grib2fdb5
-=========
+*********
 
 Inserts data into the FDB, creating a new databases if needed.
 
@@ -51,4 +51,4 @@ Check that the supplied keys match
   Processing data.grib
   Key {class=rd,expver=xxxx,type=an,stream=oper}
   FDB archive 12 fields, size 37.5412 Mbytes, in 0.086995 second (431.518 Mbytes per second)
-  fdb::service::archive: 0.087076 second elapsed, 0.087075 second cpu
+  fdb::service::archive: 0.087076 second elapsed, 0.087075 second cpu
diff --git a/...nt/technical/GeneralPurposeTools/info.rst → .../operational/GeneralPurposeTools/info.rst b/...nt/technical/GeneralPurposeTools/info.rst → .../operational/GeneralPurposeTools/info.rst
@@ -1,5 +1,5 @@
 fdb info
-========
+********
 
 Get information about the FDB configuration and binaries
 
@@ -61,4 +61,4 @@ Get location of current FDB configuration file in an easily parsable form:
 ::
   
   % fdb info --config
-  <path>/testcases/fdb5/fdb5_simple.yaml
+  <path>/testcases/fdb5/fdb5_simple.yaml
diff --git a/...nt/technical/GeneralPurposeTools/list.rst → .../operational/GeneralPurposeTools/list.rst b/...nt/technical/GeneralPurposeTools/list.rst → .../operational/GeneralPurposeTools/list.rst
@@ -1,5 +1,5 @@
 fdb list
-========
+********
 
 Lists the contents of the FDB databases.  
 In the body of the output, one line is given per field that has been archived. These (by default) present the fields that are available and will be retrievable - i.e. masked duplicates are skipped.  

diff --git a/...t/technical/GeneralPurposeTools/purge.rst → ...operational/GeneralPurposeTools/purge.rst b/...t/technical/GeneralPurposeTools/purge.rst → ...operational/GeneralPurposeTools/purge.rst
@@ -1,5 +1,5 @@
 fdb purge
-=========
+*********
 
 Purge duplicate entries from the database and remove the associated data (if the data is owned, not adopted).
 
@@ -99,4 +99,4 @@ Additionally pass the --doit flag to delete the duplicates.
   ...
 
   % du -sh /data/fdb/od\:0001\:oper\:20160907\:1200\:g/
-  20M     fdb_root/root/od:0001:oper:20160907:1200:g/
+  20M     fdb_root/root/od:0001:oper:20160907:1200:g/
diff --git a/.../technical/GeneralPurposeTools/schema.rst → ...perational/GeneralPurposeTools/schema.rst b/.../technical/GeneralPurposeTools/schema.rst → ...perational/GeneralPurposeTools/schema.rst
diff --git a/...t/technical/GeneralPurposeTools/where.rst → ...operational/GeneralPurposeTools/where.rst b/...t/technical/GeneralPurposeTools/where.rst → ...operational/GeneralPurposeTools/where.rst
@@ -1,5 +1,5 @@
 fdb where
-=========
+*********
 
 Print the location of FDB5 database.
 

diff --git a/...nt/technical/GeneralPurposeTools/wipe.rst → .../operational/GeneralPurposeTools/wipe.rst b/...nt/technical/GeneralPurposeTools/wipe.rst → .../operational/GeneralPurposeTools/wipe.rst
@@ -1,5 +1,5 @@
 fdb wipe
-========
+********
 
 Deletes FDB databases and the data therein contained. Uses the passed request to identify the database to delete.
 
@@ -142,4 +142,4 @@ Use --minimum-keys with caution! Setting --minimum-keys=class is a BAD IDEA! You
   Unlink /data/fdb5/rd:wxyz:oper:20151004:1200:g/toc
   Unlink /data/fdb5/rd:wxyz:oper:20151004:1200:g/schema
   Unlink /data/fdb5/rd:wxyz:oper:20151004:1200:g/an:pl.20170323.164359.host.30249454665731.data
-  Unlink /data/fdb5/rd:wxyz:oper:20151004:1200:g/an:pl.20170323.164359.host.30249454665730.index
+  Unlink /data/fdb5/rd:wxyz:oper:20151004:1200:g/an:pl.20170323.164359.host.30249454665730.index
diff --git a/...content/technical/SpecialPurposeTools.rst → ...ntent/operational/SpecialPurposeTools.rst b/...content/technical/SpecialPurposeTools.rst → ...ntent/operational/SpecialPurposeTools.rst
@@ -1,10 +1,10 @@
 Special Purpose Tools
-=====================
+"""""""""""""""""""""
 
 The Special Purpose tools are found here:
 
 .. toctree::
    :maxdepth: 1
    :glob:
 
-   SpecialPurposeTools/*
+   SpecialPurposeTools/*
diff --git a/...nt/technical/SpecialPurposeTools/hide.rst → .../operational/SpecialPurposeTools/hide.rst b/...nt/technical/SpecialPurposeTools/hide.rst → .../operational/SpecialPurposeTools/hide.rst
@@ -1,5 +1,5 @@
 fdb hide
-========
+********
 
 Hide the contents of one FDB database. This masks all existing entries in the database such that they are permanently inaccessible, without destructively damaging the data or indexes.
 
@@ -38,4 +38,4 @@ These changes can then be made permanent
 ::
   
   % fdb hide --doit class=rd,expver=xxxx,stream=oper,date=20160907,time=0000,domain=g
-  Hide contents of DB: TocDBReader(<path>/fdb5/root/rd:xxxx:oper:20160907:0000:g)
+  Hide contents of DB: TocDBReader(<path>/fdb5/root/rd:xxxx:oper:20160907:0000:g)
diff --git a/...nt/technical/SpecialPurposeTools/move.rst → .../operational/SpecialPurposeTools/move.rst b/...nt/technical/SpecialPurposeTools/move.rst → .../operational/SpecialPurposeTools/move.rst
@@ -1,5 +1,5 @@
 fdb move
-========
+********
 
 Move the content of one FDB database. This locks the source database, make it possible to create a second database in another root, duplicates all data. Source data are not automatically removed.
 

diff --git a/...technical/SpecialPurposeTools/overlay.rst → ...erational/SpecialPurposeTools/overlay.rst b/...technical/SpecialPurposeTools/overlay.rst → ...erational/SpecialPurposeTools/overlay.rst
@@ -1,5 +1,5 @@
 fdb overlay
-===========
+***********
 
 Make the contents of one FDB database available as though they were archived using different keys (class or expver).
 

diff --git a/...nt/technical/SpecialPurposeTools/root.rst → .../operational/SpecialPurposeTools/root.rst b/...nt/technical/SpecialPurposeTools/root.rst → .../operational/SpecialPurposeTools/root.rst
@@ -1,5 +1,5 @@
 fdb root
-========
+********
 
 Find the location on disk of the specified databases.
 
@@ -42,4 +42,4 @@ Note that the supplied keys must be sufficient to identify the database. This su
   % fdb root --create class=od,expver=0001,stream=oper,date=20160907,time=1200
 
   {class=od,expver=0001,stream=oper,date=20160907,time=1200,domain=g} (Rule[line=128])
-  TocDBWriter(/data/fdb/od:0001:oper:20160907:1200:g)
+  TocDBWriter(/data/fdb/od:0001:oper:20160907:1200:g)
diff --git a/docs/content/technical/tools.rst → docs/content/operational/tools.rst b/docs/content/technical/tools.rst → docs/content/operational/tools.rst
@@ -1,7 +1,7 @@
 .. _tools-label:
 
 FDB Tools
-=========
+`````````
 
 FDB has a set of general purpose CLI tools for archiving and retrieving user data, some special purpose tools for administrative maintenance and for developer tools for debugging.
 

diff --git a/docs/content/reference.rst b/docs/content/reference.rst
@@ -5,7 +5,8 @@ Two publications, co-authored by Simon D. Smart, Tiago Quintino, Baudouin Raoult
 describe fdb architecture and have been presented at PASC'17 `A Scalable Object Store for Meteorological and Climate Data`_ and PASC'19 `A High-Performance Distributed Object-Store for Exascale Numerical Weather Prediction and Climate`_
 
 In the following the two BibTeX snippets:
-::
+
+.. code-block:: latex
 
    @inproceedings{10.1145/3093172.3093238,
       author    = {Smart, Simon D. and Quintino, Tiago and Raoult, Baudouin},
@@ -23,8 +24,6 @@ In the following the two BibTeX snippets:
       series    = {PASC ’17}
    }
 
-
-
    @inproceedings{10.1145/3324989.3325726,
       author    = {Smart, Simon D. and Quintino, Tiago and Raoult, Baudouin},
       title     = {A High-Performance Distributed Object-Store for Exascale Numerical Weather Prediction and Climate},
@@ -43,4 +42,4 @@ In the following the two BibTeX snippets:
 
 
 .. _A Scalable Object Store for Meteorological and Climate Data: https://dl.acm.org/doi/pdf/10.1145/3093172.3093238
-.. _A High-Performance Distributed Object-Store for Exascale Numerical Weather Prediction and Climate: https://dl.acm.org/doi/pdf/10.1145/3324989.3325726
+.. _A High-Performance Distributed Object-Store for Exascale Numerical Weather Prediction and Climate: https://dl.acm.org/doi/pdf/10.1145/3324989.3325726
diff --git a/docs/content/technical-introduction.rst b/docs/content/technical-introduction.rst
@@ -1,33 +1,28 @@
 .. _technical-introduction-label:
 
 Technical Introduction
-######################
+~~~~~~~~~~~~~~~~~~~~~~
 
 |Licence|
 
 FDB (Fields DataBase) is a domain-specific object store developed at ECMWF for storing, indexing and retrieving GRIB data. Each GRIB message is stored as a field and indexed trough semantic metadata (i.e. physical variables such as temperature, pressure, ...).
 A set of fields can be retrieved specifying a request using a specific language developed for accessing :doc:`mars` Archive
 
-FDB exposes a C++ API as well as CLI :doc:`tools`. 
+FDB exposes a C++ API as well as CLI :doc:`technical/api`. 
 
 .. toctree::
    :maxdepth: 1
 
    technical/requirements
    technical/build-and-installation
+   technical/check-installation
    technical/configuration
    technical/config
    technical/schema
-   technical/implementation
-   technical/usage-examples
    technical/api
-   technical/tools
-   technical/genindex
 
 .. |Licence| image:: https://img.shields.io/badge/License-Apache%202.0-blue.svg
    :target: https://github.com/ecmwf/fdb/blob/develop/LICENSE
    :alt: Apache Licence
 
 .. _mars: mars.html
-.. _tools: tools.html
-
diff --git a/docs/content/technical/Api/api_python.rst b/docs/content/technical/Api/api_python.rst
@@ -0,0 +1,9 @@
+.. index:: Reference; Python API
+   :name: Python-reference
+
+Work on adding documentation for the FDB Python API is in progress here.
+
+The Python Api relies on a seperate project, see https://github.com/ecmwf/pyfdb.
+
+Python API
+==========