Basic implementation for ODM parsing

[matzipan]

These files are generated using a fork of xml-schema which is patched to generate quick-xml-compliant parsing. The xsd files are the following:

• OPM: https://sanaregistry.org/files/ndmxml_unqualified/ndmxml-3.0.0-opm-3.0.xsd
• OMM: https://sanaregistry.org/files/ndmxml_unqualified/ndmxml-3.0.0-omm-3.0.xsd
• OEM: https://sanaregistry.org/files/ndmxml_unqualified/ndmxml-3.0.0-oem-3.0.xsd
• OCM: https://sanaregistry.org/files/ndmxml_unqualified/ndmxml-3.0.0-ocm-3.0.xsd
• Common: https://sanaregistry.org/files/ndmxml_unqualified/ndmxml-3.0.0-common-3.0.xsd

• Commit the xsd-parser generated file and then do changes on top
• Fix missing Vec3Double
• Fix handling of simple type (xs:double) extensions in xsd-parser-rs
• Fix missing CovarianceMatrixType
• Generate structs for each xsd file with xml-schema-derive
• Fix custom serde for VecDouble.
• Implement choice workaround
• Rebase https://github.com/matzipan/xml-schema/tree/main
• Add license
• Implement OMM test
• Implement test
• Implement test
• Implement test
• Implement test
• Implement test
• Implement test
• Implement test
• Implement test
• Implement test
• Implement test
• Implement test
• Implement test
• Implement test
• Implement OEM test
• Add tests
• Implement OMM combined test
• Implement ODM test
• Add support for JSON OMM
• JSON tests https://celestrak.org/NORAD/elements/gp.php?GROUP=last-30-days&FORMAT=json
• Add support for KVN
• Parse KVN date
• Parse KVN integer
• Parse KVN float
• Parse KVN fixed point
• Implement a more systematic way for deserializing
• Implement the OPM KVN deserializer
• Implement deserializer for Option types
• Return error code when key doesn't exist, ie = BLA
• Implement deserializer for Vector types
• Return error code when the key doesn't match the expected one
• Implement deserializer for OPM
• Support _list suffix on list types
• Implement support for deserializing into unit structs
• Implement support for tuple struct
• Implement deserialzier using the OPM XML struct
• Transcribe all the todos
• Implement the OMM KVN deserializer
• Fix COMMENT line parsing
• Fix error type keywords
• Fix tests
• Fix regex unwraps
• Fix parsing of "CCSDS_OPM_VERS = 3.0" string
• Clean-up Cargo.toml
• lox_derive -> lox-derive
• Move to lox-io
• Fix clippy issues
• Fix unused with_unit
• Document limitations for the JSON (like the fact that only OMM seems to be used)
• Add KVN parser back
• Get clippy happy
• Document KVN parser
• Document limitations for the XML (like the fact that there's no data validation, version 3)
• Document the fact that KVN float parsing is more lax than the spec (ie. spec only wants 1 character in the integer part of a floating point number)
• Document the fact that string value parsing in KVN is more relaxed than the spec (Orekit test fixtures seem to suggest that some messages out there are not strictly compliant to the spec).
• Document limitation about the eager parsing of comments for KVN

After merge:

• Remove nom Remove nom from the KVN parser code #114
• Unify the deserializer generator, remove the _new suffix Unify the two proc macro chains #115
• Implement the OCM KVN deserializer Implement the OCM KVN parser #116
• Implement the OEM KVN deserializer Implement the OEM KVN parser #117
• Implement deserializer for UserDefinedType Implement deserializer for UserDefinedType and update docs #118
• Handle empty whitespace lines Handle empty whitespace lines in the KVN parser #119
• Add support for ddd day format Add support for ddd day format and update docs #120
• KVN tests: https://github.com/CS-SI/Orekit/tree/develop/src/test/resources/ccsds/odm Add KVN tests #121
• KVN tests: https://github.com/galactics/beyond/tree/master/tests/io/ccsds/data Add KVN tests #121
• Set the right value for with_unit Set the right value for with_unit for the KVN parser #122
• Implement missing message types from 505.0-B-3 and add the types back Implement missing ODM message types #123
• Cover error handling and error transformation code with unit tests Cover error handling and error transformation code with unit tests #124
• Clean-up proc macro, and use .expect instead of .unwrap (see) Replace the unwraps into expects into the kvn proc macro code #125

With xsd-parser-rs the following features are not working well:

• sequences inside groups, I have not checked how easy would be to implement
• Handling of attribute-based extensions of simple types
• the files needed quite a bit of changing to get them in a working state

With xml-schema-derive the following issues exist:

• choice types are not implemented yet (Implement choice media-io/xml-schema#36)
• extensions have weird base/extension fields instead of directly importing the fields. This is fine for the first phase, but the API ergonomics are not great and it also doesn't map well to how the JSON and KVN structs will work. So definetely something that might need to change in the future.
• enumerations are strings (minor)
• there is no validation (xsd-parser-rs had some basic validation), but this is not a blocker for the first phase, maybe a nice to have in the future
• groups were not supported, but I implemented them (Add extension group support media-io/xml-schema#33)

Overall I've had great progress with porting xml-schema-derive to generate quick-xml serde so I will keep using it for now although there's no clear winner between xml-schema-derive and xsd-parser-rs.

Closes #33

[matzipan]

In order to run xsd-parser-rs on the common xsd file, the following patch was necessary:

diff --git a/ndmxml-3.0.0-common-3.0.xsd b/ndmxml-3.0.0-common-3.0-patched.xsd
index 9417112..75afba9 100644
--- a/ndmxml-3.0.0-common-3.0.xsd
+++ b/ndmxml-3.0.0-common-3.0-patched.xsd
@@ -574,14 +574,14 @@
 </xsd:simpleType>
 
 
-<xsd:complexType name="rdmPositionType">
+<!-- <xsd:complexType name="rdmPositionType"> -->
 <!-- TO DSB: RESOLVE use=optional/use=required for position -->
-    <xsd:simpleContent>
+    <!-- <xsd:simpleContent>
         <xsd:restriction base="ndm:positionType">
             <xsd:attribute name="units" type="ndm:positionUnits" use="required" />
         </xsd:restriction>
     </xsd:simpleContent>
-</xsd:complexType>
+</xsd:complexType> -->
 
 
 <xsd:complexType name="velocityType">
@@ -593,14 +593,14 @@
 </xsd:complexType>
 
 
-<xsd:complexType name="rdmVelocityType">
+<!-- <xsd:complexType name="rdmVelocityType"> -->
 <!-- TO DSB: RESOLVE use=optional/use=required for velocity -->
-    <xsd:simpleContent>
+    <!-- <xsd:simpleContent>
         <xsd:restriction base="ndm:velocityType">
             <xsd:attribute name="units" type="ndm:velocityUnits" use="required" />
         </xsd:restriction>
     </xsd:simpleContent>
-</xsd:complexType>
+</xsd:complexType> -->
 
 
 <xsd:simpleType name="velocityUnits">
@@ -919,7 +919,7 @@
 <!--********************************************************************-->
 <!-- Start of ODM covariance type definitions                           -->
 <!--********************************************************************-->
-
+<!-- 
   <xsd:group name="covarianceMatrixElementsGroup">
     <xsd:sequence>
       <xsd:element name="CX_X" type="ndm:positionCovarianceType"/>
@@ -944,7 +944,7 @@
       <xsd:element name="CZ_DOT_Y_DOT" type="ndm:velocityCovarianceType"/>
       <xsd:element name="CZ_DOT_Z_DOT" type="ndm:velocityCovarianceType"/>
     </xsd:sequence>
-  </xsd:group>
+  </xsd:group> -->
 
    <xsd:complexType name="opmCovarianceMatrixAbstractType" abstract="true">
       <xsd:sequence>
@@ -960,7 +960,7 @@
          <xsd:element name="COV_REF_FRAME" type="xsd:string" minOccurs="0"/>
       </xsd:sequence>
    </xsd:complexType>
-
+<!-- 
    <xsd:complexType name="oemCovarianceMatrixType" mixed="false">
     <xsd:complexContent>
       <xsd:extension base="ndm:oemCovarianceMatrixAbstractType">
@@ -977,7 +977,7 @@
         <xsd:group ref="ndm:covarianceMatrixElementsGroup" />
       </xsd:extension>
     </xsd:complexContent>
-  </xsd:complexType>
+  </xsd:complexType> -->
 
   <xsd:complexType name="positionCovarianceType">
     <xsd:simpleContent>
@@ -1254,14 +1254,14 @@
 <!-- APM/AEM Quaternion Related                                         -->
 <!--********************************************************************-->
 
-<xsd:complexType name="quaternionType">
+<!-- <xsd:complexType name="quaternionType">
    <xsd:all>
       <xsd:element name="QC" type="ndm:quaternionComponentType"/>
       <xsd:element name="Q1" type="ndm:quaternionComponentType"/>
       <xsd:element name="Q2" type="ndm:quaternionComponentType"/>
       <xsd:element name="Q3" type="ndm:quaternionComponentType"/>
    </xsd:all>
-</xsd:complexType> 
+</xsd:complexType>  -->
 
 
 <xsd:simpleType name="quaternionComponentType">
@@ -1272,14 +1272,14 @@
 </xsd:simpleType>
 
 
-<xsd:complexType name="quaternionRateType">  
+<!-- <xsd:complexType name="quaternionRateType">  
    <xsd:all>
       <xsd:element name="QC_DOT" type="ndm:quaternionDotType"/>
       <xsd:element name="Q1_DOT" type="ndm:quaternionDotType"/>
       <xsd:element name="Q2_DOT" type="ndm:quaternionDotType"/>
       <xsd:element name="Q3_DOT" type="ndm:quaternionDotType"/>
    </xsd:all>
-</xsd:complexType>
+</xsd:complexType> -->
 
 
 <xsd:complexType name="quaternionDotType">

[matzipan]

With xml-schema-derive only the following patch was necessary in the common xsd file:

@@ -47,8 +47,8 @@
       <xsd:element name="header" type="ndm:odmHeader"/>
       <xsd:element name="body"   type="ndm:oemBody"/>       
    </xsd:sequence>
-   <xsd:attribute name="id"        use="required" fixed="CCSDS_OEM_VERS"/
>
-   <xsd:attribute name="version"   use="required" fixed="3.0"/>
+   <xsd:attribute name="id"      type="string" use="required" fixed="CCSD
S_OEM_VERS"/>
+   <xsd:attribute name="version" type="string"  use="required" fixed="3.0
"/>

But some patches are needed on top of the main branch of xml-schema-derive to make it work for us: media-io/xml-schema@main...matzipan:xml-schema:main

[codecov]

Codecov Report

Attention: Patch coverage is 96.18701% with 199 lines in your changes missing coverage. Please review.

Project coverage is 97.59%. Comparing base (152f2ec) to head (acaecdb).
Report is 2 commits behind head on main.

❗ Current head acaecdb differs from pull request most recent head 7ca1fb0

Please upload reports for the commit 7ca1fb0 to get more accurate results.

Files	Patch %	Lines
crates/lox-io/src/ndm/kvn/parser.rs	90.32%	78 Missing ⚠️
crates/lox-io/src/ndm/common.rs	46.77%	66 Missing ⚠️
crates/lox-io/src/ndm/xml/error.rs	20.00%	24 Missing ⚠️
crates/lox-io/src/ndm/omm.rs	98.68%	17 Missing ⚠️
crates/lox-io/src/ndm/ocm.rs	96.92%	14 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #37      +/-   ##
==========================================
- Coverage   97.82%   97.59%   -0.23%     
==========================================
  Files          59       70      +11     
  Lines       20732    26960    +6228     
==========================================
+ Hits        20281    26312    +6031     
- Misses        451      648     +197     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[matzipan]

This is my source material, from April 2023: https://public.ccsds.org/Pubs/502x0b3e1.pdf

Here's my understanding of how ODM units work.

7.7.1.1 says:

For documentation purposes and clarity only, units may be included as ASCII text after a value in the OPM and OMM. If units are displayed, they must exactly match the units (including lower/upper case) as specified in tables 3-3 and 4-3. If units are displayed, then:

a) there must be at least one blank character between the value and the units text;
b) the units must be enclosed within square brackets (e.g., ‘[km]’);
c) combinations of units shall adhere to requirements listed in 1.5.

In tables 3-3 and 4-3 each entry has at most one unit associated with it. So there's no entry which can have two different units.

This is also strengthened by the fact that it explains that the units should be included "for documentation purposes and clarity only". Which to me suggests that they do not affect the values, and are only for humans to be able to understand the data at a glance.

What makes it murky is the "combinations of units shall adhere to requirements listed in 1.5". Because this talks about units in SI. However to me it seems to refer only to combinations, which is only 1.5.1.2. 1.5.1.2 seems to only talk about combinations of units with other units and does not talk about combining with constants or something.

I can draw similar conclusions about 7.7.2 and 7.7.3.1, 7.7.3.4, 7.7.3.6.

Where it gets confusing again is 7.7.3.2 and 7.7.3.3, because it points to SANA Registry (reference [11]) which is:

“CCSDS Navigation Standards Normative Annexes.” Space Assigned Numbers Authority. https://sanaregistry.org/r/navigation_standard_normative_annexes

The link shows "content not found". So I am hoping that all of the previous conclusions will hold for this part, but I can't be 100% sure.