[uss_qualifier] Run multiple uss_qualifier configurations (#151)

* Run multiple uss_qualifier configurations

* Address comments

* Address comments

* Remove 1 second offset

.github/workflows/ci.yml

@@ -54,15 +54,15 @@ jobs:
       run: |
         cd monitoring/mock_uss
         make test
-    - name: uss_qualifier tests - U-Space Test Suite
+    - name: uss_qualifier tests (F3411-22a configuration)
       run: |
         cd monitoring/uss_qualifier
         make test
       env:
         RID_VERSION: F3411-22a
-        CONFIG_NAME: configurations.dev.local_test
+        CONFIG_NAME: ""
         USS_QUALIFIER_STOP_FAST: true
-    - name: uss_qualifier tests - ASTM NetRID F3411-19 Test Suite
+    - name: uss_qualifier tests (F3411-19 configuration)
       run: |
         cd monitoring/uss_qualifier
         make test
@@ -82,7 +82,7 @@ jobs:
         path: |
           logs
           monitoring/mock_uss/output
-    - name: Save USS qualifier report as artifact
+    - name: Save USS qualifier reports as artifact
       if: always()
       uses: actions/upload-artifact@v3
       with:

build/dev/extract_json_field.py

@@ -3,18 +3,20 @@
 import json
 import sys
 
-s = ""
-try:
-    for line in sys.stdin:
-        s += line + "\n"
-except KeyboardInterrupt:
-    pass
 
-try:
-    obj = json.loads(s)
-except ValueError as e:
-    raise ValueError(str(e) + "\nvvvvv Unable to decode JSON: vvvvv\n" + s + "^^^^^ Unable to decode JSON: ^^^^^")
+with open(sys.argv[2], "r") as f:
+    try:
+        obj = json.load(f)
+    except ValueError as e:
+        raise ValueError(f"Unable to load JSON from {sys.argv[2]}: {e}")
+
 fields = sys.argv[1].split(".")
 for field in fields:
-    obj = obj[field]
+    if field == "*":
+        obj = obj[next(iter(obj.keys()))]
+    else:
+        try:
+            obj = obj[field]
+        except KeyError:
+            raise ValueError(f"Could not find field '{field}' in '{sys.argv[1]}' for {sys.argv[2]}; available keys: {list(obj.keys())}")
 print(obj)

monitoring/uss_qualifier/README.md

@@ -10,7 +10,7 @@ The `uss_qualifier` tool is a synchronous executable built into the `interuss/mo
 
 The primary input accepted by uss_qualifier is the "configuration" specified with the `--config` option.  This option should be a [reference to a configuration file](configurations/README.md) that the user has constructed or been provided to test the desired system for the desired characteristics.  If testing a standard local system (DSS + dummy auth + mock USSs), the user can specify an alternate configuration reference as a single argument to `run_locally.sh` (the default configuration is `configurations.dev.local_test`).
 
-When building a custom configuration file, consider starting from [`configurations.dev.self_contained_f3548`](configurations/dev/self_contained_f3548.yaml), as it contains all information necessary to run the test without the usage of sometimes-configuring `$ref`s and `allOf`s.  See [configurations documentation](configurations/README.md) for more information.
+When building a custom configuration file, consider starting from [`configurations.dev.f3548_self_contained`](configurations/dev/f3548_self_contained.yaml), as it contains all information necessary to run the test without the usage of sometimes-configuring `$ref`s and `allOf`s.  See [configurations documentation](configurations/README.md) for more information.
 
 ### Quick start
 

monitoring/uss_qualifier/configurations/README.md

@@ -8,7 +8,7 @@ To execute a test run with uss_qualifier, a uss_qualifier configuration must be
 
 When referring to a configuration, three methods may be used; see [`FileReference` documentation](../fileio.py) for more details.
 
-* **Package-based**: refer to a dictionary (*.json, *.yaml) file located in a subfolder of the `uss_qualifier` folder using the Python module style, omitting the extension of the file name.  For instance, `configurations.dev.local_test` would refer to [uss_qualifier/configurations/dev/local_test.json](dev/local_test.json).
+* **Package-based**: refer to a dictionary (*.json, *.yaml) file located in a subfolder of the `uss_qualifier` folder using the Python module style, omitting the extension of the file name.  For instance, `configurations.dev.f3548` would refer to [uss_qualifier/configurations/dev/f3548.yaml](dev/f3548.yaml).
 * **Local file**: when a configuration reference is prefixed with `file://`, it refers to a local file using the path syntax of the host operating system.
 * **Web file**: when a configuration reference is prefixed with `http://` or `https://`, it refers to a file accessible at the specified URL.
 

monitoring/uss_qualifier/configurations/dev/f3548.yaml

@@ -1,8 +1,9 @@
+$content_schema: monitoring/uss_qualifier/configurations/configuration/USSQualifierConfiguration.json
 v1:
   test_run:
     resources:
       resource_declarations:
-        "$ref": resources.yaml#/f3548_che
+        $ref: ./library/resources.yaml#/f3548_che
     action:
       test_suite:
         suite_type: suites.astm.utm.f3548_21
@@ -14,11 +15,13 @@ v1:
           dss: dss
   artifacts:
     tested_roles:
-      report_path: output/tested_requirements.html
+      report_path: output/tested_requirements_f3548.html
       roles:
         - name: Strategic Coordination role
           requirement_set: astm.f3548.v21.scd
           participants:
             - uss1
             - uss2
-    "$ref": artifacts.yaml#/relative
+    report:
+      report_path: output/report_f3548.json
+      redact_access_tokens: true

monitoring/uss_qualifier/configurations/dev/self_contained_f3548.yaml → monitoring/uss_qualifier/configurations/dev/f3548_self_contained.yaml

@@ -1,3 +1,6 @@
+# See the file below (in the `schemas` folder of this repo) for the schema this file's content follows
+$content_schema: monitoring/uss_qualifier/configurations/configuration/USSQualifierConfiguration.json
+
 # This configuration uses the v1 configuration schema
 v1:
   # This block defines how to perform a test run
@@ -81,12 +84,12 @@ v1:
   artifacts:
     report:
       # Path to main report output
-      report_path: output/report.json
+      report_path: output/report_f3548_self_contained.json
       redact_access_tokens: true
 
     # Information about a report including a list of roles tested and which participants are attempting to fill them
     tested_roles:
-      report_path: output/tested_requirements.html
+      report_path: output/tested_requirements_f3548_self_contained.html
       roles:
         - name: Strategic Coordination role
           requirement_set: astm.f3548.v21.scd
@@ -96,4 +99,4 @@ v1:
 
     graph:
       # Path to output GraphViz test file summarizing test run
-      gv_path: output/report.gv
+      gv_path: output/report_f3548_self_contained.gv

monitoring/uss_qualifier/configurations/dev/general_flight_auth.yaml

@@ -0,0 +1,16 @@
+$content_schema: monitoring/uss_qualifier/configurations/configuration/USSQualifierConfiguration.json
+v1:
+  test_run:
+    resources:
+      resource_declarations:
+        $ref: ./library/resources.yaml#/general_flight_authorization
+    action:
+      test_scenario:
+        scenario_type: scenarios.interuss.flight_authorization.GeneralFlightAuthorization
+        resources:
+          table: flight_check_table
+        on_failure: Abort
+  artifacts:
+    report:
+      report_path: output/report_general_flight_auth.json
+      redact_access_tokens: true

monitoring/uss_qualifier/configurations/dev/generate_rid_test_data.json

@@ -1,9 +1,10 @@
 {
+  "$content_schema": "monitoring/uss_qualifier/configurations/configuration/USSQualifierConfiguration.json",
   "v1": {
     "test_run": {
       "resources": {
         "resource_declarations": {
-          "$ref": "resources.yaml#/net_rid_sims"
+          "$ref": "./library/resources.yaml#/net_rid_sims"
         }
       },
       "action": {

monitoring/uss_qualifier/configurations/dev/geoawareness_cis.yaml

@@ -0,0 +1,19 @@
+$content_schema: monitoring/uss_qualifier/configurations/configuration/USSQualifierConfiguration.json
+v1:
+  test_run:
+    resources:
+      resource_declarations:
+        source_document:
+          resource_type: resources.eurocae.ed269.source_document.SourceDocument
+          specification:
+            url: file://./test_data/che/geoawareness/cis_source_sample.json
+    action:
+      test_suite:
+        suite_type: suites.uspace.geo_awareness_cis
+        resources:
+          source_document: source_document
+  artifacts:
+    report:
+      report_path: output/report_geoawareness_cis.json
+    graph:
+      gv_path: output/report_geoawareness_cis.gv

monitoring/uss_qualifier/configurations/dev/geospatial_comprehension.yaml

@@ -0,0 +1,16 @@
+$content_schema: monitoring/uss_qualifier/configurations/configuration/USSQualifierConfiguration.json
+v1:
+  test_run:
+    resources:
+      resource_declarations:
+        $ref: ./library/resources.yaml#/geospatial_map
+    action:
+      test_scenario:
+        scenario_type: scenarios.interuss.geospatial_map.GeospatialFeatureComprehension
+        resources:
+          table: feature_check_table
+        on_failure: Abort
+  artifacts:
+    report:
+      report_path: output/report_geospatial_comprehension.json
+      redact_access_tokens: true

monitoring/uss_qualifier/configurations/dev/library/README.md

@@ -0,0 +1,5 @@
+# dev library
+
+This folder contains configuration components intended to be included in other components or configurations.
+
+The field `$content_schema` is used to annotate intended content type, and matching JSON Schema may be found in the [`schemas` folder of this repo](../../../../../schemas).

monitoring/uss_qualifier/configurations/dev/environment.yaml → monitoring/uss_qualifier/configurations/dev/library/environment.yaml

@@ -2,12 +2,14 @@
 
 common:
     utm_auth:
+        $content_schema: monitoring/uss_qualifier/resources/definitions/ResourceDeclaration.json
         resource_type: resources.communications.AuthAdapterResource
         specification:
             environment_variable_containing_auth_spec: AUTH_SPEC
 
 net_rid:
     netrid_service_providers_v19:
+        $content_schema: monitoring/uss_qualifier/resources/definitions/ResourceDeclaration.json
         resource_type: resources.netrid.NetRIDServiceProviders
         dependencies:
             auth_adapter: utm_auth
@@ -16,6 +18,7 @@ net_rid:
                 - participant_id: uss1
                   injection_base_url: http://host.docker.internal:8071/ridsp/injection
     netrid_service_providers_v22a:
+        $content_schema: monitoring/uss_qualifier/resources/definitions/ResourceDeclaration.json
         resource_type: resources.netrid.NetRIDServiceProviders
         dependencies:
             auth_adapter: utm_auth
@@ -24,6 +27,7 @@ net_rid:
                 - participant_id: uss1
                   injection_base_url: http://host.docker.internal:8081/ridsp/injection
     netrid_observers_v19:
+        $content_schema: monitoring/uss_qualifier/resources/definitions/ResourceDeclaration.json
         resource_type: resources.netrid.NetRIDObserversResource
         dependencies:
             auth_adapter: utm_auth
@@ -32,6 +36,7 @@ net_rid:
                 - participant_id: uss2
                   observation_base_url: http://host.docker.internal:8073/riddp/observation
     netrid_observers_v22a:
+        $content_schema: monitoring/uss_qualifier/resources/definitions/ResourceDeclaration.json
         resource_type: resources.netrid.NetRIDObserversResource
         dependencies:
             auth_adapter: utm_auth
@@ -40,6 +45,7 @@ net_rid:
                 - participant_id: uss2
                   observation_base_url: http://host.docker.internal:8083/riddp/observation
     netrid_dss_instances_v19:
+        $content_schema: monitoring/uss_qualifier/resources/definitions/ResourceDeclaration.json
         resource_type: resources.astm.f3411.DSSInstancesResource
         dependencies:
             auth_adapter: utm_auth
@@ -52,6 +58,7 @@ net_rid:
                   rid_version: F3411-19
                   base_url: http://host.docker.internal:8082
     netrid_dss_instances_v22a:
+        $content_schema: monitoring/uss_qualifier/resources/definitions/ResourceDeclaration.json
         resource_type: resources.astm.f3411.DSSInstancesResource
         dependencies:
             auth_adapter: utm_auth
@@ -66,6 +73,7 @@ net_rid:
 
 f3548:
     flight_planners:
+        $content_schema: monitoring/uss_qualifier/resources/definitions/ResourceDeclaration.json
         resource_type: resources.flight_planning.FlightPlannersResource
         dependencies:
             auth_adapter: utm_auth
@@ -84,6 +92,7 @@ f3548:
                 -   participant_id: uss2
                     injection_base_url: http://host.docker.internal:8094/scdsc
     dss:
+        $content_schema: monitoring/uss_qualifier/resources/definitions/ResourceDeclaration.json
         resource_type: resources.astm.f3548.v21.DSSInstanceResource
         dependencies:
             auth_adapter: utm_auth
@@ -93,6 +102,7 @@ f3548:
 
 f3548_single_scenario:
     uss1:
+        $content_schema: monitoring/uss_qualifier/resources/definitions/ResourceDeclaration.json
         resource_type: resources.flight_planning.FlightPlannerResource
         dependencies:
             auth_adapter: utm_auth
@@ -101,6 +111,7 @@ f3548_single_scenario:
                 participant_id: uss1
                 injection_base_url: http://host.docker.internal:8074/scdsc
     uss2:
+        $content_schema: monitoring/uss_qualifier/resources/definitions/ResourceDeclaration.json
         resource_type: resources.flight_planning.FlightPlannerResource
         dependencies:
             auth_adapter: utm_auth