[uss_qualifier] Clean up NetRID docs

monitoring/uss_qualifier/requirements/interuss/automated_testing/rid/injection.md

@@ -2,16 +2,24 @@
 
 ## Overview
 
-TODO: Link to API YAML and provide overview
+In order to test remote ID Service Providers (USSs who ingest aircraft data from operators and provide it to relevant Display Providers), InterUSS requires any Service Provider under test to implement the [InterUSS remote ID automated testing interface](https://github.com/interuss/automated_testing_interfaces/tree/main/rid) (specifically, the [injection portion](https://github.com/interuss/automated_testing_interfaces/blob/main/rid/v1/injection.yaml)).  This interface empowers uss_qualifier, as the test director, to instruct the USS to create a flight in a particular area with particular characteristics.  This is analogous to a similar verbal instruction during a manual checkout (e.g., "USS X, start your flight in the yellow area").
 
 ## Requirements
 
-TODO: Describe requirements
+In general, to be successfully tested by uss_qualifier for remote ID functionality, Service Provider USSs are expected to implement the injection API mentioned above and successfully respond to valid requests.
 
 ### <tt>UpsertTestSuccess</tt>
 
+When a properly-authorized valid request to create/start/simulate a set of flights ("test"), the USS under test must execute this request (create/start/simulate the flights) and indicate success.
+
 ### <tt>UpsertTestResult</tt>
 
+The USS under test is allowed to modify most components of the requested flights including the telemetry and details responses, including the flight ID (though not the injection_id; that is the invariant ID by which uss_qualifier refers to an injected test flight).  However, there are obviously limits on how much the injected flight may be modified and still serve the purposes of the test being conducted.  For instance, a test whose purpose was to observe interactions between multiple providers' flights in the same area could not be accomplished if all but one USS created flights in a particular city in Europe while the last USS created their flight in the United States.  The actually-injected flight returned by the USS under test in response to an injection request must be suitable for the test to be conducted.  In practice, this means that USSs should modify the injection request as little as possible while still maintaining compatibility with their system.
+
 ### <tt>ExpectedBehavior</tt>
 
+The USS under test must treat injected flights as similarly to real flights as practical.  The same pathways that accept real telemetry should be used to accept the injected telemetry.  For instance, the standard pathway to accept real telemetry should generally not accept reports from the future, so if future telemetry of the injected flight is visible appreciably before the time it is supposed to have been virtually reported then the USS into which the flight was injected is not treating the injected flight the same way a normal flight would be treated.
+
 ### <tt>DeleteTestSuccess</tt>
+
+In order to rapidly conduct sequences of automated tests, uss_qualifier (as test director) must be able to "clear the airspace" after the completion of a test so that the simulated/injected flights from this test run do not affect future test runs.  A Service Provider USS must successfully cancel/land/remove/delete the flights in a specified test upon deletion request for that test and indicate success for the deletion request.

monitoring/uss_qualifier/requirements/interuss/automated_testing/rid/observation.md

@@ -2,12 +2,16 @@
 
 ## Overview
 
-TODO: Link to API YAML and provide overview
+In order to test remote ID Display Providers (USSs who obtain aircraft data from Service Providers in order to aggregate it for their end-user viewers), InterUSS requires any Display Provider under test to implement the [InterUSS remote ID automated testing interface](https://github.com/interuss/automated_testing_interfaces/tree/main/rid) (specifically, the [observation portion](https://github.com/interuss/automated_testing_interfaces/blob/main/rid/v1/observation.yaml)).  This interface empowers uss_qualifier, as the test director, to ask the USS what flights would be visible on its Display Application(s).  This is analogous to a similar verbal question during a manual checkout (e.g., "USS Y, what flights do you see in the yellow area?").
 
 ## Requirements
 
-TODO: Describe requirements
+In general, to be successfully tested by uss_qualifier for remote ID functionality, Display Provider USSs are expected to implement the observation API mentioned above and successfully respond to valid requests.
 
 ### <tt>ObservationSuccess</tt>
 
+Upon receipt of a properly-authorized valid request to report the flights visible in a particular area, the USS under test must respond with the content visible in that area.  Even if the user/viewer would be shown an error or other off-nominal situation in the Display Application, it should still be possible to return a valid response to this question (e.g., in the case of the user being shown an error, the USS would indicate no flights and no cluster visible).
+
 ### <tt>UniqueFlights</tt>
+
+While the identifiers for displayed flights are an implementation detail of the USS (it may use NetRID identifiers, or it may assign different identifiers), the user/viewer needs the capability to obtain details about a specific individual flight.  For that reason, a USS must provide unique identifiers for each flight in a particular observation.

monitoring/uss_qualifier/requirements/interuss/uss_qualifier/unit_test.md

@@ -1,8 +1,8 @@
 # USS Qualifier unit test requirements
 
-This file is used to defines requirements to support unit testing.
+This file is used to define requirements to support unit testing.
 
 ## Dummy requirements
 * <tt>REQ001</tt>
 * <tt>REQ100</tt>
-* <tt>REQ101</tt>
+* <tt>REQ101</tt>

monitoring/uss_qualifier/scenarios/astm/netrid/common/dss/subscription_validation.py

@@ -59,7 +59,13 @@ def run(self):
 
         self._setup_case()
 
-        self._subscription_limitations_case()
+        self.begin_test_case("Subscription duration limitations")
+        self._create_too_long_subscription()
+        self.end_test_case()
+
+        self.begin_test_case("Subscription quantity limitations")
+        self._create_too_many_subscriptions()
+        self.end_test_case()
 
         self.end_test_scenario()
 
@@ -70,46 +76,33 @@ def _setup_case(self):
 
         self.end_test_case()
 
-    def _clean_any_sub(self, check: PendingCheck):
-        fetched = self._dss_wrapper.search_subs(
-            check, self._isa.footprint.to_vertices()
-        )
+    def _clean_any_sub(self):
+        with self.check(
+            "Successful subscription query", [self._dss.participant_id]
+        ) as check:
+            fetched = self._dss_wrapper.search_subs(
+                check, self._isa.footprint.to_vertices()
+            )
         for sub_id in fetched.subscriptions.keys():
-            self._dss_wrapper.cleanup_sub(check, sub_id=sub_id)
+            with self.check(
+                "Successful subscription deletion", [self._dss.participant_id]
+            ) as check:
+                self._dss_wrapper.cleanup_sub(check, sub_id=sub_id)
 
     def _ensure_clean_workspace_step(self):
         self.begin_test_step("Ensure clean workspace")
 
-        with self.check(
-            "Successful subscription query and cleanup", [self._dss.participant_id]
-        ) as check:
-            self._clean_any_sub(check)
+        self._clean_any_sub()
 
         self.end_test_step()
 
-    def _subscription_limitations_case(self):
-        self.begin_test_case("Subscription limitations")
-
-        self.begin_test_step("Subscription duration limitations")
-
-        self._create_too_long_subscription()
-
-        self.end_test_step()
-
-        self.begin_test_step("Subscription quantity limitations")
-
-        self._create_too_many_subscriptions()
-
-        self.end_test_step()
-        self.end_test_case()
-
     def _create_too_many_subscriptions(self):
+        self.begin_test_step("Create maximum number of subscriptions")
 
         with self.check(
             "Create up to the maximum allowed number of subscriptions in an area",
             [self._dss.participant_id],
         ) as check:
-            self._clean_any_sub(check)
             # Create 10 subscriptions with different ID's
             for i in range(self._dss.rid_version.dss_max_subscriptions_per_area):
                 sub_id = f"{self._sub_id[:-3]}1{i:02d}"
@@ -119,6 +112,10 @@ def _create_too_many_subscriptions(self):
                     **self._default_subscription_params(datetime.timedelta(minutes=30)),
                 )
 
+        self.end_test_step()
+
+        self.begin_test_step("Exceed maximum number of subscriptions")
+
         with self.check(
             "Enforce maximum number of subscriptions for an area",
             [self._dss.participant_id],
@@ -133,12 +130,21 @@ def _create_too_many_subscriptions(self):
                 **self._default_subscription_params(datetime.timedelta(minutes=30)),
             )
 
+        self.end_test_step()
+
+        self.begin_test_step("Clean up subscriptions")
+
+        self._clean_any_sub()
+
+        self.end_test_step()
+
     def _create_too_long_subscription(self):
+        self.begin_test_step("Try to create too-long subscription")
+
         with self.check(
-            "Enforce maximum duration of subscriptions for an area",
+            "Too-long subscription creation rejected",
             [self._dss.participant_id],
         ) as check:
-            self._clean_any_sub(check)
             # Sub with this ID does not exist and too long: we expect either a failure, or
             # that any subscription that is effectively created to be truncated at 24 hours.
             creation_attempt = self._dss_wrapper.put_sub_expect_response_code(
@@ -156,16 +162,28 @@ def _create_too_long_subscription(self):
             if creation_attempt.success:
                 self._check_properly_truncated(check, creation_attempt)
 
+        self.end_test_step()
+
+        self.begin_test_step("Try to extend subscription")
+
+        with self.check(
+            "Valid subscription created",
+            [self._dss.participant_id],
+        ) as check:
             # Create a subscription that is fine
-            self._dss_wrapper.put_sub(
+            sub = self._dss_wrapper.put_sub(
                 check=check,
                 sub_id=self._sub_id,
                 **self._default_subscription_params(
                     datetime.timedelta(hours=23, minutes=59)
                 ),
             )
 
-            # Sub with this ID does exist, an we try to extend it beyond 24 hours:
+        with self.check(
+            "Subscription duration limited during update",
+            [self._dss.participant_id],
+        ) as check:
+            # Sub with this ID does exist, and we try to extend it beyond 24 hours:
             extended_subscription = self._dss_wrapper.put_sub_expect_response_code(
                 check=check,
                 sub_id=self._sub_id,
@@ -177,6 +195,20 @@ def _create_too_long_subscription(self):
             if extended_subscription.success:
                 self._check_properly_truncated(check, extended_subscription)
 
+        self.end_test_step()
+
+        self.begin_test_step("Remove subscription")
+
+        with self.check(
+            "Subscription deleted",
+            [self._dss.participant_id],
+        ) as check:
+            self._dss_wrapper.del_sub(
+                check=check, sub_id=self._sub_id, sub_version=sub.subscription.version
+            )
+
+        self.end_test_step()
+
     def _check_properly_truncated(
         self, check: PendingCheck, changed: ChangedSubscription
     ):
@@ -191,7 +223,7 @@ def _check_properly_truncated(
                 Severity.Medium,
                 f"{self._dss.participant_id} DSS instance has returned a non-properly truncated subscription "
                 f"(duration: {duration}) "
-                f"when the expecation was either to fail or to truncate at 24 hours.",
+                f"when the expectation was either to fail or to truncate at 24 hours.",
                 query_timestamps=[changed.query.request.timestamp],
             )
             # If a subscription was created, we want to delete it before continuing:
@@ -211,9 +243,6 @@ def _default_subscription_params(self, duration: datetime.timedelta) -> Dict:
     def cleanup(self):
         self.begin_cleanup()
 
-        with self.check(
-            "Successful subscription query and cleanup", [self._dss.participant_id]
-        ) as check:
-            self._clean_any_sub(check)
+        self._clean_any_sub()
 
         self.end_cleanup()

monitoring/uss_qualifier/scenarios/astm/netrid/v19/dss/subscription_validation.md

@@ -22,43 +22,89 @@ Perform basic operations on a single DSS instance to create subscriptions and ch
 
 ### Ensure clean workspace test step
 
-This step ensures that we remove any subscription that may already exist for the service area.
+This step ensures that we remove any subscription that may already exist for the service area.  First, the DSS is queried for any applicable existing subscriptions, and then any subscriptions found are deleted.
 
-#### Successful subscription query and cleanup check
+#### Successful subscription query check
 
-We expect to be allowed to query for existing subscriptions in order to clean them up
+If the query for subscriptions fails, the "GET Subscriptions" portion of **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)** was not met.
 
-## Subscription limitations test case
+#### Successful subscription deletion
 
-### Subscription quantity limitations test step
+If the deletion attempt fails, the "DELETE Subscription" portion of **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)** was not met.
 
-The test will attempt to create 10 identical subscriptions for the same area and expect this to succeed, then create an 11th one and expect it to fail.
+## Subscription quantity limitations test case
+
+### Create maximum number of subscriptions test step
+
+The test will attempt to create 10 identical subscriptions for the same area and expect this to succeed.
 
 #### Create up to the maximum allowed number of subscriptions in an area check
 
 As per **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)**, the DSS API is expected to allow us
 to create multiple subscriptions.
 
+### Exceed maximum number of subscriptions test step
+
+Now, create an 11th one and expect it to fail.
+
 #### Enforce maximum number of subscriptions for an area check
 
 If the DSS successfully creates an 11th Subscription in the same area instead of rejecting it,
 it will not have performed the Subscription count validation as defined in **[astm.f3411.v19.DSS0050](../../../../../requirements/astm/f3411/v19.md)**
 
-### Subscription duration limitations test step
+### Clean up subscriptions test step
 
-#### Enforce maximum duration of subscriptions for an area check
+Clean up any subscriptions created.
 
-**[astm.f3411.v19.DSS0060](../../../../../requirements/astm/f3411/v19.md)** any subscription to the DSS may not exceed NetDSSMaxSubscriptionDuration (24 hours).
+#### Successful subscription query check
+
+If the query for subscriptions fails, the "GET Subscriptions" portion of **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)** was not met.
+
+#### Successful subscription deletion check
+
+If the deletion attempt fails, the "DELETE Subscription" portion of **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)** was not met.
+
+## Subscription duration limitations test case
+
+### Try to create too-long subscription test step
 
 The test will attempt to create a subscription for 24 hours and 10 minutes, and expect this to fail with an HTTP 400 error.
 
-If the creation succeeds, the test expects that the effectively created subscription has been truncated to 24 hours, with a tolerance of minus 1 minute.
+#### Too-long subscription creation rejected check
+
+**[astm.f3411.v19.DSS0060](../../../../../requirements/astm/f3411/v19.md)** any subscription to the DSS may not exceed NetDSSMaxSubscriptionDuration (24 hours).
+
+If creation of the too-long-duration subscription succeeds, the test expects that the effectively created subscription has been truncated to 24 hours, with a tolerance of minus 1 minute.
+
+### Try to extend subscription test step
 
-It will also attempt to create a valid subscription of 23 hours an 59 minutes, and then increase its duration to 24 hours and 10 minutes,
+The test will attempt to create a valid subscription of 23 hours and 59 minutes, and then increase its duration to 24 hours and 10 minutes,
 expecting this update to fail.
 
-## Cleanup
+#### Valid subscription created check
+
+The ability to create a valid subscription is required in **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)**.
+
+#### Subscription duration limited during update check
+
+If the DSS allows a user to extend an existing, valid subscription to a duration exceeding that specified in **[astm.f3411.v19.DSS0060](../../../../../requirements/astm/f3411/v19.md)**, this check will fail.
+
+### Remove subscription test step
+
+To clean up after itself, the test deletes the subscription created in the previous step.
+
+#### Subscription deleted check
 
-### Successful subscription query and cleanup check
+The ability to delete an existing subscription is required in **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)**.
+
+## Cleanup
 
 The cleanup phase of this test scenario will remove any subscription that may have been created during the test and that intersects with the test ISA.
+
+### Successful subscription query check
+
+If the query for subscriptions fails, the "GET Subscriptions" portion of **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)** was not met.
+
+### Successful subscription deletion
+
+If the deletion attempt fails, the "DELETE Subscription" portion of **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)** was not met.

monitoring/uss_qualifier/scenarios/astm/netrid/v19/dss_interoperability.md

@@ -152,6 +152,8 @@ Qualitatively proves: ISA[*P*] modified with proper response, all Subscription[i
 
 #### ISA modification triggers subscription notification requests check
 
+TODO: Implement
+
 **[astm.f3411.v19.A2-6-1,3c](../../../../requirements/astm/f3411/v19.md)**
 
 ### S6 test step
@@ -274,6 +276,8 @@ Qualitatively proves: Expired Subscription removed from ID index on primary DSS
 
 #### 404 with proper response check
 
+TODO: Investigate expected behavior and "404 with proper response" check
+
 **[astm.f3411.v19.A2-6-1,3d](../../../../requirements/astm/f3411/v19.md)**
 
 ### S15 test step

monitoring/uss_qualifier/scenarios/astm/netrid/v19/nominal_behavior.md

@@ -41,6 +41,8 @@ This check will fail if the flight was not successfully injected.
 
 #### Valid flight check
 
+TODO: Validate injected flights, especially to make sure they contain the specified injection IDs
+
 Per **[interuss.automated_testing.rid.injection.UpsertTestResult](../../../../requirements/interuss/automated_testing/rid/injection.md)**, the NetRID Service Provider under test should only make valid modifications to the injected flights.  This includes:
 * A flight with the specified injection ID must be returned.
 

monitoring/uss_qualifier/scenarios/astm/netrid/v22a/__init__.py

@@ -1,3 +1,4 @@
 from .dss_interoperability import DSSInteroperability
 from .nominal_behavior import NominalBehavior
+from .misbehavior import Misbehavior
 from .aggregate_checks import AggregateChecks