[uss_qualifier] Add scenario for testing data validation by SUT for G…

…et operational_intent (#293) * Scenario for testing data validation by SUT for Get operational_intent * Removing an import * Fix format * Add interactions requirement file * Adding more description to the scenario * Scenario documentation fix * Improved requirements description * Per PR comments, fixed module path, file name, checks and description in the test scenario and test steps readme files * Per PR comments removing py files, and improving description * Per PR review, created a new requirement for no notifications * Adding empty sceanrio and fixing documentation links * Fix format * Fix the requirement reference in documentation * Documentation change per PR * Removed requirement reference from test step, as per PR review * Added more description to test steps * Fix per PR review * Removed time_range suffix for flight intents, as per PR comment * Remover time_range_A suffix from flight intent * Revert "Remover time_range_A suffix from flight intent" This reverts commit 4a3df76. * Removed time_range_A suffix from flight intents * Removing unreferenced file and description per PR review * fixing variable name per PR reveiw * Removed link for no notification pushed * Fixed removing link * Fix link * Removing check from a precondition step as per PR review
interuss · Nov 3, 2023 · 353b9cb · seungmanlee-nasa · Dec 8, 2023 · 353b9cb
1 parent bc20f07
commit 353b9cb
Show file tree

Hide file tree

Showing 11 changed files with 348 additions and 0 deletions.
diff --git a/monitoring/uss_qualifier/requirements/interuss/f3548/notification_requirements.md b/monitoring/uss_qualifier/requirements/interuss/f3548/notification_requirements.md
@@ -0,0 +1,9 @@
+# ASTM F3548-21 notifications requirement
+
+While ASTM F3548-21 does not specifically prohibit sending notifications for operational intent changes when those notifications were not specifically prescribed by a notification that an operational intent may be relevant to a subscription (response from the DSS), InterUSS expects that notifications will only be sent at these particular times.
+SCD0085 requirement states only a positive requirement (preconditions satisified => notification must be sent).
+It doesn't establish a negative requirement (preconditions not satisfied => no notifications may be sent).
+As these negative requirements are reasonable  and related to SCD0085, interUSS has added it as a requirement for testing.
+This can happen in following situations -
+* <tt>NoDssEntityNoNotification</tt> If a USS was unable to write an entity reference to the DSS, it should not erroneously notify that operational intent, to another USS subscribed in the area.
+* <tt>NoSubscriptionNoNotification</tt> If a USS wrote an entity reference to the DSS, and was notified of no other USS subscription, then it should not send a notification to any USS. So the USS  shall (OnlyPrescribedNotifications) only send operational intent notifications when prescribed by SCD0085.
diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/__init__.py b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/__init__.py
diff --git a/...qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.md b/...qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.md
@@ -0,0 +1,112 @@
+# Data Validation of GET operational intents by USS test scenario
+
+## Description
+This test checks that the USS being tested validates the operational intents received as response to its GET request from another USS.
+Control_uss which is a mock uss plans a nearby V-shaped operation, and provides the data that tested_uss GETs.
+tested_uss validates the GET response from control_uss and accordingly plan its operation.
+Notably the following requirements:
+
+- **[astm.f3548.v21.SCD0035](../../../../requirements/astm/f3548/v21.md)**
+
+## Resources
+### flight_intents
+FlightIntentsResource provides the two V-shaped flight intents.
+The convex hulls of the 2D footprints of the two flights intersect, but the polygons do not intersect.
+There is an overlap in time and altitude of the two flights.
+- flight_1
+- flight_2
+
+### control_uss
+MockUSSResource that will be used for planning flights, controlling data shared for validation testing, and gathering interuss interactions from mock_uss.
+
+### tested_uss
+FlightPlannerResource that will be used for the USS being tested for its data validation of operational intent.
+
+### dss
+DSSInstanceResource that provides access to a DSS instance where flight creation/sharing can be verified.
+
+## Setup test case
+### Check for flight planning readiness test step
+Both USSs are queried for their readiness to ensure this test can proceed.
+
+#### Flight planning USS not ready check
+If either USS does not respond appropriately to the endpoint queried to determine readiness, this check will fail and the USS will have failed to meet **[astm.f3548.v21.GEN0310](../../../../requirements/astm/f3548/v21.md)** as the USS does not support the InterUSS implementation of that requirement.
+
+### Area clearing test step
+Both USSs are requested to remove all flights from the area under test.
+
+#### Area cleared successfully check
+**[interuss.automated_testing.flight_planning.ClearArea](../../../../requirements/interuss/automated_testing/flight_planning.md)**
+
+## Successfully plan flight near an existing flight test case
+
+### [Control_uss plans flight 2 test step](../../../flight_planning/plan_flight_intent.md)
+Flight 2 should be successfully planned by the control USS.
+
+### [Validate flight 2 sharing test step](../validate_shared_operational_intent.md)
+Validate that flight 2 is planned
+
+### Precondition - check tested_uss has no subscription in flight 2 area test step
+In order to run this test scenario, we need tested_uss to trigger GET operational intent request to mock_uss.
+So we need to make sure that there is no subscription of tested_uss, that would trigger notification of flight 2 to tested_uss.
+No notification pushed by control_uss to tested_uss, will ensure that tested_uss will make a GET request to control_uss for flight 2 details,
+while planning a nearby flight.
+If a notification is sent to tested_uss, the precondition for running this scenario will not be satisfied.
+
+### [Tested_uss plans flight 1 test step](../../../flight_planning/plan_flight_intent.md)
+The test driver attempts to plan flight 1 via the tested USS. It checks if any conflicts with flight 2
+which is of equal priority and came first.
+
+### [Validate flight 1 sharing test step](../validate_shared_operational_intent.md)
+Validate flight 1 is planned.
+
+### [Validate flight2 GET interaction test step](test_steps/validate_get_operational_intent.md)
+Validate that tested_uss makes a GET request for obtaining details of flight 2 from control_uss.
+In a previous step (Precondition - check tested_uss has no subscription in flight 2 area), we ensured that no notification of flight 2 was sent to tested_uss.
+Hence, tested_uss will need to make a GET request to obtain flight 2 details.
+
+### [Validate flight1 Notification sent to Control_uss test step](test_steps/validate_notification_operational_intent.md)
+Tested_uss notifies flight 1 to Control_uss, due to its subscription through flight 2.
+
+### [Delete tested_uss flight test step](../../../flight_planning/delete_flight_intent.md)
+Teardown
+
+### [Delete control_uss flight test step](../../../flight_planning/delete_flight_intent.md)
+Teardown
+
+## Flight planning prevented due to invalid data sharing test case
+### [Control_uss plans flight 2, sharing invalid operational intent data test step](../../../flight_planning/plan_flight_intent.md)
+Flight 2 should be successfully planned by the control_uss.
+The control_uss, which is mock_uss is instructed to share invalid data with other USS, for negative test.
+
+### [Validate flight 2 shared operational intent with invalid data test step](test_steps/validate_sharing_operational_intent_but_with_invalid_interuss_data.md)
+Validate that flight 2 is shared with invalid data as a modified behavior is injected by uss_qualifier for a negative test.
+
+### Precondition - check tested_uss has no subscription in flight 2 area test step
+In order to run this test scenario, we need tested_uss to trigger GET operational intent request to mock_uss.
+So we need to make sure that there is no subscription of tested_uss, that would trigger notification of flight 2 to tested_uss.
+No notification pushed by control_uss to tested_uss, will ensure that tested_uss will make a GET request to control_uss for flight 2 details,
+while planning a nearby flight.
+If a notification is sent to tested_uss, the precondition for running this scenario will not be satisfied.
+
+### [Test_uss attempts to plan flight 1, expect failure test step](test_steps/plan_flight_intent_expect_failed.md)
+The test driver attempts to plan the flight 1 via the tested_uss. It checks if any conflicts with flight 2
+which is of equal priority and came first.
+
+### [Validate flight 1 not shared by tested_uss test step](../validate_not_shared_operational_intent.md)
+Validate flight 1 is not shared with DSS, as plan failed.
+
+### [Validate flight 2 GET interaction test step](test_steps/validate_get_operational_intent.md)
+Validate that tested_uss makes a GET request for obtaining details of flight 2 from control_uss.
+In a previous step (Precondition - check tested_uss has no subscription in flight 2 area), we ensured that no notification of flight 2 was sent to tested_uss.
+Hence, tested_uss will need to make a GET request to obtain flight 2 details.
+
+### [Validate flight 1 Notification not sent to Control_uss test step](test_steps/validate_no_notification_operational_intent.md)
+
+### [Delete Control_uss flight test step](../../../flight_planning/delete_flight_intent.md)
+Teardown
+
+## Cleanup
+### Successful flight deletion check
+This cleanup is for both - after testcase ends and after test scenario ends
+**[interuss.automated_testing.flight_planning.DeleteFlightSuccess](../../../../requirements/interuss/automated_testing/flight_planning.md)**
diff --git a/...oring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.py b/...oring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.py
@@ -0,0 +1,52 @@
+from typing import Optional
+
+from monitoring.uss_qualifier.resources.astm.f3548.v21 import DSSInstanceResource
+from monitoring.uss_qualifier.resources.astm.f3548.v21.dss import DSSInstance
+from monitoring.uss_qualifier.resources.flight_planning import (
+    FlightIntentsResource,
+)
+from monitoring.uss_qualifier.resources.flight_planning.flight_intent import (
+    FlightIntent,
+)
+from monitoring.uss_qualifier.resources.flight_planning.flight_planner import (
+    FlightPlanner,
+)
+from monitoring.uss_qualifier.resources.flight_planning.flight_planners import (
+    FlightPlannerResource,
+)
+
+from monitoring.uss_qualifier.scenarios.scenario import (
+    TestScenario,
+)
+
+from monitoring.uss_qualifier.resources.interuss.mock_uss.client import (
+    MockUSSClient,
+    MockUSSResource,
+)
+
+
+class GetOpResponseDataValidationByUSS(TestScenario):
+    flight_1: FlightIntent
+
+    flight_2: FlightIntent
+
+    tested_uss: FlightPlanner
+    control_uss: MockUSSClient
+    dss: DSSInstance
+
+    def __init__(
+        self,
+        tested_uss: FlightPlannerResource,
+        control_uss: MockUSSResource,
+        dss: DSSInstanceResource,
+        flight_intents: Optional[FlightIntentsResource] = None,
+    ):
+        super().__init__()
+        self.tested_uss = tested_uss.flight_planner
+        self.control_uss = control_uss.mock_uss
+        self.dss = dss.dss
+
+    def run(self):
+        self.begin_test_scenario()
+        pass
+        self.end_test_scenario()
diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/__init__.py b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/__init__.py
diff --git a/...stm/utm/data_exchange_validation/test_steps/plan_flight_intent_expect_failed.md b/...stm/utm/data_exchange_validation/test_steps/plan_flight_intent_expect_failed.md
@@ -0,0 +1,14 @@
+# Plan flight Expect Failed test step
+
+This page describes the content of a common test case where a valid user flight intent fails in a flight planner, because of invalid data shared for a nearby flight shared by another USS.  See `plan_flight_intent_expect_failed` in invalid_op_test_steps.py.
+
+## Plan should fail check
+
+A USS shouldn't go ahead and plan if it doesn't have accurate information.
+As per SCD0035 a USS needs to verify a particular conflict status.
+If flight intent data shared for a nearby flight shared is invalid, the USS can't successfully perform that verification.
+It couldn't have obtained valid information about the other operational intents it's supposed to verify that conflict status against.
+Therefore, the USS should fail an attempt to plan. If the plan succeeds, we know they've violated SCD0035 because they failed to verify that particular conflict status.
+
+**[astm.f3548.v21.SCD0035](../../../../../requirements/astm/f3548/v21.md)**
+
diff --git a/...astm/utm/data_exchange_validation/test_steps/validate_get_operational_intent.md b/...astm/utm/data_exchange_validation/test_steps/validate_get_operational_intent.md
@@ -0,0 +1,10 @@
+# Validate GET interaction test step
+
+This step verifies that a USS makes a GET request to get the intent_details of an existing operation when needed as per ASTM F3548-21 by checking the interuss interactions of mock uss
+
+## MockUSS interactions request check
+**[interuss.mock_uss.hosted_instance.ExposeInterface](../../../../../requirements/interuss/mock_uss/hosted_instance.md)**.
+
+## Expect GET request check
+**[astm.f3548.v21.SCD0035](../../../../../requirements/astm/f3548/v21.md)**
+SCD0035 needs a USS to verify before transitioning to Accepted that it does not conflict with a type of operational intent, and the only way to have verified this is by knowing all operational intent details, and (from previous checks of no notifications) the only way to know the operational intent details of flight is to have requested them via a GET details interaction.
diff --git a/...a_exchange_validation/test_steps/validate_no_notification_operational_intent.md b/...a_exchange_validation/test_steps/validate_no_notification_operational_intent.md
@@ -0,0 +1,10 @@
+# Validate no notification test step
+
+This step verifies when a flight is not created, it is also not notified by checking the interuss interactions of mock_uss instance.
+
+## MockUSS interactions request check
+**[interuss.mock_uss.hosted_instance.ExposeInterface](../../../../../requirements/interuss/mock_uss/hosted_instance.md)**.
+
+## Expect Notification not sent check
+
+**[interuss.f3548.notification_requirements.NoDssEntityNoNotification](../../../../../requirements/interuss/f3548/notification_requirements.md)**
diff --git a/...data_exchange_validation/test_steps/validate_notification_operational_intent.md b/...data_exchange_validation/test_steps/validate_notification_operational_intent.md
@@ -0,0 +1,12 @@
+# Validate notification test step
+
+This step verifies that, when creating or modifying an operational intent, a USS sent the required notification for a relevant subscription owned by a mock_uss instance by checking the interactions of that mock_uss instance.
+
+## MockUSS interactions request check
+**[interuss.mock_uss.hosted_instance.ExposeInterface](../../../../../requirements/interuss/mock_uss/hosted_instance.md)**.
+
+## Expect Notification sent check
+**[astm.f3548.v21.SCD0085](../../../../../requirements/astm/f3548/v21.md)**
+
+## Notification data is valid check
+**[astm.f3548.v21.SCD0085](../../../../../requirements/astm/f3548/v21.md)**
diff --git a/...est_steps/validate_sharing_operational_intent_but_with_invalid_interuss_data.md b/...est_steps/validate_sharing_operational_intent_but_with_invalid_interuss_data.md
@@ -0,0 +1,23 @@
+# Validate flight sharing invalid data test step
+
+This step verifies that a created flight is shared properly per ASTM F3548-21 by querying the DSS for flights in the area of the flight intent, and then retrieving the details from the USS if the operational intent reference is found.  See `expect_shared_with_invalid_data` in invalid_op_test_steps.py.
+
+## DSS responses check
+
+**[astm.f3548.v21.DSS0005](../../../../../requirements/astm/f3548/v21.md)**
+
+## Operational intent shared with DSS check
+
+If a reference to the operational intent for the flight is not found in the DSS, this check will fail per **[astm.f3548.v21.USS0005](../../../../../requirements/astm/f3548/v21.md)** and **[astm.f3548.v21.OPIN0025](../../../../../requirements/astm/f3548/v21.md)**.
+
+## Operational intent details retrievable check
+
+If the operational intent details for the flight cannot be retrieved from the USS, this check will fail per **[astm.f3548.v21.USS0105](../../../../../requirements/astm/f3548/v21.md)** and **[astm.f3548.v21.OPIN0025](../../../../../requirements/astm/f3548/v21.md)**.
+
+## Invalid data in Operational intent details shared by Mock USS for negative test check
+
+Mock USS shares operational intent details response for the negative test case as per [the GetOperationalIntentDetailsResponse schema of the OpenAPI specification](https://github.com/astm-utm/Protocol/blob/v1.0.0/utm.yaml#L1120).
+If the operational intent details from mock_uss is valid, this check will fail. It would mean mock_uss is not behaving correctly.
+**[interuss.mock_uss.hosted_instance.ExposeInterface](../../../../../requirements/interuss/mock_uss/hosted_instance.md)**.
+
+
diff --git a/monitoring/uss_qualifier/test_data/usa/kentland/flight_intents/non_conflicting.yaml b/monitoring/uss_qualifier/test_data/usa/kentland/flight_intents/non_conflicting.yaml
@@ -0,0 +1,106 @@
+intents:
+  flight_1:
+    full:
+      basic_information:
+        usage_state: Planned
+        uas_state: Nominal
+        area:
+          - outline_polygon:
+              vertices:
+                - lat: 37.20642344604623
+                  lng: -80.58508994131496
+                - lat: 37.21359527809636
+                  lng: -80.5767365824006
+                - lat: 37.215812746083586
+                  lng: -80.58144645497978
+                - lat: 37.2146332499489
+                  lng: -80.58408279875103
+            altitude_lower:
+              value: 474
+              reference: W84
+              units: M
+            altitude_upper:
+              value: 560
+              reference: W84
+              units: M
+            start_time:
+              offset_from:
+                starting_from:
+                  start_of_test: { }
+                offset: -1s
+            duration: 5m
+
+      astm_f3548_21:
+        priority: 0
+
+      uspace_flight_authorisation:
+        uas_serial_number: 1AF49UL5CC5J6K
+        operation_category: Open
+        operation_mode: Vlos
+        uas_class: C0
+        identification_technologies:
+          - ASTMNetRID
+        connectivity_methods:
+          - cellular
+        endurance_minutes: 30
+        emergency_procedure_url: https://example.interussplatform.org/emergency
+        operator_id: CHEo5kut30e0mt01-qwe
+        uas_id: ''
+        uas_type_certificate: ''
+
+
+  flight_2:
+    full:
+      basic_information:
+        usage_state: Planned
+        uas_state: Nominal
+        area:
+          - outline_polygon:
+              vertices:
+                - lat: 37.214119359044275
+                  lng: -80.58443600701524
+                - lat: 37.212388260776436
+                  lng: -80.58824885811198
+                - lat: 37.20211436858821
+                  lng: -80.5856832012991
+                - lat: 37.21394908884487
+                  lng: -80.57474352572189
+                - lat: 37.206173006943104
+                  lng: -80.5852199577081
+            altitude_lower:
+              value: 483
+              reference: W84
+              units: M
+            altitude_upper:
+              value: 519
+              reference: W84
+              units: M
+            start_time:
+              offset_from:
+                starting_from:
+                  start_of_test: { }
+                offset: -1s
+            duration: 5m
+
+      astm_f3548_21:
+        priority: 0        # TODO: Remove flight_authorisation section when it is optional
+
+      uspace_flight_authorisation:
+        uas_serial_number: 1AF49UL5CC5J6K
+        operation_category: Open
+        operation_mode: Vlos
+        uas_class: C0
+        identification_technologies:
+          - ASTMNetRID
+        connectivity_methods:
+          - cellular
+        endurance_minutes: 30
+        emergency_procedure_url: https://example.interussplatform.org/emergency
+        operator_id: CHEo5kut30e0mt01-qwe
+        uas_id: ''
+        uas_type_certificate: ''
+
+
+
+
+