[uss_qualifier/scenarios/utm] Fix #178 (#214)

* [uss_qualifier/scenarios/utm] Fix #178

* add req for op intent coverage; update according to feedback; optional severity

monitoring/uss_qualifier/requirements/interuss/automated_testing/flight_planning.md

@@ -12,4 +12,9 @@ TODO: Describe requirements
 
 ### <tt>ExpectedBehavior</tt>
 
+### <tt>FlightCoveredByOperationalIntent</tt>
+For InterUSS to effectively test the requirements of ASTM F3548-21, a USS under test must act as if there is a
+regulatory requirement requiring all flights it manages to provide operational intents according to ASTM F3548-21 at all
+times for all flights it manages.
+
 ### <tt>DeleteFlightSuccess</tt>

monitoring/uss_qualifier/scenarios/astm/utm/nominal_planning/conflict_higher_priority/conflict_higher_priority.md

@@ -174,13 +174,30 @@ The test driver activates flight 2, which should be done successfully given that
 ### [Validate flight 2 sharing test step](../../validate_shared_operational_intent.md)
 
 ### [Modify activated flight 1 in conflict with activated flight 2 test step](../../../../flight_planning/modify_activated_flight_intent.md)
-Before execution of this step, flights 1 and 2 are activated and in conflict.
-The test driver modifies flight 1 in a way that still conflicts with flight 2.
-Even though flight 2 is the highest-priority flight, because the conflict existed before the modification was initiated,
-the modification is accepted per **[astm.f3548.v21.SCD0030](../../../../../requirements/astm/f3548/v21.md)**.
+Before execution of this step, flights 1 and 2 are activated and in conflict. Flight 2 is the highest-priority flight.
+The test driver attempts to modify flight 1 in a way that still conflicts with flight 2.
+
+The successful outcomes of the modification attempts:
+1. Even though flight 2 is the highest-priority flight, because the conflict existed before the modification was
+   initiated, an accepted modification is considered a success per **[astm.f3548.v21.SCD0030](../../../../../requirements/astm/f3548/v21.md)**.
+2. Due to the conflict, the USS may decide to be more conservative and to not support the modification. This is
+   considered a success as there is no positive requirement for the USS to accept the modification.
+
+A rejected modification will indicate a low severity failure. Indeed, in some situations a rejection may not be strictly
+speaking a failure to meet a requirement. This could be the case for example if the USS does not support directly update
+of intents and instead delete the previous one and create a new one. Since we cannot distinguish between an actual
+failure to meet the requirement and a reasonable behavior due to implementation limitations, we indicate a low severity
+failure which won't actually fail the test.
+
+In any case, whatever is the outcome of this step, there should not be any impact on the rest of the execution of the
+scenario. An intent should exist (this is checked in the next step) and it should be either the previous or the modified
+intent, both of which make no difference in the next steps.
 
 ### [Validate flight 1 sharing test step](../../validate_shared_operational_intent.md)
-The first flight should have been modified.
+If the modification was accepted, flight 1 should have been modified.
+If the modification was not supported, flight 1 should not have been modified.
+If the modification was rejected, flight 1 should not have been modified and should still exist. If it does not exist,
+it means that there is an active flight without an operational intent, which is a failure to meet **[interuss.automated_testing.flight_planning.FlightCoveredByOperationalIntent](../../../../../requirements/interuss/automated_testing/flight_planning.md)**.
 
 
 ## Attempt to modify activated flight in conflict test case

monitoring/uss_qualifier/scenarios/astm/utm/nominal_planning/conflict_higher_priority/conflict_higher_priority.py

@@ -7,6 +7,9 @@
 from monitoring.monitorlib.geotemporal import Volume4DCollection
 from monitoring.uss_qualifier.common_data_definitions import Severity
 from uas_standards.astm.f3548.v21.api import OperationalIntentState
+from uas_standards.interuss.automated_testing.scd.v1.api import (
+    InjectFlightResponseResult,
+)
 
 from monitoring.uss_qualifier.resources.astm.f3548.v21 import DSSInstanceResource
 from monitoring.uss_qualifier.resources.astm.f3548.v21.dss import DSSInstance
@@ -439,16 +442,23 @@ def _modify_activated_flight_conflict_preexisting(
             self._intents_extent,
             flight_1_oi_ref,
         ) as validator:
-            modify_activated_flight_intent(
+            resp = modify_activated_flight_intent(
                 self,
                 "Modify activated flight 1 in conflict with activated flight 2",
                 self.tested_uss,
                 self.flight_1_activated_time_range_A_extended.request,
                 self.flight_1_id,
+                preexisting_conflict=True,
             )
-            flight_1_oi_ref = validator.expect_shared(
-                self.flight_1_activated_time_range_A_extended.request
-            )
+
+            if resp.result == InjectFlightResponseResult.ReadyToFly:
+                flight_1_oi_ref = validator.expect_shared(
+                    self.flight_1_activated_time_range_A_extended.request
+                )
+            else:
+                flight_1_oi_ref = validator.expect_shared(
+                    self.flight_1_activated_time_range_A.request
+                )
 
         return flight_1_oi_ref, flight_2_oi_ref
 

monitoring/uss_qualifier/scenarios/astm/utm/test_steps.py

@@ -168,17 +168,36 @@ def expect_shared(
                 oi_ref = self._new_oi_ref
 
             elif self._new_oi_ref is None:
-                # we expect the original op intent to have been either modified or left untouched, thus must be among the returned op intents
-                # exception made if skip_if_not_found=True and op intent was deleted: step is skipped
+                # We expect the original op intent to have been either modified or left untouched, thus must be among
+                # the returned op intents. If additionally the op intent corresponds to an active flight, we fail a
+                # different appropriate check. Exception made if skip_if_not_found=True and op intent was deleted: step
+                # is skipped.
                 modified_oi_ref = self._find_after_oi(self._orig_oi_ref.id)
                 if modified_oi_ref is None:
                     if not skip_if_not_found:
-                        check.record_failed(
-                            summary="Operational intent reference not found in DSS",
-                            severity=Severity.High,
-                            details=f"USS {self._flight_planner.participant_id} was supposed to have shared with the DSS an updated operational intent by modifying it, but no matching operational intent references were found in the DSS in the area of the flight intent",
-                            query_timestamps=[self._after_query.request.timestamp],
-                        )
+                        if (
+                            flight_intent.operational_intent.state
+                            == OperationalIntentState.Activated
+                        ):
+                            with self._scenario.check(
+                                "Operational intent for active flight not deleted",
+                                [self._flight_planner.participant_id],
+                            ) as active_flight_check:
+                                active_flight_check.record_failed(
+                                    summary="Operational intent reference for active flight not found in DSS",
+                                    severity=Severity.High,
+                                    details=f"USS {self._flight_planner.participant_id} was supposed to have shared with the DSS an updated operational intent by modifying it, but no matching operational intent references were found in the DSS in the area of the flight intent",
+                                    query_timestamps=[
+                                        self._after_query.request.timestamp
+                                    ],
+                                )
+                        else:
+                            check.record_failed(
+                                summary="Operational intent reference not found in DSS",
+                                severity=Severity.High,
+                                details=f"USS {self._flight_planner.participant_id} was supposed to have shared with the DSS an updated operational intent by modifying it, but no matching operational intent references were found in the DSS in the area of the flight intent",
+                                query_timestamps=[self._after_query.request.timestamp],
+                            )
                     else:
                         self._scenario.record_note(
                             self._flight_planner.participant_id,

monitoring/uss_qualifier/scenarios/astm/utm/validate_shared_operational_intent.md

@@ -10,6 +10,12 @@ This step verifies that a created flight is shared properly per ASTM F3548-21 by
 
 If a reference to the operational intent for the flight is not found in the DSS, this check will fail per **astm.f3548.v21.USS0005** and **astm.f3548.v21.OPIN0025**.
 
+## Operational intent for active flight not deleted check
+
+If an activated operational intent is expected to exist after it has been modified or activated and that it is not found
+in the DSS, this means that there is an active flight without a corresponding operational intent, then this check will
+fail per **[interuss.automated_testing.flight_planning.FlightCoveredByOperationalIntent](../../../requirements/interuss/automated_testing/flight_planning.md)**.
+
 ## Operational intent details retrievable check
 
 If the operational intent details for the flight cannot be retrieved from the USS, this check will fail per **astm.f3548.v21.USS0105** and **astm.f3548.v21.OPIN0025**.

monitoring/uss_qualifier/scenarios/flight_planning/modify_activated_flight_intent.md

@@ -1,15 +1,41 @@
 # Modify activated flight test step
 
-This page describes the content of a common test case where a valid user flight intent in activated state should be
-successfully modified by a flight planner.  See `modify_activated_flight_intent` in [test_steps.py](test_steps.py).
+This page describes the content of a common test case where a valid user flight intent in activated state is tentatively
+modified by a flight planned. Multiple outcomes may be valid.
+See `modify_activated_flight_intent` in [test_steps.py](test_steps.py).
 
 ## Successful modification check
 
-All flight intent data provided is correct and valid and free of conflict in space and time, therefore it should have
-been modified by the USS per **interuss.automated_testing.flight_planning.ExpectedBehavior**.
+All flight intent data provided is correct and valid. The (already activated) provided flight intent may be in conflict
+with another activated flight, but only if this conflict already existed before the modification was initiated.
+
+If the provided flight intent is not in conflict with another intent the USS should have successfully modified the
+flight per **[astm.f3548.v21.SCD0030](../../requirements/astm/f3548/v21.md)**.
 If the USS fails to modify the flight, wrongly indicates a conflict, or wrongly indicates the activated state of the
 flight, this check will fail.
 
+If the provided flight intent is in conflict with another intent and that a pre-existing conflict was present, the USS
+may have decided to be more conservative and to not support modification.
+In such case, the USS may indicate that the operation is not supported instead of modifying the flight per **[astm.f3548.v21.SCD0030](../../requirements/astm/f3548/v21.md)**.
+If the USS fails to modify the flight, or fails to indicate that the modification is not supported, or wrongly indicates
+the activated state of the flight, this check will fail.
+
+Do take note that if the USS rejects the modification when a pre-existing conflict was present, this check will not fail,
+but the following *Rejected modification check* will. Refer to this check for more information.
+
+## Rejected modification check
+
+If the provided flight intent is in conflict with another intent and that a pre-existing conflict was present, the USS
+may have rejected the modification instead of modifying it or indicating that the modification is not supported. This
+could be the case for example if the USS does not support directly update of intents and instead delete the previous one
+and create a new one. This may or may not be strictly speaking a failure to meet a requirement, but we cannot
+distinguish between an actual failure to meet the requirement and a reasonable behavior due to implementation
+limitations.
+
+As such, if the pre-existing conflict was present, and that the USS rejected the modification, this check will fail with
+a low severity per **[astm.f3548.v21.SCD0030](../../requirements/astm/f3548/v21.md)**. This won't actually fail the test
+but will serve as a warning.
+
 ## Failure check
 
 All flight intent data provided was complete and correct. It should have been processed successfully, allowing the USS

monitoring/uss_qualifier/scenarios/flight_planning/test_steps.py

@@ -1,5 +1,5 @@
 import inspect
-from typing import List, Optional, Tuple, Iterable, Set, Dict
+from typing import List, Optional, Tuple, Iterable, Set, Dict, Union
 
 from monitoring.monitorlib.geotemporal import Volume4DCollection
 from uas_standards.astm.f3548.v21.api import OperationalIntentState
@@ -182,8 +182,10 @@ def modify_activated_flight_intent(
     flight_planner: FlightPlanner,
     flight_intent: InjectFlightRequest,
     flight_id: str,
+    preexisting_conflict: bool = False,
 ) -> InjectFlightResponse:
-    """Modify an activated flight intent that should result in success.
+    """Attempt to modify an activated flight intent.
+    If present, a pre-existing conflict must be indicated with `preexisting_conflict=True`.
 
     This function implements the test step described in
     modify_activated_flight_intent.md.
@@ -194,12 +196,35 @@ def modify_activated_flight_intent(
         flight_intent, OperationalIntentState.Activated, scenario, test_step
     )
 
+    if preexisting_conflict:
+        expected_results = {
+            InjectFlightResponseResult.ReadyToFly,
+            InjectFlightResponseResult.NotSupported,
+            # the following two results are considered expected in order to fail another check as low severity
+            InjectFlightResponseResult.Rejected,
+            InjectFlightResponseResult.ConflictWithFlight,
+        }
+        failed_checks = {
+            InjectFlightResponseResult.Failed: "Failure",
+            InjectFlightResponseResult.Rejected: (
+                "Rejected modification",
+                Severity.Low,
+            ),
+            InjectFlightResponseResult.ConflictWithFlight: (
+                "Rejected modification",
+                Severity.Low,
+            ),
+        }
+    else:
+        expected_results = {InjectFlightResponseResult.ReadyToFly}
+        failed_checks = {InjectFlightResponseResult.Failed: "Failure"}
+
     return submit_flight_intent(
         scenario,
         test_step,
         "Successful modification",
-        {InjectFlightResponseResult.ReadyToFly},
-        {InjectFlightResponseResult.Failed: "Failure"},
+        expected_results,
+        failed_checks,
         flight_planner,
         flight_intent,
         flight_id,
@@ -211,13 +236,14 @@ def submit_flight_intent(
     test_step: str,
     success_check: str,
     expected_results: Set[InjectFlightResponseResult],
-    failed_checks: Dict[InjectFlightResponseResult, str],
+    failed_checks: Dict[InjectFlightResponseResult, Union[str, Tuple[str, Severity]]],
     flight_planner: FlightPlanner,
     flight_intent: InjectFlightRequest,
     flight_id: Optional[str] = None,
 ) -> Tuple[InjectFlightResponse, Optional[str]]:
     """Submit a flight intent with an expected result.
-    A check fail is considered of high severity and as such will raise an ScenarioCannotContinueError.
+    A check fail is considered by default of high severity and as such will raise an ScenarioCannotContinueError.
+    The severity of each failed check may be overridden if needed.
 
     This function does not directly implement a test step.
 
@@ -244,13 +270,19 @@ def submit_flight_intent(
         notes_suffix = f': "{resp.notes}"' if "notes" in resp and resp.notes else ""
 
         for unexpected_result, failed_test_check in failed_checks.items():
+            if isinstance(failed_test_check, str):
+                check_name = failed_test_check
+                check_severity = Severity.High
+            else:
+                check_name, check_severity = failed_test_check
+
             with scenario.check(
-                failed_test_check, [flight_planner.participant_id]
+                check_name, [flight_planner.participant_id]
             ) as specific_failed_check:
                 if resp.result == unexpected_result:
                     specific_failed_check.record_failed(
                         summary=f"Flight unexpectedly {resp.result}",
-                        severity=Severity.High,
+                        severity=check_severity,
                         details=f'{flight_planner.participant_id} indicated {resp.result} rather than the expected {" or ".join(expected_results)}{notes_suffix}',
                         query_timestamps=[query.request.timestamp],
                     )

monitoring/uss_qualifier/suites/astm/utm/f3548_21.md

@@ -112,7 +112,7 @@
     <td><a href="../../../scenarios/astm/utm/nominal_planning/conflict_higher_priority/conflict_higher_priority.md">Nominal planning: conflict with higher priority</a><br><a href="../../../scenarios/astm/utm/nominal_planning/conflict_equal_priority_not_permitted/conflict_equal_priority_not_permitted.md">Nominal planning: not permitted conflict with equal priority</a><br><a href="../../../scenarios/astm/utm/flight_intent_validation/flight_intent_validation.md">Validation of operational intents</a></td>
   </tr>
   <tr>
-    <td rowspan="3" style="vertical-align:top;"><a href="../../../requirements/interuss/automated_testing/flight_planning.md">interuss<br>.automated_testing<br>.flight_planning</a></td>
+    <td rowspan="4" style="vertical-align:top;"><a href="../../../requirements/interuss/automated_testing/flight_planning.md">interuss<br>.automated_testing<br>.flight_planning</a></td>
     <td><a href="../../../requirements/interuss/automated_testing/flight_planning.md">ClearArea</a></td>
     <td>Implemented</td>
     <td><a href="../../../scenarios/astm/utm/nominal_planning/conflict_higher_priority/conflict_higher_priority.md">Nominal planning: conflict with higher priority</a><br><a href="../../../scenarios/astm/utm/nominal_planning/conflict_equal_priority_not_permitted/conflict_equal_priority_not_permitted.md">Nominal planning: not permitted conflict with equal priority</a><br><a href="../../../scenarios/astm/utm/flight_intent_validation/flight_intent_validation.md">Validation of operational intents</a></td>
@@ -127,4 +127,9 @@
     <td>Implemented</td>
     <td><a href="../../../scenarios/astm/utm/nominal_planning/conflict_higher_priority/conflict_higher_priority.md">Nominal planning: conflict with higher priority</a><br><a href="../../../scenarios/astm/utm/nominal_planning/conflict_equal_priority_not_permitted/conflict_equal_priority_not_permitted.md">Nominal planning: not permitted conflict with equal priority</a><br><a href="../../../scenarios/astm/utm/flight_intent_validation/flight_intent_validation.md">Validation of operational intents</a></td>
   </tr>
+  <tr>
+    <td><a href="../../../requirements/interuss/automated_testing/flight_planning.md">FlightCoveredByOperationalIntent</a></td>
+    <td>Implemented</td>
+    <td><a href="../../../scenarios/astm/utm/nominal_planning/conflict_higher_priority/conflict_higher_priority.md">Nominal planning: conflict with higher priority</a><br><a href="../../../scenarios/astm/utm/nominal_planning/conflict_equal_priority_not_permitted/conflict_equal_priority_not_permitted.md">Nominal planning: not permitted conflict with equal priority</a><br><a href="../../../scenarios/astm/utm/flight_intent_validation/flight_intent_validation.md">Validation of operational intents</a></td>
+  </tr>
 </table>