diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.md b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.md index 4fbb2cdfb4..61af0ba764 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.md +++ b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.md @@ -2,8 +2,8 @@ ## Description This test checks that the USS being tested validates the operational intents received as response to its GET request from another USS. -Control_uss which is a mock uss plans a nearby V-shaped operation, and provides the data that tested_uss GETs. -tested_uss validates the GET response from control_uss and accordingly plan its operation. +mock_uss plans a nearby V-shaped operation, and provides the data that tested_uss GETs. +tested_uss validates the GET response from mock_uss and accordingly plan its operation. Notably the following requirements: - **[astm.f3548.v21.SCD0035](../../../../requirements/astm/f3548/v21.md)** @@ -18,7 +18,7 @@ There is an overlap in time and altitude of the two flights. - flight_1 - flight_2 -### control_uss +### mock_uss MockUSSResource that will be used for planning flights, controlling data shared for validation testing, and gathering interuss interactions from mock_uss. ### tested_uss @@ -29,74 +29,68 @@ DSSInstanceResource that provides access to a DSS instance where flight creation ## Successfully plan flight near an existing flight test case -### [Control_uss plans flight 2 test step](../../../flight_planning/plan_flight_intent.md) +### [mock_uss plans flight 2 test step](../../../flight_planning/plan_flight_intent.md) Flight 2 should be successfully planned by the control USS. ### [Validate flight 2 sharing test step](../validate_shared_operational_intent.md) Validate that flight 2 is planned -### Check for notification to tested_uss due to subscription in flight 2 area test step -In order to run this test scenario, we need tested_uss to trigger GET operational intent request to mock_uss. -But, if there is a subscription by tested_uss, that would trigger notification of flight 2 to tested_uss. -Some USSes will not make a GET request to control_uss for flight 2 details, while planning a nearby flight, -if they got a notification for flight2. Hence, if a USS didn't make a GET request, we will only fail it if didn't get -a notification, or else, a precondition for the test will not be met. Some USSes might make a GET request despite getting -a notification, but as it would not be clear whether invalid information through notification or GET request was used for planning, -the test will be not be continued. - -### [Tested_uss plans flight 1 test step](../../../flight_planning/plan_flight_intent.md) +### [tested_uss plans flight 1 test step](../../../flight_planning/plan_flight_intent.md) The test driver attempts to plan flight 1 via the tested USS. It checks if any conflicts with flight 2 which is of equal priority and came first. ### [Validate flight 1 sharing test step](../validate_shared_operational_intent.md) Validate flight 1 is planned. +### Check for notification to tested_uss due to subscription in flight 2 area test step +In the following test step, we want to assert that tested_uss must have retrieved operational intent details from +mock_uss via a GET request. This assertion is only valid, however, if tested_uss did not obtain the operational +intent details in a different way -- specifically, a notification due to a pre-existing subscription. In this test +step, we determine if tested_uss had a pre-existing subscription by: + +#### [checking if mock_uss sent a notification to tested_uss](test_steps/query_mock_uss_interactions.md) + ### [Validate flight2 GET interaction, if no notification test step](test_steps/validate_get_operational_intent.md) -Validate that tested_uss makes a GET request for obtaining details of flight 2 from control_uss. -In a previous step (Precondition - check tested_uss has no subscription in flight 2 area), we ensured that no notification of flight 2 was sent to tested_uss. -Hence, tested_uss will need to make a GET request to obtain flight 2 details. +This step is skipped if a notification to tested_uss was found in the previous step. -### [Validate flight1 Notification sent to Control_uss test step](test_steps/validate_notification_operational_intent.md) -Tested_uss notifies flight 1 to Control_uss, due to its subscription through flight 2. +### [Validate flight1 Notification sent to mock_uss test step](test_steps/validate_notification_operational_intent.md) +tested_uss notifies flight 1 to mock_uss, due to its subscription through flight 2. ### [Delete tested_uss flight test step](../../../flight_planning/delete_flight_intent.md) Teardown -### [Delete control_uss flight test step](../../../flight_planning/delete_flight_intent.md) +### [Delete mock_uss flight test step](../../../flight_planning/delete_flight_intent.md) Teardown ## Flight planning prevented due to invalid data sharing test case -### [Control_uss plans flight 2, sharing invalid operational intent data test step](../../../flight_planning/plan_flight_intent.md) -Flight 2 should be successfully planned by the control_uss. -The control_uss, which is mock_uss is instructed to share invalid data with other USS, for negative test. +### [mock_uss plans flight 2, sharing invalid operational intent data test step](../../../flight_planning/plan_flight_intent.md) +Flight 2 should be successfully planned by the mock_uss. +The mock_uss is instructed to share invalid data with other USS, for negative test. ### [Validate flight 2 shared operational intent with invalid data test step](test_steps/validate_sharing_operational_intent_but_with_invalid_interuss_data.md) Validate that flight 2 is shared with invalid data as a modified behavior is injected by uss_qualifier for a negative test. -### Check for notification to tested_uss due to subscription in flight 2 area test step -In order to run this test scenario, we need tested_uss to trigger GET operational intent request to mock_uss. -But, if there is a subscription by tested_uss, that would trigger notification of flight 2 to tested_uss. -Some USSes will not make a GET request to control_uss for flight 2 details, while planning a nearby flight, -if they got a notification for flight2. Hence, if a USS didn't make a GET request, we will only fail it if didn't get -a notification, or else, a precondition for the test will not be met. Some USSes might make a GET request despite getting -a notification, but as it would not be clear whether invalid information through notification or GET request was used for planning, -the test will be not be continued. - -### [Tested_uss attempts to plan flight 1, expect failure test step](test_steps/plan_flight_intent_expect_failed.md) +### [tested_uss attempts to plan flight 1, expect failure test step](test_steps/plan_flight_intent_expect_failed.md) The test driver attempts to plan the flight 1 via the tested_uss. It checks if any conflicts with flight 2 which is of equal priority and came first. ### [Validate flight 1 not shared by tested_uss test step](../validate_not_shared_operational_intent.md) Validate flight 1 is not shared with DSS, as plan failed. +### Check for notification to tested_uss due to subscription in flight 2 area test step +In the following test step, we want to assert that tested_uss must have retrieved operational intent details from +mock_uss via a GET request. This assertion is only valid, however, if tested_uss did not obtain the operational +intent details in a different way -- specifically, a notification due to a pre-existing subscription. In this test +step, we determine if tested_uss had a pre-existing subscription by: + +#### [checking if mock_uss sent a notification to tested_uss](test_steps/query_mock_uss_interactions.md) + ### [Validate flight2 GET interaction, if no notification test step](test_steps/validate_get_operational_intent.md) -Validate that tested_uss makes a GET request for obtaining details of flight 2 from control_uss. -In a previous step (Precondition - check tested_uss has no subscription in flight 2 area), we ensured that no notification of flight 2 was sent to tested_uss. -Hence, tested_uss will need to make a GET request to obtain flight 2 details. +This step is skipped if a notification to tested_uss was found in the previous step. -### [Validate flight 1 Notification not sent to Control_uss test step](test_steps/validate_no_notification_operational_intent.md) +### [Validate flight 1 Notification not sent to mock_uss test step](test_steps/validate_no_notification_operational_intent.md) -### [Delete Control_uss flight test step](../../../flight_planning/delete_flight_intent.md) +### [Delete mock_uss flight test step](../../../flight_planning/delete_flight_intent.md) Teardown ## Cleanup diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.py b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.py index 9e5a341c72..29c5fa86f6 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.py +++ b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.py @@ -1,7 +1,13 @@ from typing import Optional, Dict + +from monitoring.monitorlib.clients.flight_planning.flight_info import ( + AirspaceUsageState, + UasState, +) from monitoring.monitorlib.clients.flight_planning.flight_info_template import ( FlightInfoTemplate, ) +from monitoring.monitorlib.clients.mock_uss.interactions import QueryDirection from monitoring.monitorlib.temporal import TimeDuringTest import arrow from monitoring.monitorlib.temporal import Time @@ -11,6 +17,13 @@ from monitoring.uss_qualifier.resources.flight_planning import ( FlightIntentsResource, ) +from monitoring.uss_qualifier.resources.flight_planning.flight_intent import ( + FlightIntent, +) +from monitoring.uss_qualifier.resources.flight_planning.flight_intent_validation import ( + ExpectedFlightIntent, + validate_flight_intent_templates, +) from monitoring.uss_qualifier.resources.flight_planning.flight_planners import ( FlightPlannerResource, ) @@ -28,10 +41,10 @@ OpIntentValidationFailureType, ) from monitoring.uss_qualifier.scenarios.astm.utm.data_exchange_validation.test_steps.expected_interactions_test_steps import ( - expect_interuss_post_interactions, - expect_get_requests_to_mock_uss_when_no_notification, expect_no_interuss_post_interactions, - check_any_notification, + expect_mock_uss_receives_op_intent_notification, + mock_uss_interactions, + is_op_intent_notification_with_id, ) from monitoring.monitorlib.clients.mock_uss.mock_uss_scd_injection_api import ( MockUssFlightBehavior, @@ -46,6 +59,7 @@ delete_flight, ) from monitoring.uss_qualifier.suites.suite import ExecutionContext +from uas_standards.astm.f3548.v21.api import OperationID class GetOpResponseDataValidationByUSS(TestScenario): @@ -53,21 +67,21 @@ class GetOpResponseDataValidationByUSS(TestScenario): flight_2: FlightInfoTemplate tested_uss_client: FlightPlannerClient - control_uss: MockUSSClient - control_uss_client: FlightPlannerClient + mock_uss: MockUSSClient + mock_uss_client: FlightPlannerClient dss: DSSInstance def __init__( self, tested_uss: FlightPlannerResource, - control_uss: MockUSSResource, + mock_uss: MockUSSResource, dss: DSSInstanceResource, flight_intents: Optional[FlightIntentsResource] = None, ): super().__init__() self.tested_uss_client = tested_uss.client - self.control_uss = control_uss.mock_uss - self.control_uss_client = control_uss.mock_uss.flight_planner + self.mock_uss = mock_uss.mock_uss + self.mock_uss_client = mock_uss.mock_uss.flight_planner self.dss = dss.dss if not flight_intents: @@ -78,40 +92,46 @@ def __init__( ) raise ScenarioCannotContinueError(msg) - _flight_intents = flight_intents.get_flight_intents() - - t_now = Time(arrow.utcnow().datetime) - times = { - TimeDuringTest.StartOfTestRun: t_now, - TimeDuringTest.StartOfScenario: t_now, - TimeDuringTest.TimeOfEvaluation: t_now, - } - extents = [] - for intent in _flight_intents.values(): - extents.append(intent.resolve(times).basic_information.area.bounding_volume) - self._intents_extent = Volume4DCollection(extents).bounding_volume.to_f3548v21() - + expected_flight_intents = [ + ExpectedFlightIntent( + "flight_1", + "Flight 1", + must_not_conflict_with=["Flight 2"], + f3548v21_priority_equal_to=["Flight 2"], + usage_state=AirspaceUsageState.Planned, + uas_state=UasState.Nominal, + # TODO: Must intersect bounding box of Flight 2 + ), + ExpectedFlightIntent( + "flight_2", + "Flight 2", + must_not_conflict_with=["Flight 1"], + f3548v21_priority_equal_to=["Flight 1"], + usage_state=AirspaceUsageState.Planned, + uas_state=UasState.Nominal, + # TODO: Must intersect bounding box of Flight 1 + ), + ] + + templates = flight_intents.get_flight_intents() try: - (self.flight_1, self.flight_2,) = ( - _flight_intents["flight_1"], - _flight_intents["flight_2"], - ) - - assert not self.flight_1.resolve( - times - ).basic_information.area.intersects_vol4s( - self.flight_2.resolve(times).basic_information.area - ), "flight_1 and flight_2 must not intersect" - - except KeyError as e: - raise ValueError( - f"`{self.me()}` TestScenario requirements for flight_intents not met: missing flight intent {e}" - ) - except AssertionError as e: + validate_flight_intent_templates(templates, expected_flight_intents) + except ValueError as e: raise ValueError( f"`{self.me()}` TestScenario requirements for flight_intents not met: {e}" ) + extents = [] + for efi in expected_flight_intents: + intent = FlightIntent.from_flight_info_template(templates[efi.intent_id]) + extents.extend(intent.request.operational_intent.volumes) + extents.extend(intent.request.operational_intent.off_nominal_volumes) + setattr(self, efi.intent_id, templates[efi.intent_id]) + + self._intents_extent = Volume4DCollection.from_interuss_scd_api( + extents + ).bounding_volume.to_f3548v21() + def run(self, context: ExecutionContext): times = { TimeDuringTest.StartOfTestRun: Time(context.start_time), @@ -125,54 +145,41 @@ def run(self, context: ExecutionContext): ) self.record_note( "Control USS", - f"{self.control_uss_client.participant_id}", + f"{self.mock_uss_client.participant_id}", ) self.begin_test_case("Successfully plan flight near an existing flight") - self._tested_uss_plans_deconflicted_flight_near_existing_flight(times) + self._plan_successfully_test_case(times) self.end_test_case() self.begin_test_case("Flight planning prevented due to invalid data sharing") - self._tested_uss_unable_to_plan_flight_near_invalid_shared_existing_flight( - times - ) + self._plan_unsuccessfully_test_case(times) self.end_test_case() self.end_test_scenario() - def _tested_uss_plans_deconflicted_flight_near_existing_flight( - self, times: Dict[TimeDuringTest, Time] - ): + def _plan_successfully_test_case(self, times: Dict[TimeDuringTest, Time]): times[TimeDuringTest.TimeOfEvaluation] = Time(arrow.utcnow().datetime) flight_2 = self.flight_2.resolve(times) with OpIntentValidator( self, - self.control_uss_client, + self.mock_uss_client, self.dss, "Validate flight 2 sharing", self._intents_extent, ) as validator: - planning_time = Time(arrow.utcnow().datetime) + self.begin_test_step("mock_uss plans flight 2") + flight_2_planning_time = Time(arrow.utcnow().datetime) _, self.flight_2_id = plan_flight( self, - "Control_uss plans flight 2", - self.control_uss_client, + self.mock_uss_client, flight_2, ) + self.end_test_step() flight_2_oi_ref = validator.expect_shared(flight_2) - self.begin_test_step( - "Check for notification to tested_uss due to subscription in flight 2 area" - ) - tested_uss_notified = check_any_notification( - self, - self.control_uss, - planning_time, - ) - self.end_test_step() - times[TimeDuringTest.TimeOfEvaluation] = Time(arrow.utcnow().datetime) flight_1 = self.flight_1.resolve(times) @@ -183,47 +190,77 @@ def _tested_uss_plans_deconflicted_flight_near_existing_flight( "Validate flight 1 sharing", self._intents_extent, ) as validator: - planning_time = Time(arrow.utcnow().datetime) + self.begin_test_step("tested_uss plans flight 1") + flight_1_planning_time = Time(arrow.utcnow().datetime) plan_res, self.flight_1_id = plan_flight( self, - "Tested_uss plans flight 1", self.tested_uss_client, flight_1, ) + self.end_test_step() validator.expect_shared( flight_1, ) - if not tested_uss_notified: - expect_get_requests_to_mock_uss_when_no_notification( - self, - self.control_uss, - planning_time, - self.control_uss.base_url, - flight_2_oi_ref.id, - self.tested_uss_client.participant_id, - "Validate flight2 GET interaction, if no notification", + + self.begin_test_step( + "Check for notification to tested_uss due to subscription in flight 2 area" + ) + tested_uss_notifications, _ = mock_uss_interactions( + scenario=self, + mock_uss=self.mock_uss, + op_id=OperationID.NotifyOperationalIntentDetailsChanged, + direction=QueryDirection.Outgoing, + since=flight_2_planning_time, + is_applicable=is_op_intent_notification_with_id(flight_2_oi_ref.id), + ) + self.end_test_step() + + self.begin_test_step("Validate flight2 GET interaction, if no notification") + if not tested_uss_notifications: + tested_uss_get_requests, query = mock_uss_interactions( + scenario=self, + mock_uss=self.mock_uss, + op_id=OperationID.GetOperationalIntentDetails, + direction=QueryDirection.Incoming, + since=flight_1_planning_time, + query_params={"entity_id": flight_2_oi_ref.id}, + ) + with self.check( + "Expect GET request when no notification", + [self.tested_uss_client.participant_id], + ) as check: + if not tested_uss_get_requests: + check.record_failed( + summary=f"mock_uss did not GET op intent details when planning", + details=f"mock_uss did not receive a request to GET operational intent details for operational intent {flight_2_oi_ref.id}. tested_uss was not sent a notification with the operational intent details, so they should have requested the operational intent details during planning.", + query_timestamps=[query.request.timestamp], + ) + else: + self.record_note( + "No flight 2a GET expected reason", + f"Notifications found to {', '.join(n.query.request.url for n in tested_uss_notifications)}", ) + self.end_test_step() - expect_interuss_post_interactions( + self.begin_test_step("Validate flight1 Notification sent to mock_uss") + expect_mock_uss_receives_op_intent_notification( self, - self.control_uss, - planning_time, - self.control_uss.base_url, + self.mock_uss, + flight_1_planning_time, self.tested_uss_client.participant_id, plan_res.queries[0].request.timestamp, - "Validate flight1 Notification sent to Control_uss", ) + self.end_test_step() - delete_flight( - self, "Delete tested_uss flight", self.tested_uss_client, self.flight_1_id - ) - delete_flight( - self, "Delete control_uss flight", self.control_uss_client, self.flight_2_id - ) + self.begin_test_step("Delete tested_uss flight") + delete_flight(self, self.tested_uss_client, self.flight_1_id) + self.end_test_step() - def _tested_uss_unable_to_plan_flight_near_invalid_shared_existing_flight( - self, times: Dict[TimeDuringTest, Time] - ): + self.begin_test_step("Delete mock_uss flight") + delete_flight(self, self.mock_uss_client, self.flight_2_id) + self.end_test_step() + + def _plan_unsuccessfully_test_case(self, times: Dict[TimeDuringTest, Time]): times[TimeDuringTest.TimeOfEvaluation] = Time(arrow.utcnow().datetime) flight_info = self.flight_2.resolve(times) @@ -241,35 +278,28 @@ def _tested_uss_unable_to_plan_flight_near_invalid_shared_existing_flight( additional_fields = {"behavior": behavior} with OpIntentValidator( self, - self.control_uss_client, + self.mock_uss_client, self.dss, "Validate flight 2 shared operational intent with invalid data", self._intents_extent, ) as validator: - planning_time = Time(arrow.utcnow().datetime) + self.begin_test_step( + "mock_uss plans flight 2, sharing invalid operational intent data" + ) + flight_2_planning_time = Time(arrow.utcnow().datetime) _, self.flight_2_id = plan_flight( self, - "Control_uss plans flight 2, sharing invalid operational intent data", - self.control_uss_client, + self.mock_uss_client, flight_info, additional_fields, ) + self.end_test_step() flight_2_oi_ref = validator.expect_shared_with_invalid_data( flight_info, validation_failure_type=OpIntentValidationFailureType.DataFormat, invalid_fields=[modify_field1, modify_field2], ) - self.begin_test_step( - "Check for notification to tested_uss due to subscription in flight 2 area" - ) - tested_uss_notified = check_any_notification( - self, - self.control_uss, - planning_time, - ) - self.end_test_step() - times[TimeDuringTest.TimeOfEvaluation] = Time(arrow.utcnow().datetime) flight_1 = self.flight_1.resolve(times) with OpIntentValidator( @@ -279,41 +309,70 @@ def _tested_uss_unable_to_plan_flight_near_invalid_shared_existing_flight( "Validate flight 1 not shared by tested_uss", self._intents_extent, ) as validator: - planning_time = Time(arrow.utcnow().datetime) + self.begin_test_step("tested_uss attempts to plan flight 1, expect failure") + flight_1_planning_time = Time(arrow.utcnow().datetime) _, self.flight_1_id = plan_flight_intent_expect_failed( self, - "Tested_uss attempts to plan flight 1, expect failure", self.tested_uss_client, flight_1, ) + self.end_test_step() validator.expect_not_shared() - if not tested_uss_notified: - expect_get_requests_to_mock_uss_when_no_notification( - self, - self.control_uss, - planning_time, - self.control_uss.base_url, - flight_2_oi_ref.id, - self.tested_uss_client.participant_id, - "Validate flight2 GET interaction, if no notification", + self.begin_test_step( + "Check for notification to tested_uss due to subscription in flight 2 area" + ) + tested_uss_notifications, _ = mock_uss_interactions( + scenario=self, + mock_uss=self.mock_uss, + op_id=OperationID.NotifyOperationalIntentDetailsChanged, + direction=QueryDirection.Outgoing, + since=flight_2_planning_time, + is_applicable=is_op_intent_notification_with_id(flight_2_oi_ref.id), + ) + self.end_test_step() + + self.begin_test_step("Validate flight2 GET interaction, if no notification") + if not tested_uss_notifications: + tested_uss_get_requests, query = mock_uss_interactions( + scenario=self, + mock_uss=self.mock_uss, + op_id=OperationID.GetOperationalIntentDetails, + direction=QueryDirection.Incoming, + since=flight_1_planning_time, + query_params={"entity_id": flight_2_oi_ref.id}, + ) + with self.check( + "Expect GET request when no notification", + [self.tested_uss_client.participant_id], + ) as check: + if not tested_uss_get_requests: + check.record_failed( + summary=f"mock_uss did not GET op intent details when planning", + details=f"mock_uss did not receive a request to GET operational intent details for operational intent {flight_2_oi_ref.id}. tested_uss was not sent a notification with the operational intent details, so they should have requested the operational intent details during planning.", + query_timestamps=[query.request.timestamp], + ) + else: + self.record_note( + "No flight 2b GET expected reason", + f"Notifications found to {', '.join(n.query.request.url for n in tested_uss_notifications)}", ) + self.end_test_step() + self.begin_test_step("Validate flight 1 Notification not sent to mock_uss") expect_no_interuss_post_interactions( self, - self.control_uss, - planning_time, + self.mock_uss, + flight_1_planning_time, self.tested_uss_client.participant_id, - "Validate flight 1 Notification not sent to Control_uss", ) + self.end_test_step() - delete_flight( - self, "Delete Control_uss flight", self.control_uss_client, self.flight_2_id - ) + self.begin_test_step("Delete mock_uss flight") + delete_flight(self, self.mock_uss_client, self.flight_2_id) + self.end_test_step() def cleanup(self): self.begin_cleanup() - cleanup_flights_fp_client( - self, (self.control_uss_client, self.tested_uss_client) - ), + cleanup_flights_fp_client(self, (self.mock_uss_client, self.tested_uss_client)), self.end_cleanup() diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/expected_interactions_test_steps.py b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/expected_interactions_test_steps.py index 0278488416..20c178b88d 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/expected_interactions_test_steps.py +++ b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/expected_interactions_test_steps.py @@ -1,70 +1,68 @@ from __future__ import annotations -import datetime -from typing import List, Tuple, Optional +from datetime import datetime, timedelta +import re +from typing import Callable, Dict, List, Tuple, Optional import time -from monitoring.monitorlib.fetch import QueryError, Query -from monitoring.uss_qualifier.common_data_definitions import Severity -from monitoring.uss_qualifier.scenarios.scenario import TestScenarioType -from monitoring.uss_qualifier.resources.interuss.mock_uss.client import MockUSSClient +import arrow from implicitdict import StringBasedDateTime from loguru import logger +from uas_standards.astm.f3548.v21 import api +from uas_standards.astm.f3548.v21.api import OperationID, EntityID + from monitoring.monitorlib.clients.mock_uss.interactions import Interaction +from monitoring.monitorlib.clients.mock_uss.interactions import QueryDirection +from monitoring.monitorlib.fetch import QueryError, Query +from monitoring.uss_qualifier.common_data_definitions import Severity +from monitoring.uss_qualifier.resources.interuss.mock_uss.client import MockUSSClient from monitoring.uss_qualifier.scenarios.astm.utm.data_exchange_validation.test_steps.constants import ( MaxTimeToWaitForSubscriptionNotificationSeconds as max_wait_time, ) +from monitoring.uss_qualifier.scenarios.scenario import TestScenarioType # Interval to wait for checking notification received -WAIT_INTERVAL = 1 +WAIT_INTERVAL_SECONDS = 1 -def expect_interuss_post_interactions( +def expect_mock_uss_receives_op_intent_notification( scenario: TestScenarioType, mock_uss: MockUSSClient, st: StringBasedDateTime, - posted_to_url: str, participant_id: str, - plan_request_time: datetime.datetime, - test_step: str, + plan_request_time: datetime, ): - """ - This step checks if a notification was sent to a subscribed USS, from time 'st' to now + """This step checks if a notification is sent to mock_uss within the required time window. + Args: - posted_to_url: url of the subscribed USS + st: the earliest time a notification may have been sent participant_id: id of the participant responsible to send the notification plan_request_time: timestamp of the flight plan query that would lead to sending notification - """ - scenario.begin_test_step(test_step) # Check for 'notification found' will be done periodically by waiting for a duration till max_wait_time - time_waited = 0 - duration = 0 - while time_waited <= max_wait_time: - time.sleep(duration) - interactions, query = _get_interuss_interactions_with_check( - scenario, - mock_uss, - st, + wait_until = arrow.utcnow().datetime + timedelta(seconds=max_wait_time) + while arrow.utcnow().datetime < wait_until: + found, query = mock_uss_interactions( + scenario=scenario, + mock_uss=mock_uss, + op_id=OperationID.NotifyOperationalIntentDetailsChanged, + direction=QueryDirection.Incoming, + since=st, ) - found = _any_oi_notification_in_interactions(interactions, posted_to_url) - time_waited += duration if found: - logger.debug(f"Waited for {time_waited} to check notifications.") break - # wait for WAIT_INTERVAL till max_wait_time reached - duration = min(WAIT_INTERVAL, max_wait_time - time_waited) + dt = (wait_until - arrow.utcnow().datetime).total_seconds() + if dt > 0: + time.sleep(min(dt, WAIT_INTERVAL_SECONDS)) with scenario.check("Expect Notification sent", [participant_id]) as check: if not found: check.record_failed( - summary=f"Notification to {posted_to_url} not sent", - severity=Severity.Medium, - details=f"Notification to {posted_to_url} not sent even though DSS instructed the planning USS to notify due to subscription.", + summary=f"Notification not sent", + details=f"Notification to USS with pre-existing relevant operational intent not sent even though DSS instructed the planning USS to notify due to subscription.", query_timestamps=[plan_request_time, query.request.timestamp], ) - scenario.end_test_step() def expect_no_interuss_post_interactions( @@ -72,161 +70,91 @@ def expect_no_interuss_post_interactions( mock_uss: MockUSSClient, st: StringBasedDateTime, participant_id: str, - test_step: str, ): - """ - This step checks no notification was sent to any USS as no DSS entity was created, from time 'st' to now + """This step checks no notification is sent to any USS within the required time window (as no DSS entity was created). + Args: + st: the earliest time a notification may have been sent participant_id: id of the participant responsible to send the notification """ - scenario.begin_test_step(test_step) - # Wait for next MaxTimeToWaitForSubscriptionNotificationSeconds duration to capture any notification time.sleep(max_wait_time) - interactions, query = _get_interuss_interactions_with_check( - scenario, - mock_uss, - st, + found, query = mock_uss_interactions( + scenario=scenario, + mock_uss=mock_uss, + op_id=OperationID.NotifyOperationalIntentDetailsChanged, + direction=QueryDirection.Incoming, + since=st, ) - found = _any_oi_notification_in_interactions(interactions) with scenario.check("Expect Notification not sent", [participant_id]) as check: if found: check.record_failed( summary=f"Notification was wrongly sent for an entity not created.", - severity=Severity.Medium, details=f"Notification was wrongly sent for an entity not created.", query_timestamps=[query.request.timestamp], ) - scenario.end_test_step() - - -def expect_get_requests_to_mock_uss_when_no_notification( - scenario: TestScenarioType, - mock_uss: MockUSSClient, - st: StringBasedDateTime, - mock_uss_base_url: str, - id: str, - participant_id: str, - test_step: str, -): - """ - This step checks a GET request was made to mock_uss for an existing entity, from time 'st' to now - Args: - mock_uss_base_url: url of the mock_uss that is managing the entity - id: entity id - participant_id: id of the participant responsible to send GET request - - """ - scenario.begin_test_step(test_step) - interactions, query = _get_interuss_interactions_with_check(scenario, mock_uss, st) - logger.debug(f"Checking for GET request to {mock_uss_base_url} for id {id}") - get_requested = False - for interaction in interactions: - method = interaction.query.request.method - url = interaction.query.request.url - if method == "GET" and url.startswith(mock_uss_base_url) and id in url: - get_requested = True - break - if not get_requested: - with scenario.check( - "Expect GET request when no notification", [participant_id] - ) as check: - check.record_failed( - summary=f"No GET request received at {mock_uss_base_url} for {id} ", - severity=Severity.High, - details=f"No GET request received at {mock_uss_base_url} for {id}. A planning USS in the area should have sent a reques to get the intent details.", - query_timestamps=[query.request.timestamp], - ) - scenario.end_test_step() -def _get_interuss_interactions_with_check( +def mock_uss_interactions( scenario: TestScenarioType, mock_uss: MockUSSClient, - st: StringBasedDateTime, + op_id: OperationID, + direction: QueryDirection, + since: StringBasedDateTime, + query_params: Optional[Dict[str, str]] = None, + is_applicable: Optional[Callable[[Interaction], bool]] = None, ) -> Tuple[List[Interaction], Query]: - """ - Method to get interuss interactions with a scenario check from mock_uss from time 'st' to now. - """ + """Determine if mock_uss recorded an interaction for the specified operation in the specified direction.""" + with scenario.check( "MockUSS interactions request", [mock_uss.participant_id] ) as check: try: - interactions, query = _get_interuss_interactions( - mock_uss, - st, - ) + interactions, query = mock_uss.get_interactions(since) scenario.record_query(query) - return interactions, query except QueryError as e: for q in e.queries: scenario.record_query(q) check.record_failed( summary=f"Error from mock_uss when attempting to get interactions from_time {st}", - severity=Severity.High, details=f"{str(e)}\n\nStack trace:\n{e.stacktrace}", query_timestamps=[q.request.timestamp for q in e.queries], ) + op = api.OPERATIONS[op_id] -def _get_interuss_interactions( - mock_uss: MockUSSClient, - st: StringBasedDateTime, -) -> Tuple[List[Interaction], Query]: - """ - Method to get interuss interactions from mock_uss from time 'st' to now. - """ - all_interactions, query = mock_uss.get_interactions(st) - exclude_sub = mock_uss.session.auth_adapter.get_sub() - - def is_uss_interaction(interaction: Interaction, excl_sub: str) -> bool: - sub = interaction.query.get_client_sub() - if sub: - if sub == excl_sub: - return False - else: - return True - else: - logger.error(f"Interaction received without Authorization : {interaction}") - return False - - interuss_interactions = [] - for interaction in all_interactions: - if is_uss_interaction(interaction, exclude_sub): - interuss_interactions.append(interaction) - - return interuss_interactions, query + op_path = op.path + if query_params is None: + query_params = {} + for m in re.findall(r"\{[^}]+\}", op_path): + param_name = m[1:-1] + op_path = op_path.replace(m, query_params.get(param_name, r"[^/]+")) - -def check_any_notification( - scenario: TestScenarioType, - mock_uss: MockUSSClient, - st: StringBasedDateTime, -) -> bool: - """ - This method helps check any notification have been sent, to or from mock_uss. - - Returns: True if any notification found, otherwise False - """ - interactions, query = _get_interuss_interactions( - mock_uss, - st, - ) - scenario.record_query(query) - return _any_oi_notification_in_interactions(interactions) - - -def _any_oi_notification_in_interactions( - interactions: List[Interaction], recipient_base_url: Optional[str] = None -) -> bool: - """ - Checks if there is any POST request made to 'recipient_base_url', and returns True if found. - If 'recipient_base_url' is None, any POST request found returns True. - """ + if is_applicable is None: + is_applicable = lambda i: True + result = [] for interaction in interactions: - method = interaction.query.request.method - url = interaction.query.request.url - if method == "POST": - if recipient_base_url is None or url.startswith(recipient_base_url): - return True - return False + if ( + interaction.direction == direction + and interaction.query.request.method == op.verb + and re.search(op_path, interaction.query.request.url) + and is_applicable(interaction) + ): + result.append(interaction) + return result, query + + +def is_op_intent_notification_with_id( + op_intent_id: EntityID, +) -> Callable[[Interaction], bool]: + """Returns an `is_applicable` function that detects whether an op intent notification refers to the specified operational intent.""" + + def is_applicable(interaction: Interaction) -> bool: + if "json" in interaction.query.request and interaction.query.request.json: + return ( + interaction.query.request.json.get("operational_intent_id", None) + == op_intent_id + ) + return False + + return is_applicable diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/invalid_op_test_steps.py b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/invalid_op_test_steps.py index ed65613779..0b043d4e96 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/invalid_op_test_steps.py +++ b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/invalid_op_test_steps.py @@ -16,7 +16,6 @@ def plan_flight_intent_expect_failed( scenario: TestScenarioType, - test_step: str, flight_planner: FlightPlannerClient, flight_intent: FlightInfo, ) -> Tuple[PlanningActivityResponse, Optional[str]]: @@ -30,7 +29,6 @@ def plan_flight_intent_expect_failed( return submit_flight( scenario, - test_step, "Plan should fail", {(PlanningActivityResult.Failed, FlightPlanStatus.NotPlanned)}, {}, diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/plan_flight_intent_expect_failed.md b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/plan_flight_intent_expect_failed.md index 7afaa7b24a..8d053af834 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/plan_flight_intent_expect_failed.md +++ b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/plan_flight_intent_expect_failed.md @@ -2,7 +2,7 @@ This page describes the content of a common test case where a valid user flight intent fails in a flight planner, because of invalid data shared for a nearby flight shared by another USS. See `plan_flight_intent_expect_failed` in invalid_op_test_steps.py. -## Plan should fail check +## 🛑 Plan should fail check A USS shouldn't go ahead and plan if it doesn't have accurate information. As per SCD0035 a USS needs to verify a particular conflict status. diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/query_mock_uss_interactions.md b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/query_mock_uss_interactions.md new file mode 100644 index 0000000000..26862c3f8e --- /dev/null +++ b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/query_mock_uss_interactions.md @@ -0,0 +1,8 @@ +# Query mock_uss interactions test step fragment + +This step obtains interactions of interest from mock_uss. + +## 🛑 MockUSS interactions request check + +If the query to mock_uss fails or uss_qualifier is otherwise unable to retrieve the interactions of interest, the mock_uss provider does not meet +**[interuss.mock_uss.hosted_instance.ExposeInterface](../../../../../requirements/interuss/mock_uss/hosted_instance.md)**. diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_get_operational_intent.md b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_get_operational_intent.md index 82e6c5c709..88909a995b 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_get_operational_intent.md +++ b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_get_operational_intent.md @@ -2,9 +2,9 @@ This step verifies that a USS makes a GET request to get the intent_details of an existing operation when needed as per ASTM F3548-21 by checking the interuss interactions of mock uss -## MockUSS interactions request check +## 🛑 MockUSS interactions request check **[interuss.mock_uss.hosted_instance.ExposeInterface](../../../../../requirements/interuss/mock_uss/hosted_instance.md)**. -## Expect GET request when no notification check +## 🛑 Expect GET request when no notification check **[astm.f3548.v21.SCD0035](../../../../../requirements/astm/f3548/v21.md)** SCD0035 needs a USS to verify before transitioning to Accepted that it does not conflict with a type of operational intent, and the only way to have verified this is by knowing all operational intent details, and (from previous checks of no notifications) the only way to know the operational intent details of flight is to have requested them via a GET details interaction. diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_no_notification_operational_intent.md b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_no_notification_operational_intent.md index d05c766676..3a7127a5a8 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_no_notification_operational_intent.md +++ b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_no_notification_operational_intent.md @@ -2,10 +2,10 @@ This step verifies when a flight is not created, it is also not notified by checking the interuss interactions of mock_uss instance. -## MockUSS interactions request check +## 🛑 MockUSS interactions request check **[interuss.mock_uss.hosted_instance.ExposeInterface](../../../../../requirements/interuss/mock_uss/hosted_instance.md)**. -## Expect Notification not sent check +## 🛑 Expect Notification not sent check **[interuss.f3548.notification_requirements.NoDssEntityNoNotification](../../../../../requirements/interuss/f3548/notification_requirements.md)** diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_notification_operational_intent.md b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_notification_operational_intent.md index 48c83c8e34..f43e81e86e 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_notification_operational_intent.md +++ b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_notification_operational_intent.md @@ -2,10 +2,10 @@ This step verifies that, when creating or modifying an operational intent, a USS sent the required notification for a relevant subscription owned by a mock_uss instance by checking the interactions of that mock_uss instance. -## MockUSS interactions request check +## 🛑 MockUSS interactions request check **[interuss.mock_uss.hosted_instance.ExposeInterface](../../../../../requirements/interuss/mock_uss/hosted_instance.md)**. -## Expect Notification sent check +## ⚠️ Expect Notification sent check As per **[astm.f3548.v21.SCD0085](../../../../../requirements/astm/f3548/v21.md)**, the notification should be sent by a USS about its operational intent to the subscribing USS in no more than MaxRespondToSubscriptionNotification (5) seconds, 95 percent of the time. @@ -18,7 +18,7 @@ with 95 percentile at 5 seconds, then with the standard deviation of 3.04, we ge Hence, for test cases that check notification sent for an operational intent, we will wait for notifications till threshold [MaxTimeToWaitForSubscriptionNotificationSeconds](./constants.py) (rounding to 7 seconds). -####Note - +#### Note As per **[astm.f3548.v21.SCD0085](../../../../../requirements/astm/f3548/v21.md)**, MaxRespondToSubscriptionNotification is measured from time_start - Receipt of subscription notification from DSS - till time_end - Entity details sent to subscribing USS. @@ -26,8 +26,8 @@ To make sure the test driver gives enough time for notifications to be received it marks the time to get interactions from mock_uss as - the time test driver initiates the plan. The sequence of events is - 1. Test driver initiates plan to tested_uss. t0 -2. Tested_uss shares the plan with DSS and receives DSS response. t_time_start. -3. Tested_uss responds to test driver with Completed. +2. tested_uss shares the plan with DSS and receives DSS response. t_time_start. +3. tested_uss responds to test driver with Completed. 4. Test driver checks for shared operational_intent in DSS and checks its retrievable. t1 5. Test driver waits for MaxTimeToWaitForSubscriptionNotificationSeconds. 6. Test driver retrieves interactions from mock_uss. t1 + MaxTimeToWaitForSubscriptionNotificationSeconds @@ -39,7 +39,5 @@ So, it starts waiting from a point of time after the t_time_start that is t1. This ensures that test driver waits for a long enough duration before getting the interactions. Hence, we get a high confidence that the test driver correctly verifies if a notification was sent by tested_uss. - - -## Notification data is valid check +## 🛑 Notification data is valid check **[astm.f3548.v21.SCD0085](../../../../../requirements/astm/f3548/v21.md)** diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_sharing_operational_intent_but_with_invalid_interuss_data.md b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_sharing_operational_intent_but_with_invalid_interuss_data.md index 46b70a3e15..12f4ce1496 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_sharing_operational_intent_but_with_invalid_interuss_data.md +++ b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_sharing_operational_intent_but_with_invalid_interuss_data.md @@ -2,19 +2,19 @@ This step verifies that a created flight is shared properly per ASTM F3548-21 by querying the DSS for flights in the area of the flight intent, and then retrieving the details from the USS if the operational intent reference is found. See `expect_shared_with_invalid_data` in invalid_op_test_steps.py. -## DSS responses check +## 🛑 DSS responses check **[astm.f3548.v21.DSS0005](../../../../../requirements/astm/f3548/v21.md)** -## Operational intent shared correctly check +## 🛑 Operational intent shared correctly check If a reference to the operational intent for the flight is not found in the DSS, this check will fail per **[astm.f3548.v21.USS0005](../../../../../requirements/astm/f3548/v21.md)** and **[astm.f3548.v21.OPIN0025](../../../../../requirements/astm/f3548/v21.md)**. -## Operational intent details retrievable check +## 🛑 Operational intent details retrievable check If the operational intent details for the flight cannot be retrieved from the USS, this check will fail per **[astm.f3548.v21.USS0105](../../../../../requirements/astm/f3548/v21.md)** and **[astm.f3548.v21.OPIN0025](../../../../../requirements/astm/f3548/v21.md)**. -## Invalid data in Operational intent details shared by Mock USS for negative test check +## 🛑 Invalid data in Operational intent details shared by Mock USS for negative test check **[interuss.mock_uss.hosted_instance.ExposeInterface](../../../../../requirements/interuss/mock_uss/hosted_instance.md)**. diff --git a/monitoring/uss_qualifier/scenarios/flight_planning/test_steps.py b/monitoring/uss_qualifier/scenarios/flight_planning/test_steps.py index aff6c3037d..9ca0e6ea8e 100644 --- a/monitoring/uss_qualifier/scenarios/flight_planning/test_steps.py +++ b/monitoring/uss_qualifier/scenarios/flight_planning/test_steps.py @@ -383,14 +383,13 @@ def cleanup_flights( def plan_flight( scenario: TestScenarioType, - test_step: str, flight_planner: FlightPlannerClient, flight_info: FlightInfo, additional_fields: Optional[dict] = None, ) -> Tuple[PlanningActivityResponse, Optional[str]]: """Plan a flight intent that should result in success. - This function implements the test step described in + This function implements the test step fragment described in plan_flight_intent.md. Returns: @@ -399,7 +398,6 @@ def plan_flight( """ return submit_flight( scenario=scenario, - test_step=test_step, success_check="Successful planning", expected_results={(PlanningActivityResult.Completed, FlightPlanStatus.Planned)}, failed_checks={PlanningActivityResult.Failed: "Failure"}, @@ -411,10 +409,9 @@ def plan_flight( def submit_flight( scenario: TestScenarioType, - test_step: str, success_check: str, expected_results: Set[Tuple[PlanningActivityResult, FlightPlanStatus]], - failed_checks: Dict[PlanningActivityResult, Union[str, Tuple[str, Severity]]], + failed_checks: Dict[PlanningActivityResult, str], flight_planner: FlightPlannerClient, flight_info: FlightInfo, flight_id: Optional[str] = None, @@ -431,7 +428,6 @@ def submit_flight( * The ID of the injected flight if it is returned, None otherwise. """ - scenario.begin_test_step(test_step) with scenario.check(success_check, [flight_planner.participant_id]) as check: try: resp, query, flight_id = request_flight( @@ -442,7 +438,6 @@ def submit_flight( scenario.record_query(q) check.record_failed( summary=f"Error from {flight_planner.participant_id} when attempting to submit a flight intent (flight ID: {flight_id})", - severity=Severity.High, details=f"{str(e)}\n\nStack trace:\n{e.stacktrace}", query_timestamps=[q.request.timestamp for q in e.queries], ) @@ -450,11 +445,7 @@ def submit_flight( notes_suffix = f': "{resp.notes}"' if "notes" in resp and resp.notes else "" for unexpected_result, failed_test_check in failed_checks.items(): - if isinstance(failed_test_check, str): - check_name = failed_test_check - check_severity = Severity.High - else: - check_name, check_severity = failed_test_check + check_name = failed_test_check with scenario.check( check_name, [flight_planner.participant_id] @@ -462,18 +453,15 @@ def submit_flight( if resp.activity_result == unexpected_result: specific_failed_check.record_failed( summary=f"Flight unexpectedly {resp.activity_result}", - severity=check_severity, details=f'{flight_planner.participant_id} indicated {resp.activity_result} rather than the expected {" or ".join(r[0] for r in expected_results)}{notes_suffix}', query_timestamps=[query.request.timestamp], ) if (resp.activity_result, resp.flight_plan_status) in expected_results: - scenario.end_test_step() return resp, flight_id else: check.record_failed( summary=f"Flight planning activity outcome was not expected", - severity=Severity.High, details=f'{flight_planner.participant_id} indicated {resp.activity_result} with flight plan status {resp.flight_plan_status} rather than the expected {" or ".join([f"({expected_result[0]}, {expected_result[1]})" for expected_result in expected_results])}{notes_suffix}', query_timestamps=[query.request.timestamp], ) @@ -534,7 +522,6 @@ def cleanup_flight( def delete_flight( scenario: TestScenarioType, - test_step: str, flight_planner: FlightPlannerClient, flight_id: str, ) -> PlanningActivityResponse: @@ -545,7 +532,6 @@ def delete_flight( Returns: The deletion response. """ - scenario.begin_test_step(test_step) with scenario.check( "Successful deletion", [flight_planner.participant_id] ) as check: @@ -567,7 +553,6 @@ def delete_flight( resp.activity_result == PlanningActivityResult.Completed and resp.flight_plan_status == FlightPlanStatus.Closed ): - scenario.end_test_step() return resp else: check.record_failed( diff --git a/monitoring/uss_qualifier/suites/astm/utm/f3548_21.yaml b/monitoring/uss_qualifier/suites/astm/utm/f3548_21.yaml index 8e0f38809e..0d4e3b7299 100644 --- a/monitoring/uss_qualifier/suites/astm/utm/f3548_21.yaml +++ b/monitoring/uss_qualifier/suites/astm/utm/f3548_21.yaml @@ -124,14 +124,14 @@ actions: test_scenario: scenario_type: scenarios.astm.utm.data_exchange_validation.GetOpResponseDataValidationByUSS resources: - tested_uss: uss1 - control_uss: mock_uss + tested_uss: tested_uss + mock_uss: mock_uss dss: dss flight_intents: non_conflicting_flights on_failure: Continue flight_planners_source: flight_planners roles: - - uss1 + - tested_uss on_failure: Continue - action_generator: generator_type: action_generators.flight_planning.FlightPlannerCombinations