diff --git a/monitoring/uss_qualifier/requirements/astm/f3411/v22a.md b/monitoring/uss_qualifier/requirements/astm/f3411/v22a.md index e8c500b24a..1c4319d4a7 100644 --- a/monitoring/uss_qualifier/requirements/astm/f3411/v22a.md +++ b/monitoring/uss_qualifier/requirements/astm/f3411/v22a.md @@ -81,6 +81,44 @@ For information on these requirements, refer to [the ASTM standard F3411-22a](ht * NET0430 * NET0440 * NET0450 +* NET0450,Table1,1: UAS ID, any option + * NET0450,Table1,1a: UAS ID, option 1 - Serial Number + * NET0450,Table1,1b: UAS ID, option 2 - Registration ID + * NET0450,Table1,1c: UAS ID, option 3 - UTM (UUID) + * NET0450,Table1,1d: UAS ID, option 4 - Specific Session ID + * NET0450,Table1,2: UA Type + * NET0450,Table1,3: UA Classification + * NET0450,Table1,4: UA Classification Type + * NET0450,Table1,5: Timestamp + * NET0450,Table1,6: Timestamp Accuracy + * NET0450,Table1,7: Operational Status + * NET0450,Table1,8: Operation Description + * NET0450,Table1,9: Operator ID + * NET0450,Table1,10: Latitude + * NET0450,Table1,11: Longitude + * NET0450,Table1,12: Geodetic Altitude + * NET0450,Table1,13: Pressure Altitude + * NET0450,Table1,14: Height + * NET0450,Table1,15: Height type + * NET0450,Table1,16: Geodetic Vertical Accuracy + * NET0450,Table1,17: Horizontal Accuracy + * NET0450,Table1,18: Speed Accuracy + * NET0450,Table1,19: Track Direction + * NET0450,Table1,20: Speed + * NET0450,Table1,21: Vertical Speed + * NET0450,Table1,22: Auth Data + * NET0450,Table1,23: Operator Latitude + * NET0450,Table1,24: Operator Longitude + * NET0450,Table1,25: Operator Altitude + * NET0450,Table1,26: Operator Location Type + * NET0450,Table1,27: Operating Area Radius + * NET0450,Table1,28: Operating Area Polygon + * NET0450,Table1,29: Operating Area Type + * NET0450,Table1,30: Operating Area Count + * NET0450,Table1,31: Operating Area Floor + * NET0450,Table1,32: Operating Area Ceiling + * NET0450,Table1,33: Operating Area Start + * NET0450,Table1,34: Operating Area End * NET0460 * NET0470 * NET0470,Table1,1: UAS ID, any option diff --git a/monitoring/uss_qualifier/requirements/astm/f3411/v22a/display_provider.md b/monitoring/uss_qualifier/requirements/astm/f3411/v22a/display_provider.md index ecbbfe4c2b..a6544af3e5 100644 --- a/monitoring/uss_qualifier/requirements/astm/f3411/v22a/display_provider.md +++ b/monitoring/uss_qualifier/requirements/astm/f3411/v22a/display_provider.md @@ -49,141 +49,180 @@ This file describes the set of ASTM F3411-22a requirements with which a USS fulf #### UAS ID transmitter + * **astm.f3411.v22a.NET0450,Table1,1** * **astm.f3411.v22a.NET0470,Table1,1** #### UAS ID Serial Number transmitter + * **astm.f3411.v22a.NET0450,Table1,1a** * **astm.f3411.v22a.NET0470,Table1,1a** #### UAS ID Registration ID transmitter + * **astm.f3411.v22a.NET0450,Table1,1b** * **astm.f3411.v22a.NET0470,Table1,1b** #### UAS ID UTM (UUID) transmitter + * **astm.f3411.v22a.NET0450,Table1,1c** * **astm.f3411.v22a.NET0470,Table1,1c** #### UAS ID Specific Session ID transmitter + * **astm.f3411.v22a.NET0450,Table1,1d** * **astm.f3411.v22a.NET0470,Table1,1d** #### UA Type transmitter + * **astm.f3411.v22a.NET0450,Table1,2** * **astm.f3411.v22a.NET0470,Table1,2** #### UA Classification transmitter + * **astm.f3411.v22a.NET0450,Table1,3** * **astm.f3411.v22a.NET0470,Table1,3** #### UA Classification Type transmitter + * **astm.f3411.v22a.NET0450,Table1,4** * **astm.f3411.v22a.NET0470,Table1,4** #### Timestamp transmitter + * **astm.f3411.v22a.NET0450,Table1,5** * **astm.f3411.v22a.NET0470,Table1,5** #### Timestamp Accuracy transmitter + * **astm.f3411.v22a.NET0450,Table1,6** * **astm.f3411.v22a.NET0470,Table1,6** #### Operational Status transmitter + * **astm.f3411.v22a.NET0450,Table1,7** * **astm.f3411.v22a.NET0470,Table1,7** #### Operational Description transmitter + * **astm.f3411.v22a.NET0450,Table1,8** * **astm.f3411.v22a.NET0470,Table1,8** #### Operator ID transmitter + * **astm.f3411.v22a.NET0450,Table1,9** * **astm.f3411.v22a.NET0470,Table1,9** #### Current Position transmitter + * **astm.f3411.v22a.NET0450,Table1,10** + * **astm.f3411.v22a.NET0450,Table1,11** * **astm.f3411.v22a.NET0470,Table1,10** * **astm.f3411.v22a.NET0470,Table1,11** #### Geodetic Altitude transmitter + * **astm.f3411.v22a.NET0450,Table1,12** * **astm.f3411.v22a.NET0470,Table1,12** #### Pressure Altitude transmitter + * **astm.f3411.v22a.NET0450,Table1,13** * **astm.f3411.v22a.NET0470,Table1,13** #### Height transmitter + * **astm.f3411.v22a.NET0450,Table1,14** + * **astm.f3411.v22a.NET0450,Table1,15** * **astm.f3411.v22a.NET0470,Table1,14** * **astm.f3411.v22a.NET0470,Table1,15** #### Geodetic Vertical Accuracy transmitter + * **astm.f3411.v22a.NET0450,Table1,16** * **astm.f3411.v22a.NET0470,Table1,16** #### Horizontal Accuracy transmitter + * **astm.f3411.v22a.NET0450,Table1,17** * **astm.f3411.v22a.NET0470,Table1,17** #### Speed Accuracy transmitter + * **astm.f3411.v22a.NET0450,Table1,18** * **astm.f3411.v22a.NET0470,Table1,18** #### Track Direction transmitter + * **astm.f3411.v22a.NET0450,Table1,19** * **astm.f3411.v22a.NET0470,Table1,19** #### Speed transmitter + * **astm.f3411.v22a.NET0450,Table1,20** * **astm.f3411.v22a.NET0470,Table1,20** #### Vertical Speed transmitter + * **astm.f3411.v22a.NET0450,Table1,21** * **astm.f3411.v22a.NET0470,Table1,21** #### Auth Data transmitter + * **astm.f3411.v22a.NET0450,Table1,22** * **astm.f3411.v22a.NET0470,Table1,22** #### Operator Position transmitter + * **astm.f3411.v22a.NET0450,Table1,23** + * **astm.f3411.v22a.NET0450,Table1,24** * **astm.f3411.v22a.NET0470,Table1,23** * **astm.f3411.v22a.NET0470,Table1,24** #### Operator Altitude transmitter + * **astm.f3411.v22a.NET0450,Table1,25** * **astm.f3411.v22a.NET0470,Table1,25** #### Operator Location Type transmitter + * **astm.f3411.v22a.NET0450,Table1,26** * **astm.f3411.v22a.NET0470,Table1,26** #### Operating Area Radius transmitter + * **astm.f3411.v22a.NET0450,Table1,27** + * **astm.f3411.v22a.NET0450,Table1,29** * **astm.f3411.v22a.NET0470,Table1,27** * **astm.f3411.v22a.NET0470,Table1,29** #### Operating Area Polygon transmitter + * **astm.f3411.v22a.NET0450,Table1,28** + * **astm.f3411.v22a.NET0450,Table1,29** * **astm.f3411.v22a.NET0470,Table1,28** * **astm.f3411.v22a.NET0470,Table1,29** #### Operating Area Count transmitter + * **astm.f3411.v22a.NET0450,Table1,30** * **astm.f3411.v22a.NET0470,Table1,30** #### Operating Area Floor transmitter + * **astm.f3411.v22a.NET0450,Table1,31** * **astm.f3411.v22a.NET0470,Table1,31** #### Operating Area Ceiling transmitter + * **astm.f3411.v22a.NET0450,Table1,32** * **astm.f3411.v22a.NET0470,Table1,32** #### Operating Area Start transmitter + * **astm.f3411.v22a.NET0450,Table1,33** * **astm.f3411.v22a.NET0470,Table1,33** #### Operating Area End transmitter + * **astm.f3411.v22a.NET0450,Table1,34** * **astm.f3411.v22a.NET0470,Table1,34** diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.py b/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.py index 7fc9a69c06..228d4f1659 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.py +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.py @@ -3,6 +3,12 @@ from implicitdict import StringBasedDateTime from typing import List, Optional import s2sphere +from uas_standards.interuss.automated_testing.rid.v1.injection import ( + TestFlightDetails, + RIDFlightDetails, + TestFlight, + RIDAircraftState, +) from uas_standards.interuss.automated_testing.rid.v1.observation import ( GetDetailsResponse, @@ -84,12 +90,14 @@ def evaluate_sp_flights( # Evaluate on all flights regardless of where they came from self._evaluate_operational_status( f.operational_status, + None, participants, ) def evaluate_dp_flight( self, observed_flight: Flight, + injected_telemetry: RIDAircraftState, participants: List[str], ): current_state = observed_flight.current_state @@ -97,7 +105,9 @@ def evaluate_dp_flight( self._evaluate_track(current_state.track, participants) self._evaluate_timestamp(current_state.timestamp, participants) self._evaluate_operational_status( - current_state.operational_status, participants + current_state.operational_status, + injected_telemetry.operational_status, + participants, ) self._evaluate_position(observed_flight.most_recent_position, participants) self._evaluate_height( @@ -194,8 +204,20 @@ def fail_check(): fail_check() def evaluate_sp_details(self, details: FlightDetails, participants: List[str]): + # branch NET0450-match-injected-data + # > fields from uspace / DP dans network_identification: only from DP, not from SP + # - display_provider_operator_id_transmitter # Operator Registration -> OK + # - display_provider_uas_id_serial_number_transmitter # Serial Number -> TODO just like operator ID (flight details) + # - display_provider_current_position_transmitter # UA Position -> TODO implementation may already exists + # - display_provider_height_transmitter # Height -> TODO: not injected, depend on https://github.com/interuss/uas_standards/pull/12, and on https://github.com/interuss/monitoring/pull/150 + # - display_provider_timestamp_transmitter # Timestamp -> TODO: just like operator status, get it from current state + # - display_provider_track_direction_transmitter # Track -> TODO: just like operator status (current state) + # - display_provider_speed_transmitter # Speed -> TODO: just like operator status (current state) + # - display_provider_operator_position_transmitter # Operator Location -> TODO just like operator ID (flight details) + # - display_provider_operational_status_transmitter # Operational Status -> OK + # TODO: check for freshness of data (if not implemented) self._evaluate_uas_id(details.raw.get("uas_id"), participants) - self._evaluate_operator_id(details.operator_id, participants) + self._evaluate_operator_id(details.operator_id, None, participants) self._evaluate_operator_location( details.operator_location.position, details.operator_location.get("altitude"), @@ -204,7 +226,10 @@ def evaluate_sp_details(self, details: FlightDetails, participants: List[str]): ) def evaluate_dp_details( - self, observed_details: Optional[GetDetailsResponse], participants: List[str] + self, + observed_details: Optional[GetDetailsResponse], + injected_details: Optional[RIDFlightDetails], + participants: List[str], ): if not observed_details: return @@ -214,7 +239,9 @@ def evaluate_dp_details( ) operator = observed_details.get("operator", {}) - self._evaluate_operator_id(operator.get("id"), participants) + self._evaluate_operator_id( + operator.get("id"), injected_details.get("operator_id"), participants + ) operator_location = operator.get("location", {}) operator_altitude = operator.get("altitude", {}) @@ -334,8 +361,23 @@ def _evaluate_timestamp(self, timestamp: str, participants: List[str]): message=f"Unsupported version {self._rid_version}: skipping timestamp evaluation", ) - def _evaluate_operator_id(self, value: Optional[str], participants: List[str]): + def _evaluate_operator_id( + self, + value: Optional[str], + injected_value: Optional[str], + participants: List[str], + ): if self._rid_version == RIDVersion.f3411_22a: + if injected_value: + with self._test_scenario.check( + "Correct up-to-date Operator ID", participants + ) as check: + if value != injected_value: + check.record_failed( + "Observed Operator ID do not match injected value", + details=f"{injected_value} (injected) != {value} (observed)", + severity=Severity.Medium, + ) if value: with self._test_scenario.check( "Operator ID consistency with Common Dictionary", participants @@ -346,6 +388,7 @@ def _evaluate_operator_id(self, value: Optional[str], participants: List[str]): "Operator ID contains non-ascii characters", severity=Severity.Medium, ) + else: self._test_scenario.record_note( key="skip_reason", @@ -551,9 +594,22 @@ def _evaluate_operator_location( ) def _evaluate_operational_status( - self, value: Optional[str], participants: List[str] + self, + value: Optional[str], + injected_value: Optional[str], + participants: List[str], ): if self._rid_version == RIDVersion.f3411_22a: + if injected_value: + with self._test_scenario.check( + "Correct up-to-date Operational Status", participants + ) as check: + if value != injected_value: + check.record_failed( + "Observed Operational Status do not match injected value", + details=f"{injected_value} (injected) != {value} (observed)", + severity=Severity.Medium, + ) if value: with self._test_scenario.check( "Operational Status consistency with Common Dictionary", diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator_test.py b/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator_test.py index c451041bb3..46ca985e39 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator_test.py +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator_test.py @@ -190,7 +190,7 @@ def step_under_test(self: UnitTestScenario): rid_version=RIDVersion.f3411_22a, ) - evaluator._evaluate_operational_status(value, []) + evaluator._evaluate_operational_status(value, None, []) unit_test_scenario = UnitTestScenario(step_under_test).execute_unit_test() assert unit_test_scenario.get_report().successful == outcome diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/display_data_evaluator.py b/monitoring/uss_qualifier/scenarios/astm/netrid/display_data_evaluator.py index ebe77ceab8..49cfa288c9 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/display_data_evaluator.py +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/display_data_evaluator.py @@ -18,7 +18,10 @@ Position, ) from monitoring.uss_qualifier.resources.astm.f3411.dss import DSSInstance -from uas_standards.interuss.automated_testing.rid.v1.injection import RIDAircraftState +from uas_standards.interuss.automated_testing.rid.v1.injection import ( + RIDAircraftState, + TestFlightDetails, +) from uas_standards.interuss.automated_testing.rid.v1.observation import ( Flight, GetDisplayDataResponse, @@ -181,6 +184,14 @@ def _make_flight_mapping( return mapping +def get_effective_details( + injected_details: List[TestFlightDetails], reference_time: str +): + for i in injected_details: + if arrow.get(i.effective_after) < arrow.get(reference_time): + return i + + class RIDObservationEvaluator(object): """Evaluates observations of an RID system over time. @@ -384,7 +395,12 @@ def _evaluate_normal_observation( ) self._common_dictionary_evaluator.evaluate_dp_flight( - mapping.observed_flight, [observer.participant_id] + observed_flight=mapping.observed_flight, + injected_telemetry=injected_telemetry, + participants=[ + observer.participant_id, + mapping.injected_flight.uss_participant_id, + ], ) # Check that flights using telemetry are not using extrapolated position data for mapping in mapping_by_injection_id.values(): @@ -428,11 +444,22 @@ def _evaluate_normal_observation( mapping.observed_flight.id, self._rid_version ) self._test_scenario.record_query(query) + if not details: + self._test_scenario.record_note( + f"Unable to retrieve details from {observer.participant_id}" + ) + return + ref = query.request.initiated_at + injected_details = get_effective_details( + mapping.injected_flight.flight.details_responses, ref + ) self._common_dictionary_evaluator.evaluate_dp_details( details, + injected_details.details, participants=[ observer.participant_id, + mapping.injected_flight.uss_participant_id, ], ) diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/nominal_behavior.md b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/nominal_behavior.md index 7afadc53aa..0b2aa85b0e 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/nominal_behavior.md +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/nominal_behavior.md @@ -204,10 +204,18 @@ This check validates that the display area of a cluster, measured and provided i **[astm.f3411.v22a.NET0470](../../../../requirements/astm/f3411/v22a.md)** requires that Net-RID Display Provider shall provide access to required and optional fields to Remote ID Display Applications according to the Common Dictionary. This check validates that timestamps are expressed with a minimum resolution of one tenth of a second and relative to UTC. (**[astm.f3411.v22a.NET0470,Table1,5](../../../../requirements/astm/f3411/v22a.md)**) +#### Correct up-to-date Operational Status check + +**[astm.f3411.v22a.NET0450](../../../../requirements/astm/f3411/v22a.md)** requires that Net-RID Display Provider shall provide the most recent data available to Remote ID Display Applications. If the observed Operational Status do not match the injected value, this check will fail. (**[astm.f3411.v22a.NET0450,Table1,7](../../../../requirements/astm/f3411/v22a.md)**) + #### Operational Status consistency with Common Dictionary check **[astm.f3411.v22a.NET0470](../../../../requirements/astm/f3411/v22a.md)** requires that Net-RID Display Provider shall provide access to required and optional fields to Remote ID Display Applications according to the Common Dictionary. This check validates that the Operational Status, if present, is valid. (**[astm.f3411.v22a.NET0470,Table1,7](../../../../requirements/astm/f3411/v22a.md)**) +#### Correct up-to-date Operator ID check + +**[astm.f3411.v22a.NET0450](../../../../requirements/astm/f3411/v22a.md)** requires that Net-RID Display Provider shall provide the most recent data available to Remote ID Display Applications. If the observed Operator ID do not match the injected value, this check will fail. (**[astm.f3411.v22a.NET0450,Table1,9](../../../../requirements/astm/f3411/v22a.md)**) + #### Operator ID consistency with Common Dictionary check **[astm.f3411.v22a.NET0470](../../../../requirements/astm/f3411/v22a.md)** requires that Net-RID Display Provider shall (NET0470) provide access to required and optional fields to Remote ID Display Applications according to the Common Dictionary. This check validates that the Operator ID, if present, is valid. (**[astm.f3411.v22a.NET0470,Table1,9](../../../../requirements/astm/f3411/v22a.md)**) diff --git a/monitoring/uss_qualifier/suites/uspace/network_identification.md b/monitoring/uss_qualifier/suites/uspace/network_identification.md index 3871b57a6f..1d8f476587 100644 --- a/monitoring/uss_qualifier/suites/uspace/network_identification.md +++ b/monitoring/uss_qualifier/suites/uspace/network_identification.md @@ -22,7 +22,7 @@ ASTM F3411-22a NetRID aggregate checks - astm
.f3411
.v22a
+ astm
.f3411
.v22a
DSS0030 Implemented ASTM NetRID nominal behavior @@ -107,6 +107,21 @@ Implemented ASTM F3411-22a NetRID aggregate checks + + NET0450 + Implemented + ASTM NetRID nominal behavior + + + NET0450,Table1,7 + Implemented + ASTM NetRID nominal behavior + + + NET0450,Table1,9 + Implemented + ASTM NetRID nominal behavior + NET0470 In progress diff --git a/monitoring/uss_qualifier/suites/uspace/required_services.md b/monitoring/uss_qualifier/suites/uspace/required_services.md index e3ba2e4225..af6d499781 100644 --- a/monitoring/uss_qualifier/suites/uspace/required_services.md +++ b/monitoring/uss_qualifier/suites/uspace/required_services.md @@ -23,7 +23,7 @@ ASTM F3411-22a NetRID aggregate checks - astm
.f3411
.v22a
+ astm
.f3411
.v22a
DSS0030 Implemented ASTM NetRID nominal behavior @@ -108,6 +108,21 @@ Implemented ASTM F3411-22a NetRID aggregate checks + + NET0450 + Implemented + ASTM NetRID nominal behavior + + + NET0450,Table1,7 + Implemented + ASTM NetRID nominal behavior + + + NET0450,Table1,9 + Implemented + ASTM NetRID nominal behavior + NET0470 In progress