Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[uss_qualifier] nominal_behavior – align requirements on observations with the standard #374

Merged
merged 1 commit into from
Nov 29, 2023
Merged

[uss_qualifier] nominal_behavior – align requirements on observations with the standard #374

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 19 additions & 27 deletions monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
import s2sphere
from arrow import ParserError
from implicitdict import StringBasedDateTime
from loguru import logger
from uas_standards.ansi_cta_2063_a import SerialNumber
from uas_standards.astm.f3411 import v22a
from uas_standards.astm.f3411.v22a.api import UASID
Expand Down Expand Up @@ -88,34 +87,27 @@ def evaluate_dp_flight(
observed_flight: observation_api.Flight,
participants: List[str],
):
with self._test_scenario.check("Current state present", participants) as check:
if not observed_flight.has_field_with_value("current_state"):
check.record_failed(
f"Current state for flight {observed_flight.id}",
details=f"The current state must be specified.",
severity=Severity.High,
)

logger.debug(f"Injected flight: {injected_flight}")
logger.debug(f"Observed flight: {observed_flight.current_state}")
# If the state is present, we do validate its content,
# but its presence is optional
if injected_flight.has_field_with_value("current_state"):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@BenjaminPelletier with this change (and the fact that the check was removed from the doc) we assume that there is no need to record specifically that the field is present (I was wondering if this would be required as part of a capability, but I've never set one up myself so am quite unsure)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capabilities are built on requirements, so if there's no requirement for a field to be present, being present shouldn't be part of the capability. But, when the field is present, there is a requirement that it satisfies some conditions, and verification of that requirement (which will only happen when the field is present) is what the capability would be based on.

self._evaluate_speed(
injected_flight.speed, observed_flight.current_state.speed, participants
)
self._evaluate_track(
injected_flight.track, observed_flight.current_state.track, participants
)
self._evaluate_timestamp(
injected_flight.timestamp,
observed_flight.current_state.timestamp,
participants,
)

self._evaluate_speed(
injected_flight.speed, observed_flight.current_state.speed, participants
)
self._evaluate_track(
injected_flight.track, observed_flight.current_state.track, participants
)
self._evaluate_timestamp(
injected_flight.timestamp,
observed_flight.current_state.timestamp,
participants,
)
# TODO check if worth adding correctness check here, it requires some slight (possibly non-trivial)
# changes in evaluate_sp_flights as well
self._evaluate_operational_status(
observed_flight.current_state.operational_status, participants
)

# TODO check if worth adding correctness check here, it requires some slight (possibly non-trivial)
# changes in evaluate_sp_flights as well
self._evaluate_operational_status(
observed_flight.current_state.operational_status, participants
)
self._evaluate_position(
injected_flight.position, observed_flight.most_recent_position, participants
)
Expand Down
12 changes: 1 addition & 11 deletions monitoring/uss_qualifier/scenarios/astm/netrid/display_data_evaluator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,19 +358,9 @@ def _evaluate_normal_observation(
observed_position = mapping.observed_flight.most_recent_position
injected_position = injected_telemetry.position

with self._test_scenario.check(
"Altitude is present", [observer.participant_id]
) as check:
if "alt" not in observed_position:
check.record_failed(
summary="Displayed flight is missing altitude",
severity=Severity.Medium,
details=f"Displayed data for injected flight {mapping.injected_flight.flight.injection_id} in test {mapping.injected_flight.test_id} at {injected_telemetry.timestamp} does not have altitude",
)

if "alt" in observed_position:
with self._test_scenario.check(
"Correct up-to-date altitude",
"Correct up-to-date altitude if present",
[
observer.participant_id,
mapping.injected_flight.uss_participant_id,
Expand Down
12 changes: 2 additions & 10 deletions monitoring/uss_qualifier/scenarios/astm/netrid/v19/nominal_behavior.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,13 +132,9 @@ The timestamps of the injected telemetry usually start in the future. If a flig

**[astm.f3411.v19.NET0290](../../../../requirements/astm/f3411/v19.md)** requires a SP uses Telemetry vs extrapolation when telemetry is present.

#### Altitude is present check
#### Correct up-to-date altitude if present check

Failure to report the altitude of a flight is a violation of **[astm.f3411.v19.NET0470](../../../../requirements/astm/f3411/v19.md)**.

#### Correct up-to-date altitude check

If the observed altitude of a flight does not match the altitude of the injected telemetry, the display provider is not providing precise and up-to-date information, and thus does not respect **[astm.f3411.v19.NET0450](../../../../requirements/astm/f3411/v19.md)**.
If the observed altitude of a flight is reported, but it does not match the altitude of the injected telemetry, the display provider is not providing precise and up-to-date information, and thus does not respect **[astm.f3411.v19.NET0450](../../../../requirements/astm/f3411/v19.md)**.

#### Area too large check

Expand Down Expand Up @@ -174,10 +170,6 @@ This check validates that the display area of a cluster, measured and provided i

Per **[interuss.automated_testing.rid.observation.ObservationSuccess](../../../../requirements/interuss/automated_testing/rid/observation.md)**, the call for flight details is expected to succeed since a valid ID was provided by uss_qualifier.

#### Current state present check

**[astm.f3411.v19.NET0470](../../../../requirements/astm/f3411/v19.md)** requires that Net-RID Display Provider shall provide access to required and optional fields to Remote ID Display Applications according to the Common Dictionary. This check validates that the current state is present. If it is not, this check will fail.

## Cleanup

The cleanup phase of this test scenario attempts to remove injected data from all SPs.
Expand Down
12 changes: 2 additions & 10 deletions monitoring/uss_qualifier/scenarios/astm/netrid/v22a/nominal_behavior.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,13 +164,9 @@ NET0260 requires a SP to provide flights up to *NetMaxNearRealTimeDataPeriod* in

**[astm.f3411.v22a.NET0290](../../../../requirements/astm/f3411/v22a.md)** requires a SP uses Telemetry vs extrapolation when telemetry is present.

#### Altitude is present check
#### Correct up-to-date altitude if present check

Per **[astm.f3411.v22a.NET0470](../../../../requirements/astm/f3411/v22a.md)**, failure to report the altitude of a flight is a violation of **[astm.f3411.v22a.NET0470,Table1,12](../../../../requirements/astm/f3411/v22a.md)**.

#### Correct up-to-date altitude check

If the observed altitude of a flight does not match the altitude of the injected telemetry, the display provider is not providing precise and up-to-date information, and thus does not respect **[astm.f3411.v22a.NET0450](../../../../requirements/astm/f3411/v22a.md)**.
If the observed altitude of a flight is reported, but it does not match the altitude of the injected telemetry, the display provider is not providing precise and up-to-date information, and thus does not respect **[astm.f3411.v22a.NET0450](../../../../requirements/astm/f3411/v22a.md)**.

#### Area too large check

Expand Down Expand Up @@ -206,10 +202,6 @@ This check validates that the display area of a cluster, measured and provided i

Per **[interuss.automated_testing.rid.observation.ObservationSuccess](../../../../requirements/interuss/automated_testing/rid/observation.md)**, the call for flight details is expected to succeed since a valid ID was provided by uss_qualifier.

#### Current state present check

**[astm.f3411.v22a.NET0470](../../../../requirements/astm/f3411/v22a.md)** requires that Net-RID Display Provider shall provide access to required and optional fields to Remote ID Display Applications according to the Common Dictionary. This check validates that the current state is present. If it is not, this check will fail.

#### UAS ID presence in flight details check

**[astm.f3411.v22a.NET0470](../../../../requirements/astm/f3411/v22a.md)** requires that Net-RID Display Provider shall provide access to required and optional fields to Remote ID Display Applications according to the Common Dictionary. This check validates that the UAS ID is present in the information sent by the Display Provider. (**[astm.f3411.v22a.NET0470,Table1,1](../../../../requirements/astm/f3411/v22a.md)**)
Expand Down
7 changes: 1 addition & 6 deletions monitoring/uss_qualifier/suites/astm/netrid/f3411_19.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<th><a href="../../README.md#checked-in">Checked in</a></th>
</tr>
<tr>
<td rowspan="62" style="vertical-align:top;"><a href="../../../requirements/astm/f3411/v19.md">astm<br>.f3411<br>.v19</a></td>
<td rowspan="61" style="vertical-align:top;"><a href="../../../requirements/astm/f3411/v19.md">astm<br>.f3411<br>.v19</a></td>
<td><a href="../../../requirements/astm/f3411/v19.md">DSS0030,a</a></td>
<td>Implemented</td>
<td><a href="../../../scenarios/astm/netrid/v19/dss_interoperability.md">ASTM F3411-19 NetRID DSS interoperability</a><br><a href="../../../scenarios/astm/netrid/v19/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../../scenarios/astm/netrid/v19/dss/isa_subscription_interactions.md">ASTM NetRID DSS: ISA Subscription Interactions</a><br><a href="../../../scenarios/astm/netrid/v19/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../../scenarios/astm/netrid/v19/dss/isa_validation.md">ASTM NetRID DSS: Submitted ISA Validations</a><br><a href="../../../scenarios/astm/netrid/v19/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a></td>
Expand Down Expand Up @@ -286,11 +286,6 @@
<td>Implemented</td>
<td><a href="../../../scenarios/astm/netrid/v19/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
</tr>
<tr>
<td><a href="../../../requirements/astm/f3411/v19.md">NET0470</a></td>
<td>Implemented</td>
<td><a href="../../../scenarios/astm/netrid/v19/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
</tr>
<tr>
<td><a href="../../../requirements/astm/f3411/v19.md">NET0480</a></td>
<td>Implemented</td>
Expand Down
9 changes: 2 additions & 7 deletions monitoring/uss_qualifier/suites/astm/netrid/f3411_22a.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<th><a href="../../README.md#checked-in">Checked in</a></th>
</tr>
<tr>
<td rowspan="87" style="vertical-align:top;"><a href="../../../requirements/astm/f3411/v22a.md">astm<br>.f3411<br>.v22a</a></td>
<td rowspan="86" style="vertical-align:top;"><a href="../../../requirements/astm/f3411/v22a.md">astm<br>.f3411<br>.v22a</a></td>
<td><a href="../../../requirements/astm/f3411/v22a.md">DSS0030</a></td>
<td>Implemented</td>
<td><a href="../../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a></td>
Expand Down Expand Up @@ -356,11 +356,6 @@
<td>TODO</td>
<td><a href="../../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
</tr>
<tr>
<td><a href="../../../requirements/astm/f3411/v22a.md">NET0470,Table1,12</a></td>
<td>Implemented</td>
<td><a href="../../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
</tr>
<tr>
<td><a href="../../../requirements/astm/f3411/v22a.md">NET0470,Table1,15</a></td>
<td>Implemented</td>
Expand Down Expand Up @@ -497,6 +492,6 @@
<tr>
<td><a href="../../../requirements/interuss/f3411/dss_endpoints.md">SearchISAs</a></td>
<td>Implemented</td>
<td><a href="../../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../../scenarios/astm/netrid/v19/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a><br><a href="../../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
<td><a href="../../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../../scenarios/astm/netrid/v22a/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a><br><a href="../../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
</tr>
</table>
2 changes: 1 addition & 1 deletion monitoring/uss_qualifier/suites/astm/netrid/f3411_22a/dss_probing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,6 @@
<tr>
<td><a href="../../../../requirements/interuss/f3411/dss_endpoints.md">SearchISAs</a></td>
<td>Implemented</td>
<td><a href="../../../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../../../scenarios/astm/netrid/v19/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a></td>
<td><a href="../../../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../../../scenarios/astm/netrid/v22a/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a></td>
</tr>
</table>
2 changes: 1 addition & 1 deletion monitoring/uss_qualifier/suites/interuss/dss/all_tests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,6 @@
<tr>
<td><a href="../../../requirements/interuss/f3411/dss_endpoints.md">SearchISAs</a></td>
<td>Implemented</td>
<td><a href="../../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../../scenarios/astm/netrid/v19/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a></td>
<td><a href="../../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../../scenarios/astm/netrid/v22a/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a></td>
</tr>
</table>
9 changes: 2 additions & 7 deletions monitoring/uss_qualifier/suites/uspace/network_identification.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<th><a href="../README.md#checked-in">Checked in</a></th>
</tr>
<tr>
<td rowspan="87" style="vertical-align:top;"><a href="../../requirements/astm/f3411/v22a.md">astm<br>.f3411<br>.v22a</a></td>
<td rowspan="86" style="vertical-align:top;"><a href="../../requirements/astm/f3411/v22a.md">astm<br>.f3411<br>.v22a</a></td>
<td><a href="../../requirements/astm/f3411/v22a.md">DSS0030</a></td>
<td>Implemented</td>
<td><a href="../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a></td>
Expand Down Expand Up @@ -351,11 +351,6 @@
<td>TODO</td>
<td><a href="../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
</tr>
<tr>
<td><a href="../../requirements/astm/f3411/v22a.md">NET0470,Table1,12</a></td>
<td>Implemented</td>
<td><a href="../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
</tr>
<tr>
<td><a href="../../requirements/astm/f3411/v22a.md">NET0470,Table1,15</a></td>
<td>Implemented</td>
Expand Down Expand Up @@ -492,6 +487,6 @@
<tr>
<td><a href="../../requirements/interuss/f3411/dss_endpoints.md">SearchISAs</a></td>
<td>Implemented</td>
<td><a href="../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../scenarios/astm/netrid/v19/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a><br><a href="../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
<td><a href="../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../scenarios/astm/netrid/v22a/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a><br><a href="../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
</tr>
</table>
9 changes: 2 additions & 7 deletions monitoring/uss_qualifier/suites/uspace/required_services.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
<th><a href="../README.md#checked-in">Checked in</a></th>
</tr>
<tr>
<td rowspan="87" style="vertical-align:top;"><a href="../../requirements/astm/f3411/v22a.md">astm<br>.f3411<br>.v22a</a></td>
<td rowspan="86" style="vertical-align:top;"><a href="../../requirements/astm/f3411/v22a.md">astm<br>.f3411<br>.v22a</a></td>
<td><a href="../../requirements/astm/f3411/v22a.md">DSS0030</a></td>
<td>Implemented</td>
<td><a href="../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a></td>
Expand Down Expand Up @@ -353,11 +353,6 @@
<td>TODO</td>
<td><a href="../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
</tr>
<tr>
<td><a href="../../requirements/astm/f3411/v22a.md">NET0470,Table1,12</a></td>
<td>Implemented</td>
<td><a href="../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
</tr>
<tr>
<td><a href="../../requirements/astm/f3411/v22a.md">NET0470,Table1,15</a></td>
<td>Implemented</td>
Expand Down Expand Up @@ -631,7 +626,7 @@
<tr>
<td><a href="../../requirements/interuss/f3411/dss_endpoints.md">SearchISAs</a></td>
<td>Implemented</td>
<td><a href="../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../scenarios/astm/netrid/v19/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a><br><a href="../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
<td><a href="../../scenarios/astm/netrid/v22a/dss/isa_expiry.md">ASTM NetRID DSS: ISA Expiry</a><br><a href="../../scenarios/astm/netrid/v22a/dss/isa_simple.md">ASTM NetRID DSS: Simple ISA</a><br><a href="../../scenarios/astm/netrid/v22a/dss/token_validation.md">ASTM NetRID DSS: Token Validation</a><br><a href="../../scenarios/astm/netrid/v22a/nominal_behavior.md">ASTM NetRID nominal behavior</a></td>
</tr>
<tr>
<td rowspan="1" style="vertical-align:top;"><a href="../../requirements/interuss/f3548/notification_requirements.md">interuss<br>.f3548<br>.notification_requirements</a></td>
Expand Down
Loading