From 5b51a767725a339375c95d411489602dfe6a64e0 Mon Sep 17 00:00:00 2001 From: Benjamin Pelletier Date: Tue, 12 Sep 2023 16:55:10 -0700 Subject: [PATCH] [uss_qualifier] Clean up NetRID docs (#199) * Clean up RID documentation * make format * Replace wbr tag with hopefully more GitHub-friendly option * Replace unicode breaking non-space with true space for GitHub * Force line breaks for package components to clean up table * Fix F3411-22a Misbehavior references --- .../interuss/automated_testing/rid.md | 10 -- .../automated_testing/rid/injection.md | 12 ++- .../automated_testing/rid/observation.md | 8 +- .../interuss/uss_qualifier/unit_test.md | 4 +- .../common/dss/subscription_validation.py | 101 +++++++++++------- .../netrid/v19/dss/subscription_validation.md | 72 ++++++++++--- .../astm/netrid/v19/dss_interoperability.md | 4 + .../astm/netrid/v19/nominal_behavior.md | 2 + .../scenarios/astm/netrid/v22a/__init__.py | 1 + .../astm/netrid/v22a/dss/isa_simple.md | 8 +- .../v22a/dss/subscription_validation.md | 72 ++++++++++--- .../astm/netrid/v22a/dss_interoperability.md | 4 + .../astm/netrid/v22a/nominal_behavior.md | 2 + .../uss_qualifier/scenarios/scenario.py | 10 +- .../suites/astm/netrid/f3411_19.md | 12 +-- .../astm/netrid/f3411_19/dss_probing.md | 2 +- .../suites/astm/netrid/f3411_22a.md | 31 +++--- .../suites/astm/netrid/f3411_22a.yaml | 2 +- .../astm/netrid/f3411_22a/dss_probing.md | 7 +- .../uss_qualifier/suites/astm/utm/f3548_21.md | 4 +- .../suites/documentation/documentation.py | 3 +- .../documentation/format_documentation.py | 4 +- .../suites/faa/uft/message_signing.md | 4 +- .../suites/uspace/flight_auth.md | 4 +- .../suites/uspace/network_identification.md | 29 +++-- .../suites/uspace/required_services.md | 33 +++--- 26 files changed, 299 insertions(+), 146 deletions(-) delete mode 100644 monitoring/uss_qualifier/requirements/interuss/automated_testing/rid.md diff --git a/monitoring/uss_qualifier/requirements/interuss/automated_testing/rid.md b/monitoring/uss_qualifier/requirements/interuss/automated_testing/rid.md deleted file mode 100644 index 0c5725a6f7..0000000000 --- a/monitoring/uss_qualifier/requirements/interuss/automated_testing/rid.md +++ /dev/null @@ -1,10 +0,0 @@ -# InterUSS RemoteID Testing Requirements - -## Overview - -TODO: Describe overall approach, link to injection & observation - -## Requirements - -TODO: Describe requirements - diff --git a/monitoring/uss_qualifier/requirements/interuss/automated_testing/rid/injection.md b/monitoring/uss_qualifier/requirements/interuss/automated_testing/rid/injection.md index c653a5a92d..f3f3590e2c 100644 --- a/monitoring/uss_qualifier/requirements/interuss/automated_testing/rid/injection.md +++ b/monitoring/uss_qualifier/requirements/interuss/automated_testing/rid/injection.md @@ -2,16 +2,24 @@ ## Overview -TODO: Link to API YAML and provide overview +In order to test remote ID Service Providers (USSs who ingest aircraft data from operators and provide it to relevant Display Providers), InterUSS requires any Service Provider under test to implement the [InterUSS remote ID automated testing interface](https://github.com/interuss/automated_testing_interfaces/tree/main/rid) (specifically, the [injection portion](https://github.com/interuss/automated_testing_interfaces/blob/main/rid/v1/injection.yaml)). This interface empowers uss_qualifier, as the test director, to instruct the USS to create a flight in a particular area with particular characteristics. This is analogous to a similar verbal instruction during a manual checkout (e.g., "USS X, start your flight in the yellow area"). ## Requirements -TODO: Describe requirements +In general, to be successfully tested by uss_qualifier for remote ID functionality, Service Provider USSs are expected to implement the injection API mentioned above and successfully respond to valid requests. ### UpsertTestSuccess +When a properly-authorized valid request to create/start/simulate a set of flights ("test"), the USS under test must execute this request (create/start/simulate the flights) and indicate success. + ### UpsertTestResult +The USS under test is allowed to modify most components of the requested flights including the telemetry and details responses, including the flight ID (though not the injection_id; that is the invariant ID by which uss_qualifier refers to an injected test flight). However, there are obviously limits on how much the injected flight may be modified and still serve the purposes of the test being conducted. For instance, a test whose purpose was to observe interactions between multiple providers' flights in the same area could not be accomplished if all but one USS created flights in a particular city in Europe while the last USS created their flight in the United States. The actually-injected flight returned by the USS under test in response to an injection request must be suitable for the test to be conducted. In practice, this means that USSs should modify the injection request as little as possible while still maintaining compatibility with their system. + ### ExpectedBehavior +The USS under test must treat injected flights as similarly to real flights as practical. The same pathways that accept real telemetry should be used to accept the injected telemetry. For instance, the standard pathway to accept real telemetry should generally not accept reports from the future, so if future telemetry of the injected flight is visible appreciably before the time it is supposed to have been virtually reported then the USS into which the flight was injected is not treating the injected flight the same way a normal flight would be treated. + ### DeleteTestSuccess + +In order to rapidly conduct sequences of automated tests, uss_qualifier (as test director) must be able to "clear the airspace" after the completion of a test so that the simulated/injected flights from this test run do not affect future test runs. A Service Provider USS must successfully cancel/land/remove/delete the flights in a specified test upon deletion request for that test and indicate success for the deletion request. diff --git a/monitoring/uss_qualifier/requirements/interuss/automated_testing/rid/observation.md b/monitoring/uss_qualifier/requirements/interuss/automated_testing/rid/observation.md index 2073090de5..1e9f4da48c 100644 --- a/monitoring/uss_qualifier/requirements/interuss/automated_testing/rid/observation.md +++ b/monitoring/uss_qualifier/requirements/interuss/automated_testing/rid/observation.md @@ -2,12 +2,16 @@ ## Overview -TODO: Link to API YAML and provide overview +In order to test remote ID Display Providers (USSs who obtain aircraft data from Service Providers in order to aggregate it for their end-user viewers), InterUSS requires any Display Provider under test to implement the [InterUSS remote ID automated testing interface](https://github.com/interuss/automated_testing_interfaces/tree/main/rid) (specifically, the [observation portion](https://github.com/interuss/automated_testing_interfaces/blob/main/rid/v1/observation.yaml)). This interface empowers uss_qualifier, as the test director, to ask the USS what flights would be visible on its Display Application(s). This is analogous to a similar verbal question during a manual checkout (e.g., "USS Y, what flights do you see in the yellow area?"). ## Requirements -TODO: Describe requirements +In general, to be successfully tested by uss_qualifier for remote ID functionality, Display Provider USSs are expected to implement the observation API mentioned above and successfully respond to valid requests. ### ObservationSuccess +Upon receipt of a properly-authorized valid request to report the flights visible in a particular area, the USS under test must respond with the content visible in that area. Even if the user/viewer would be shown an error or other off-nominal situation in the Display Application, it should still be possible to return a valid response to this question (e.g., in the case of the user being shown an error, the USS would indicate no flights and no cluster visible). + ### UniqueFlights + +While the identifiers for displayed flights are an implementation detail of the USS (it may use NetRID identifiers, or it may assign different identifiers), the user/viewer needs the capability to obtain details about a specific individual flight. For that reason, a USS must provide unique identifiers for each flight in a particular observation. diff --git a/monitoring/uss_qualifier/requirements/interuss/uss_qualifier/unit_test.md b/monitoring/uss_qualifier/requirements/interuss/uss_qualifier/unit_test.md index bd04f6f06b..8693e0a333 100644 --- a/monitoring/uss_qualifier/requirements/interuss/uss_qualifier/unit_test.md +++ b/monitoring/uss_qualifier/requirements/interuss/uss_qualifier/unit_test.md @@ -1,8 +1,8 @@ # USS Qualifier unit test requirements -This file is used to defines requirements to support unit testing. +This file is used to define requirements to support unit testing. ## Dummy requirements * REQ001 * REQ100 -* REQ101 \ No newline at end of file +* REQ101 diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/common/dss/subscription_validation.py b/monitoring/uss_qualifier/scenarios/astm/netrid/common/dss/subscription_validation.py index 6aa7f45c44..1c89883739 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/common/dss/subscription_validation.py +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/common/dss/subscription_validation.py @@ -59,7 +59,13 @@ def run(self): self._setup_case() - self._subscription_limitations_case() + self.begin_test_case("Subscription duration limitations") + self._create_too_long_subscription() + self.end_test_case() + + self.begin_test_case("Subscription quantity limitations") + self._create_too_many_subscriptions() + self.end_test_case() self.end_test_scenario() @@ -70,46 +76,33 @@ def _setup_case(self): self.end_test_case() - def _clean_any_sub(self, check: PendingCheck): - fetched = self._dss_wrapper.search_subs( - check, self._isa.footprint.to_vertices() - ) + def _clean_any_sub(self): + with self.check( + "Successful subscription query", [self._dss.participant_id] + ) as check: + fetched = self._dss_wrapper.search_subs( + check, self._isa.footprint.to_vertices() + ) for sub_id in fetched.subscriptions.keys(): - self._dss_wrapper.cleanup_sub(check, sub_id=sub_id) + with self.check( + "Successful subscription deletion", [self._dss.participant_id] + ) as check: + self._dss_wrapper.cleanup_sub(check, sub_id=sub_id) def _ensure_clean_workspace_step(self): self.begin_test_step("Ensure clean workspace") - with self.check( - "Successful subscription query and cleanup", [self._dss.participant_id] - ) as check: - self._clean_any_sub(check) + self._clean_any_sub() self.end_test_step() - def _subscription_limitations_case(self): - self.begin_test_case("Subscription limitations") - - self.begin_test_step("Subscription duration limitations") - - self._create_too_long_subscription() - - self.end_test_step() - - self.begin_test_step("Subscription quantity limitations") - - self._create_too_many_subscriptions() - - self.end_test_step() - self.end_test_case() - def _create_too_many_subscriptions(self): + self.begin_test_step("Create maximum number of subscriptions") with self.check( "Create up to the maximum allowed number of subscriptions in an area", [self._dss.participant_id], ) as check: - self._clean_any_sub(check) # Create 10 subscriptions with different ID's for i in range(self._dss.rid_version.dss_max_subscriptions_per_area): sub_id = f"{self._sub_id[:-3]}1{i:02d}" @@ -119,6 +112,10 @@ def _create_too_many_subscriptions(self): **self._default_subscription_params(datetime.timedelta(minutes=30)), ) + self.end_test_step() + + self.begin_test_step("Exceed maximum number of subscriptions") + with self.check( "Enforce maximum number of subscriptions for an area", [self._dss.participant_id], @@ -133,12 +130,21 @@ def _create_too_many_subscriptions(self): **self._default_subscription_params(datetime.timedelta(minutes=30)), ) + self.end_test_step() + + self.begin_test_step("Clean up subscriptions") + + self._clean_any_sub() + + self.end_test_step() + def _create_too_long_subscription(self): + self.begin_test_step("Try to create too-long subscription") + with self.check( - "Enforce maximum duration of subscriptions for an area", + "Too-long subscription creation rejected", [self._dss.participant_id], ) as check: - self._clean_any_sub(check) # Sub with this ID does not exist and too long: we expect either a failure, or # that any subscription that is effectively created to be truncated at 24 hours. creation_attempt = self._dss_wrapper.put_sub_expect_response_code( @@ -156,8 +162,16 @@ def _create_too_long_subscription(self): if creation_attempt.success: self._check_properly_truncated(check, creation_attempt) + self.end_test_step() + + self.begin_test_step("Try to extend subscription") + + with self.check( + "Valid subscription created", + [self._dss.participant_id], + ) as check: # Create a subscription that is fine - self._dss_wrapper.put_sub( + sub = self._dss_wrapper.put_sub( check=check, sub_id=self._sub_id, **self._default_subscription_params( @@ -165,7 +179,11 @@ def _create_too_long_subscription(self): ), ) - # Sub with this ID does exist, an we try to extend it beyond 24 hours: + with self.check( + "Subscription duration limited during update", + [self._dss.participant_id], + ) as check: + # Sub with this ID does exist, and we try to extend it beyond 24 hours: extended_subscription = self._dss_wrapper.put_sub_expect_response_code( check=check, sub_id=self._sub_id, @@ -177,6 +195,20 @@ def _create_too_long_subscription(self): if extended_subscription.success: self._check_properly_truncated(check, extended_subscription) + self.end_test_step() + + self.begin_test_step("Remove subscription") + + with self.check( + "Subscription deleted", + [self._dss.participant_id], + ) as check: + self._dss_wrapper.del_sub( + check=check, sub_id=self._sub_id, sub_version=sub.subscription.version + ) + + self.end_test_step() + def _check_properly_truncated( self, check: PendingCheck, changed: ChangedSubscription ): @@ -191,7 +223,7 @@ def _check_properly_truncated( Severity.Medium, f"{self._dss.participant_id} DSS instance has returned a non-properly truncated subscription " f"(duration: {duration}) " - f"when the expecation was either to fail or to truncate at 24 hours.", + f"when the expectation was either to fail or to truncate at 24 hours.", query_timestamps=[changed.query.request.timestamp], ) # If a subscription was created, we want to delete it before continuing: @@ -211,9 +243,6 @@ def _default_subscription_params(self, duration: datetime.timedelta) -> Dict: def cleanup(self): self.begin_cleanup() - with self.check( - "Successful subscription query and cleanup", [self._dss.participant_id] - ) as check: - self._clean_any_sub(check) + self._clean_any_sub() self.end_cleanup() diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/v19/dss/subscription_validation.md b/monitoring/uss_qualifier/scenarios/astm/netrid/v19/dss/subscription_validation.md index 75e04682e6..865410114f 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/v19/dss/subscription_validation.md +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/v19/dss/subscription_validation.md @@ -22,43 +22,89 @@ Perform basic operations on a single DSS instance to create subscriptions and ch ### Ensure clean workspace test step -This step ensures that we remove any subscription that may already exist for the service area. +This step ensures that we remove any subscription that may already exist for the service area. First, the DSS is queried for any applicable existing subscriptions, and then any subscriptions found are deleted. -#### Successful subscription query and cleanup check +#### Successful subscription query check -We expect to be allowed to query for existing subscriptions in order to clean them up +If the query for subscriptions fails, the "GET Subscriptions" portion of **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)** was not met. -## Subscription limitations test case +#### Successful subscription deletion -### Subscription quantity limitations test step +If the deletion attempt fails, the "DELETE Subscription" portion of **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)** was not met. -The test will attempt to create 10 identical subscriptions for the same area and expect this to succeed, then create an 11th one and expect it to fail. +## Subscription quantity limitations test case + +### Create maximum number of subscriptions test step + +The test will attempt to create 10 identical subscriptions for the same area and expect this to succeed. #### Create up to the maximum allowed number of subscriptions in an area check As per **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)**, the DSS API is expected to allow us to create multiple subscriptions. +### Exceed maximum number of subscriptions test step + +Now, create an 11th one and expect it to fail. + #### Enforce maximum number of subscriptions for an area check If the DSS successfully creates an 11th Subscription in the same area instead of rejecting it, it will not have performed the Subscription count validation as defined in **[astm.f3411.v19.DSS0050](../../../../../requirements/astm/f3411/v19.md)** -### Subscription duration limitations test step +### Clean up subscriptions test step -#### Enforce maximum duration of subscriptions for an area check +Clean up any subscriptions created. -**[astm.f3411.v19.DSS0060](../../../../../requirements/astm/f3411/v19.md)** any subscription to the DSS may not exceed NetDSSMaxSubscriptionDuration (24 hours). +#### Successful subscription query check + +If the query for subscriptions fails, the "GET Subscriptions" portion of **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)** was not met. + +#### Successful subscription deletion check + +If the deletion attempt fails, the "DELETE Subscription" portion of **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)** was not met. + +## Subscription duration limitations test case + +### Try to create too-long subscription test step The test will attempt to create a subscription for 24 hours and 10 minutes, and expect this to fail with an HTTP 400 error. -If the creation succeeds, the test expects that the effectively created subscription has been truncated to 24 hours, with a tolerance of minus 1 minute. +#### Too-long subscription creation rejected check + +**[astm.f3411.v19.DSS0060](../../../../../requirements/astm/f3411/v19.md)** any subscription to the DSS may not exceed NetDSSMaxSubscriptionDuration (24 hours). + +If creation of the too-long-duration subscription succeeds, the test expects that the effectively created subscription has been truncated to 24 hours, with a tolerance of minus 1 minute. + +### Try to extend subscription test step -It will also attempt to create a valid subscription of 23 hours an 59 minutes, and then increase its duration to 24 hours and 10 minutes, +The test will attempt to create a valid subscription of 23 hours and 59 minutes, and then increase its duration to 24 hours and 10 minutes, expecting this update to fail. -## Cleanup +#### Valid subscription created check + +The ability to create a valid subscription is required in **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)**. + +#### Subscription duration limited during update check + +If the DSS allows a user to extend an existing, valid subscription to a duration exceeding that specified in **[astm.f3411.v19.DSS0060](../../../../../requirements/astm/f3411/v19.md)**, this check will fail. + +### Remove subscription test step + +To clean up after itself, the test deletes the subscription created in the previous step. + +#### Subscription deleted check -### Successful subscription query and cleanup check +The ability to delete an existing subscription is required in **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)**. + +## Cleanup The cleanup phase of this test scenario will remove any subscription that may have been created during the test and that intersects with the test ISA. + +### Successful subscription query check + +If the query for subscriptions fails, the "GET Subscriptions" portion of **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)** was not met. + +### Successful subscription deletion + +If the deletion attempt fails, the "DELETE Subscription" portion of **[astm.f3411.v19.DSS0030](../../../../../requirements/astm/f3411/v19.md)** was not met. diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/v19/dss_interoperability.md b/monitoring/uss_qualifier/scenarios/astm/netrid/v19/dss_interoperability.md index 461a013678..f822ff8e74 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/v19/dss_interoperability.md +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/v19/dss_interoperability.md @@ -152,6 +152,8 @@ Qualitatively proves: ISA[*P*] modified with proper response, all Subscription[i #### ISA modification triggers subscription notification requests check +TODO: Implement + **[astm.f3411.v19.A2-6-1,3c](../../../../requirements/astm/f3411/v19.md)** ### S6 test step @@ -274,6 +276,8 @@ Qualitatively proves: Expired Subscription removed from ID index on primary DSS #### 404 with proper response check +TODO: Investigate expected behavior and "404 with proper response" check + **[astm.f3411.v19.A2-6-1,3d](../../../../requirements/astm/f3411/v19.md)** ### S15 test step diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/v19/nominal_behavior.md b/monitoring/uss_qualifier/scenarios/astm/netrid/v19/nominal_behavior.md index 803088ea41..1318567e73 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/v19/nominal_behavior.md +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/v19/nominal_behavior.md @@ -41,6 +41,8 @@ This check will fail if the flight was not successfully injected. #### Valid flight check +TODO: Validate injected flights, especially to make sure they contain the specified injection IDs + Per **[interuss.automated_testing.rid.injection.UpsertTestResult](../../../../requirements/interuss/automated_testing/rid/injection.md)**, the NetRID Service Provider under test should only make valid modifications to the injected flights. This includes: * A flight with the specified injection ID must be returned. diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/__init__.py b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/__init__.py index 1552d4ee7d..ab08466a88 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/__init__.py +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/__init__.py @@ -1,3 +1,4 @@ from .dss_interoperability import DSSInteroperability from .nominal_behavior import NominalBehavior +from .misbehavior import Misbehavior from .aggregate_checks import AggregateChecks diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/dss/isa_simple.md b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/dss/isa_simple.md index 2a033790da..f8b0b8d157 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/dss/isa_simple.md +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/dss/isa_simple.md @@ -80,10 +80,14 @@ The API for **[astm.f3411.v22a.DSS0030](../../../../../requirements/astm/f3411/v The cleanup phase of this test scenario attempts to remove the ISA if the test ended prematurely. -#### Successful ISA query check +### Successful ISA query check **[astm.f3411.v22a.DSS0030](../../../../../requirements/astm/f3411/v22a.md)** requires the implementation of the DSS endpoint enabling retrieval of information about a specific ISA; if the individual ISA cannot be retrieved and the error isn't a 404, then this requirement isn't met. -#### Removed pre-existing ISA check +### Removed pre-existing ISA check If an ISA with the intended ID is still present in the DSS, it needs to be removed before exiting the test. If that ISA cannot be deleted, then the **[astm.f3411.v22a.DSS0030](../../../../../requirements/astm/f3411/v22a.md)** requirement to implement the ISA deletion endpoint might not be met. + +### Notified subscriber check + +When an ISA is deleted, subscribers must be notified. If a subscriber cannot be notified, that subscriber USS did not correctly implement "POST Identification Service Area" in **[astm.f3411.v22a.NET0730](../../../../../requirements/astm/f3411/v22a.md)**. diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/dss/subscription_validation.md b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/dss/subscription_validation.md index 1c5fcfeaee..4d5881e62e 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/dss/subscription_validation.md +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/dss/subscription_validation.md @@ -22,43 +22,89 @@ Perform basic operations on a single DSS instance to create subscriptions and ch ### Ensure clean workspace test step -This step ensures that we remove any subscription that may already exist for the service area. +This step ensures that we remove any subscription that may already exist for the service area. First, the DSS is queried for any applicable existing subscriptions, and then any subscriptions found are deleted. -#### Successful subscription query and cleanup check +#### Successful subscription query check -We expect to be allowed to query for existing subscriptions in order to clean them up +If the query for subscriptions fails, the "GET Subscriptions" portion of **[astm.f3411.v22a.DSS0030](../../../../../requirements/astm/f3411/v22a.md)** was not met. -## Subscription limitations test case +#### Successful subscription deletion -### Subscription quantity limitations test step +If the deletion attempt fails, the "DELETE Subscription" portion of **[astm.f3411.v22a.DSS0030](../../../../../requirements/astm/f3411/v22a.md)** was not met. -The test will attempt to create 10 identical subscriptions for the same area and expect this to succeed, then create an 11th one and expect it to fail. +## Subscription quantity limitations test case + +### Create maximum number of subscriptions test step + +The test will attempt to create 10 identical subscriptions for the same area and expect this to succeed. #### Create up to the maximum allowed number of subscriptions in an area check As per **[astm.f3411.v22a.DSS0030](../../../../../requirements/astm/f3411/v22a.md)**, the DSS API is expected to allow us to create multiple subscriptions. +### Exceed maximum number of subscriptions test step + +Now, create an 11th one and expect it to fail. + #### Enforce maximum number of subscriptions for an area check If the DSS successfully creates an 11th Subscription in the same area instead of rejecting it, it will not have performed the Subscription count validation as defined in **[astm.f3411.v22a.DSS0050](../../../../../requirements/astm/f3411/v22a.md)** -### Subscription duration limitations test step +### Clean up subscriptions test step -#### Enforce maximum duration of subscriptions for an area check +Clean up any subscriptions created. -**[astm.f3411.v22a.DSS0060](../../../../../requirements/astm/f3411/v22a.md)** any subscription to the DSS may not exceed NetDSSMaxSubscriptionDuration (24 hours). +#### Successful subscription query check + +If the query for subscriptions fails, the "GET Subscriptions" portion of **[astm.f3411.v22a.DSS0030](../../../../../requirements/astm/f3411/v22a.md)** was not met. + +#### Successful subscription deletion check + +If the deletion attempt fails, the "DELETE Subscription" portion of **[astm.f3411.v22a.DSS0030](../../../../../requirements/astm/f3411/v22a.md)** was not met. + +## Subscription duration limitations test case + +### Try to create too-long subscription test step The test will attempt to create a subscription for 24 hours and 10 minutes, and expect this to fail with an HTTP 400 error. -If the creation succeeds, the test expects that the effectively created subscription has been truncated to 24 hours, with a tolerance of minus 1 minute. +#### Too-long subscription creation rejected check + +**[astm.f3411.v22a.DSS0060](../../../../../requirements/astm/f3411/v22a.md)** any subscription to the DSS may not exceed NetDSSMaxSubscriptionDuration (24 hours). + +If creation of the too-long-duration subscription succeeds, the test expects that the effectively created subscription has been truncated to 24 hours, with a tolerance of minus 1 minute. + +### Try to extend subscription test step -It will also attempt to create a valid subscription of 23 hours an 59 minutes, and then increase its duration to 24 hours and 10 minutes, +The test will attempt to create a valid subscription of 23 hours and 59 minutes, and then increase its duration to 24 hours and 10 minutes, expecting this update to fail. -## Cleanup +#### Valid subscription created check + +The ability to create a valid subscription is required in **[astm.f3411.v22a.DSS0030](../../../../../requirements/astm/f3411/v22a.md)**. + +#### Subscription duration limited during update check + +If the DSS allows a user to extend an existing, valid subscription to a duration exceeding that specified in **[astm.f3411.v22a.DSS0060](../../../../../requirements/astm/f3411/v22a.md)**, this check will fail. + +### Remove subscription test step + +To clean up after itself, the test deletes the subscription created in the previous step. + +#### Subscription deleted check -### Successful subscription query and cleanup check +The ability to delete an existing subscription is required in **[astm.f3411.v22a.DSS0030](../../../../../requirements/astm/f3411/v22a.md)**. + +## Cleanup The cleanup phase of this test scenario will remove any subscription that may have been created during the test and that intersects with the test ISA. + +### Successful subscription query check + +If the query for subscriptions fails, the "GET Subscriptions" portion of **[astm.f3411.v22a.DSS0030](../../../../../requirements/astm/f3411/v22a.md)** was not met. + +### Successful subscription deletion + +If the deletion attempt fails, the "DELETE Subscription" portion of **[astm.f3411.v22a.DSS0030](../../../../../requirements/astm/f3411/v22a.md)** was not met. diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/dss_interoperability.md b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/dss_interoperability.md index 0c04986259..85898a4f18 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/dss_interoperability.md +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/dss_interoperability.md @@ -151,6 +151,8 @@ Qualitatively proves: ISA[*P*] modified with proper response, all Subscription[i #### ISA modification triggers subscription notification requests check +TODO: Implement + **[astm.f3411.v22a.A2-6-1,3c](../../../../requirements/astm/f3411/v22a.md)** ### S6 test step @@ -273,6 +275,8 @@ Qualitatively proves: Expired Subscription removed from ID index on primary DSS #### 404 with proper response check +TODO: Investigate expected behavior and "404 with proper response" check + **[astm.f3411.v22a.A2-6-1,3d](../../../../requirements/astm/f3411/v22a.md)** ### S15 test step diff --git a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/nominal_behavior.md b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/nominal_behavior.md index b5994a24eb..beb41a6b49 100644 --- a/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/nominal_behavior.md +++ b/monitoring/uss_qualifier/scenarios/astm/netrid/v22a/nominal_behavior.md @@ -41,6 +41,8 @@ This check will fail if the flight was not successfully injected. #### Valid flight check +TODO: Validate injected flights, especially to make sure they contain the specified injection IDs + Per **[interuss.automated_testing.rid.injection.UpsertTestResult](../../../../requirements/interuss/automated_testing/rid/injection.md)**, the NetRID Service Provider under test should only make valid modifications to the injected flights. This includes: * A flight with the specified injection ID must be returned. diff --git a/monitoring/uss_qualifier/scenarios/scenario.py b/monitoring/uss_qualifier/scenarios/scenario.py index 78a0791fb3..a411543358 100644 --- a/monitoring/uss_qualifier/scenarios/scenario.py +++ b/monitoring/uss_qualifier/scenarios/scenario.py @@ -367,8 +367,16 @@ def check( name=name, applicable_requirements=[], has_todo=False ) else: + test_step_name = ( + self._current_step.name if self._current_step else "" + ) + test_case_name = ( + self._current_case.name + if self._current_case + else "" + ) raise RuntimeError( - f'Test scenario `{self.me()}` was instructed to prepare to record outcome for check "{name}" during test step "{self._current_step.name}" during test case "{self._current_case.name}", but that check is not declared in documentation; declared checks are: {check_list}' + f'Test scenario `{self.me()}` was instructed to prepare to record outcome for check "{name}" during test step "{test_step_name}" during test case "{test_case_name}", but that check is not declared in documentation; declared checks are: {check_list}' ) return PendingCheck( phase=self._phase, diff --git a/monitoring/uss_qualifier/suites/astm/netrid/f3411_19.md b/monitoring/uss_qualifier/suites/astm/netrid/f3411_19.md index 3d6500092d..089ea93df4 100644 --- a/monitoring/uss_qualifier/suites/astm/netrid/f3411_19.md +++ b/monitoring/uss_qualifier/suites/astm/netrid/f3411_19.md @@ -21,7 +21,7 @@ Checked in - astm.f3411.v19 + astm
.f3411
.v19 A2-6-1,1a Implemented ASTM F3411-19 NetRID DSS interoperability @@ -63,12 +63,12 @@ A2-6-1,3c - Implemented + In progress ASTM F3411-19 NetRID DSS interoperability A2-6-1,3d - Implemented + In progress ASTM F3411-19 NetRID DSS interoperability @@ -237,7 +237,7 @@ ASTM NetRID nominal behavior - interuss.automated_testing.rid.injection + interuss
.automated_testing
.rid
.injection DeleteTestSuccess Implemented ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior @@ -249,7 +249,7 @@ UpsertTestResult - Implemented + Planned ASTM NetRID nominal behavior @@ -258,7 +258,7 @@ ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior - interuss.automated_testing.rid.observation + interuss
.automated_testing
.rid
.observation ObservationSuccess Implemented ASTM NetRID nominal behavior diff --git a/monitoring/uss_qualifier/suites/astm/netrid/f3411_19/dss_probing.md b/monitoring/uss_qualifier/suites/astm/netrid/f3411_19/dss_probing.md index 0fb808c086..2bf986cf77 100644 --- a/monitoring/uss_qualifier/suites/astm/netrid/f3411_19/dss_probing.md +++ b/monitoring/uss_qualifier/suites/astm/netrid/f3411_19/dss_probing.md @@ -16,7 +16,7 @@ Checked in - astm.f3411.v19 + astm
.f3411
.v19 DSS0030 Implemented ASTM NetRID DSS: Subscription Validation diff --git a/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a.md b/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a.md index c289870712..6096cf8563 100644 --- a/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a.md +++ b/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a.md @@ -8,7 +8,7 @@ 1. Suite: [DSS instance probing for ASTM NetRID F3411-22a](f3411_22a/dss_probing.md) ([`suites.astm.netrid.f3411_22a.dss_probing`](f3411_22a/dss_probing.yaml)) 2. Scenario: [ASTM F3411-22a NetRID DSS interoperability](../../../scenarios/astm/netrid/v22a/dss_interoperability.md) ([`scenarios.astm.netrid.v22a.DSSInteroperability`](../../../scenarios/astm/netrid/v22a/dss_interoperability.py)) 3. Scenario: [ASTM NetRID nominal behavior](../../../scenarios/astm/netrid/v22a/nominal_behavior.md) ([`scenarios.astm.netrid.v22a.NominalBehavior`](../../../scenarios/astm/netrid/v22a/nominal_behavior.py)) -4. Scenario: [ASTM NetRID SP clients misbehavior handling](../../../scenarios/astm/netrid/v19/misbehavior.md) ([`scenarios.astm.netrid.v19.Misbehavior`](../../../scenarios/astm/netrid/v19/misbehavior.py)) +4. Scenario: [ASTM NetRID SP clients misbehavior handling](../../../scenarios/astm/netrid/v22a/misbehavior.md) ([`scenarios.astm.netrid.v22a.Misbehavior`](../../../scenarios/astm/netrid/v22a/misbehavior.py)) 4. Scenario: [ASTM F3411-22a NetRID aggregate checks](../../../scenarios/astm/netrid/v22a/aggregate_checks.md) ([`scenarios.astm.netrid.v22a.AggregateChecks`](../../../scenarios/astm/netrid/v22a/aggregate_checks.py)) ## Checked requirements @@ -21,13 +21,7 @@ Checked in - astm.f3411.v19 - NET0500 - Implemented - ASTM NetRID SP clients misbehavior handling - - - astm.f3411.v22a + astm
.f3411
.v22a A2-6-1,1a Implemented ASTM F3411-22a NetRID DSS interoperability @@ -69,12 +63,12 @@ A2-6-1,3c - Implemented + In progress ASTM F3411-22a NetRID DSS interoperability A2-6-1,3d - Implemented + In progress ASTM F3411-22a NetRID DSS interoperability @@ -270,7 +264,7 @@ NET0500 Implemented - ASTM NetRID nominal behavior + ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior NET0610 @@ -283,10 +277,15 @@ ASTM NetRID DSS: Simple ISA
ASTM NetRID nominal behavior - interuss.automated_testing.rid.injection + NET0730 + Implemented + ASTM NetRID DSS: Simple ISA + + + interuss
.automated_testing
.rid
.injection DeleteTestSuccess Implemented - ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior + ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior ExpectedBehavior @@ -295,16 +294,16 @@ UpsertTestResult - Implemented + Planned ASTM NetRID nominal behavior UpsertTestSuccess Implemented - ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior + ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior - interuss.automated_testing.rid.observation + interuss
.automated_testing
.rid
.observation ObservationSuccess Implemented ASTM NetRID nominal behavior diff --git a/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a.yaml b/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a.yaml index 2bf688260b..88d4f2011a 100644 --- a/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a.yaml +++ b/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a.yaml @@ -41,7 +41,7 @@ actions: dss_pool: dss_instances on_failure: Continue - test_scenario: - scenario_type: scenarios.astm.netrid.v19.Misbehavior + scenario_type: scenarios.astm.netrid.v22a.Misbehavior resources: flights_data: flights_data service_providers: service_providers diff --git a/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a/dss_probing.md b/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a/dss_probing.md index 2b58a18ee9..8a60f30246 100644 --- a/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a/dss_probing.md +++ b/monitoring/uss_qualifier/suites/astm/netrid/f3411_22a/dss_probing.md @@ -17,7 +17,7 @@ Checked in - astm.f3411.v22a + astm
.f3411
.v22a DSS0030 Implemented ASTM NetRID DSS: Simple ISA
ASTM NetRID DSS: Subscription Validation @@ -37,4 +37,9 @@ Implemented ASTM NetRID DSS: Simple ISA + + NET0730 + Implemented + ASTM NetRID DSS: Simple ISA + diff --git a/monitoring/uss_qualifier/suites/astm/utm/f3548_21.md b/monitoring/uss_qualifier/suites/astm/utm/f3548_21.md index d27cf50d52..b6f70a1d08 100644 --- a/monitoring/uss_qualifier/suites/astm/utm/f3548_21.md +++ b/monitoring/uss_qualifier/suites/astm/utm/f3548_21.md @@ -21,7 +21,7 @@ Checked in - astm.f3548.v21 + astm
.f3548
.v21 DSS0005 Implemented Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Validation of operational intents @@ -112,7 +112,7 @@ Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Validation of operational intents - interuss.automated_testing.flight_planning + interuss
.automated_testing
.flight_planning ClearArea Implemented Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Validation of operational intents diff --git a/monitoring/uss_qualifier/suites/documentation/documentation.py b/monitoring/uss_qualifier/suites/documentation/documentation.py index c708113606..8ad4f3c6e6 100644 --- a/monitoring/uss_qualifier/suites/documentation/documentation.py +++ b/monitoring/uss_qualifier/suites/documentation/documentation.py @@ -158,7 +158,8 @@ def make_test_suite_documentation( req_md_path = os.path.relpath( req_ids_by_package[package][0].md_file_path(), start=base_path ) - package_line = f' {package}' + package_caption = "
.".join(package.split(".")) + package_line = f' {package_caption}' for req_id in sorted(req_ids_by_package[package]): req_text = f'{req_id.requirement_name()}' diff --git a/monitoring/uss_qualifier/suites/documentation/format_documentation.py b/monitoring/uss_qualifier/suites/documentation/format_documentation.py index 0859588a5d..680ef89e3d 100644 --- a/monitoring/uss_qualifier/suites/documentation/format_documentation.py +++ b/monitoring/uss_qualifier/suites/documentation/format_documentation.py @@ -40,7 +40,9 @@ def main(lint: bool) -> int: continue changes = True if lint: - print(f"Test suite documentation must be regenerated: {suite_doc_file}") + print( + f"Test suite documentation must be regenerated with `make format`: {suite_doc_file}" + ) else: with open(suite_doc_file, "w") as f: f.write(suite_doc_content) diff --git a/monitoring/uss_qualifier/suites/faa/uft/message_signing.md b/monitoring/uss_qualifier/suites/faa/uft/message_signing.md index 3968595b3f..3cfd3ac839 100644 --- a/monitoring/uss_qualifier/suites/faa/uft/message_signing.md +++ b/monitoring/uss_qualifier/suites/faa/uft/message_signing.md @@ -18,7 +18,7 @@ Checked in - astm.f3548.v21 + astm
.f3548
.v21 DSS0005 Implemented Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Validation of operational intents @@ -109,7 +109,7 @@ Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Validation of operational intents - interuss.automated_testing.flight_planning + interuss
.automated_testing
.flight_planning ClearArea Implemented Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Validation of operational intents diff --git a/monitoring/uss_qualifier/suites/uspace/flight_auth.md b/monitoring/uss_qualifier/suites/uspace/flight_auth.md index a2800dee46..22af2d494b 100644 --- a/monitoring/uss_qualifier/suites/uspace/flight_auth.md +++ b/monitoring/uss_qualifier/suites/uspace/flight_auth.md @@ -18,7 +18,7 @@ Checked in - astm.f3548.v21 + astm
.f3548
.v21 DSS0005 Implemented Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Validation of operational intents @@ -109,7 +109,7 @@ Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Validation of operational intents - interuss.automated_testing.flight_planning + interuss
.automated_testing
.flight_planning ClearArea Implemented Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Validation of operational intents diff --git a/monitoring/uss_qualifier/suites/uspace/network_identification.md b/monitoring/uss_qualifier/suites/uspace/network_identification.md index 3d1e73ffae..ec7130edb8 100644 --- a/monitoring/uss_qualifier/suites/uspace/network_identification.md +++ b/monitoring/uss_qualifier/suites/uspace/network_identification.md @@ -16,13 +16,7 @@ Checked in - astm.f3411.v19 - NET0500 - Implemented - ASTM NetRID SP clients misbehavior handling - - - astm.f3411.v22a + astm
.f3411
.v22a A2-6-1,1a Implemented ASTM F3411-22a NetRID DSS interoperability @@ -64,12 +58,12 @@ A2-6-1,3c - Implemented + In progress ASTM F3411-22a NetRID DSS interoperability A2-6-1,3d - Implemented + In progress ASTM F3411-22a NetRID DSS interoperability @@ -265,7 +259,7 @@ NET0500 Implemented - ASTM NetRID nominal behavior + ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior NET0610 @@ -278,10 +272,15 @@ ASTM NetRID DSS: Simple ISA
ASTM NetRID nominal behavior - interuss.automated_testing.rid.injection + NET0730 + Implemented + ASTM NetRID DSS: Simple ISA + + + interuss
.automated_testing
.rid
.injection DeleteTestSuccess Implemented - ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior + ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior ExpectedBehavior @@ -290,16 +289,16 @@ UpsertTestResult - Implemented + Planned ASTM NetRID nominal behavior UpsertTestSuccess Implemented - ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior + ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior - interuss.automated_testing.rid.observation + interuss
.automated_testing
.rid
.observation ObservationSuccess Implemented ASTM NetRID nominal behavior diff --git a/monitoring/uss_qualifier/suites/uspace/required_services.md b/monitoring/uss_qualifier/suites/uspace/required_services.md index fbaf8dece6..1e24a9d65f 100644 --- a/monitoring/uss_qualifier/suites/uspace/required_services.md +++ b/monitoring/uss_qualifier/suites/uspace/required_services.md @@ -17,13 +17,7 @@ Checked in - astm.f3411.v19 - NET0500 - Implemented - ASTM NetRID SP clients misbehavior handling - - - astm.f3411.v22a + astm
.f3411
.v22a A2-6-1,1a Implemented ASTM F3411-22a NetRID DSS interoperability @@ -65,12 +59,12 @@ A2-6-1,3c - Implemented + In progress ASTM F3411-22a NetRID DSS interoperability A2-6-1,3d - Implemented + In progress ASTM F3411-22a NetRID DSS interoperability @@ -266,7 +260,7 @@ NET0500 Implemented - ASTM NetRID nominal behavior + ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior NET0610 @@ -279,7 +273,12 @@ ASTM NetRID DSS: Simple ISA
ASTM NetRID nominal behavior - astm.f3548.v21 + NET0730 + Implemented + ASTM NetRID DSS: Simple ISA + + + astm
.f3548
.v21 DSS0005 Implemented Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Validation of operational intents @@ -370,7 +369,7 @@ Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Validation of operational intents - interuss.automated_testing.flight_planning + interuss
.automated_testing
.flight_planning ClearArea Implemented Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Validation of operational intents @@ -386,10 +385,10 @@ Flight authorisation validation
Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Validation of operational intents - interuss.automated_testing.rid.injection + interuss
.automated_testing
.rid
.injection DeleteTestSuccess Implemented - ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior + ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior ExpectedBehavior @@ -398,16 +397,16 @@ UpsertTestResult - Implemented + Planned ASTM NetRID nominal behavior UpsertTestSuccess Implemented - ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior + ASTM NetRID SP clients misbehavior handling
ASTM NetRID nominal behavior - interuss.automated_testing.rid.observation + interuss
.automated_testing
.rid
.observation ObservationSuccess Implemented ASTM NetRID nominal behavior