diff --git a/.github/workflows/CI.md b/.github/workflows/CI.md index 45b9860ad9..a89f3ad5a8 100644 --- a/.github/workflows/CI.md +++ b/.github/workflows/CI.md @@ -1,37 +1,31 @@ # [Continuous integration](ci.yml) +## Overview + Before a pull request can be merged into the main branch, it must pass all automated tests for the repository. This document describes the tests and how to run them locally. To run the equivalent of the full CI suite locally, simply execute `make presubmit`. This should perform all of the steps below, including bringing up a local interoperability ecosystem and all local USS mocks necessary to conduct tests. Therefore, `make presubmit` can (and should) be run starting without any pre-existing local infrastructure (interoperability ecosystem and/or USS mocks). The downside of `make presubmit` is that it takes a long time, and a lot of that time is spent bringing up and tearing down local infrastructure (interoperability ecosystem and USS mocks). If your changes do not affect one or both of these things, you may save substantial time by bringing up an interoperability ecosystem once and leaving it up while developing, and possibly bringing up a set of USS mocks ([mock_uss](../../monitoring/mock_uss) instances) and leaving them up while developing (though obviously this will not work if developing changes that affect mock_uss). See uss_qualifier tests section below. -## Repository hygiene (`make check-hygiene`) - -This set of tests ensures consistent formatting and performs other hygiene-related tasks that are not usually related to the actual functionality of the repository. +## Checks -### Python lint (`make python-lint`) +### Repository hygiene (`make check-hygiene`) -To maintain consistency across this large codebase, InterUSS employs a very strict Python linter. But, there is no need to satisfy the linter manually; simply run `make format` from the repo root to resolve most mere-formatting issues that this check will detect. +This set of tests including [miscellaneous hygiene checks](../../test/repo_hygiene/README.md) ensures consistent formatting and performs other hygiene-related tasks that are not usually related to the actual functionality of the repository. -### Automated hygiene verification (`make hygiene`) +#### Python lint (`make python-lint`) -This check performs [miscellaneous hygiene checks](../../test/repo_hygiene/README.md). Currently, it ensures that local links in Markdown (*.md) files are not broken. +To maintain consistency across this large codebase, InterUSS employs a very strict Python linter. But, there is no need to satisfy the linter manually; simply run `make format` from the repo root to resolve most mere-formatting issues that this check will detect. -### uss_qualifier documentation validation (`make validate-uss-qualifier-docs`) +#### uss_qualifier documentation validation (`make validate-uss-qualifier-docs`) uss_qualifier [scenario documentation](../../monitoring/uss_qualifier/scenarios/README.md#documentation) is required and has strict requirements to ensure a strong, consistent, and correct interface between regulators and others concerned only with the concept of what scenarios are doing, and InterUSS contributors concerned with writing the actual code to perform the test. This check validates many of these documentation requirements. -### Shell lint (`make shell-lint`) +#### Shell lint (`make shell-lint`) This repository contains a large number of shell scripts. This check attempts to have these scripts follow best practices and avoid common pitfalls. -## `monitoring` tests (`make check-monitoring`) - -These tests verify functional behavior of the repository content. - -### monitorlib tests (`make test` in monitoring/monitorlib) - ### mock_uss tests (`make test` in monitoring/mock_uss) This check runs unit tests for mock_uss. @@ -56,4 +50,34 @@ uss_qualifier's run_locally.sh ([monitoring/uss_qualifier/run_locally.sh](../../ ### prober tests (`make test` in monitoring/prober) -[prober](../../monitoring/prober/README.md) is a legacy test suite dedicated to integration testing of DSS instances. It is being migrated to test scenarios and suites in uss_qualifier, but in the mean time, it is still the standard way to validate DSS functionality. It runs on the DSS provided as part of the local interoperability ecosystem. +[prober](../../monitoring/prober/README.md) is a legacy test suite dedicated to integration testing of DSS instances. It is being migrated to test scenarios and suites in uss_qualifier, but in the meantime, it is still the standard way to validate DSS functionality. It runs on the DSS provided as part of the local interoperability ecosystem. + +## Troubleshooting + +Whenever CI is run on GitHub, a large amount of documentation on that test run is generated. Each of the checks is documented separately. When CI is run upon merging a PR to the main branch, the list of checks is accessible by clicking on the green checkmark (or red x) near the top of [the repo homepage](https://github.com/interuss/monitoring): + +![Repo homepage checks location](../../assets/ci/checks_repo_homepage.png) + +When CI is run for a PR, the list of checks is visible near the bottom of the PR page. If all checks passed, the list of checks must be opened by clicking "Show all checks". + +In either case, the details of a particular check can be viewed by clicking on the "Details" link to the right of the check name: + +![CI check details](../../assets/ci/check_details.png) + +This GitHub Actions job run page shows the console output from the job. If there was an exception during the run, the error message of interest will usually be near the end of the last console output for the job. + +### uss_qualifier artifacts + +Some of the most useful outputs of uss_qualifier are the artifacts generated from the uss_qualifier test runs that are part of the CI. When a PR is merged to the main branch of the repository, some of these artifacts are published to [https://interuss.github.io/monitoring](https://interuss.github.io/monitoring/). These artifacts (and some additional ones) are available for every CI run. To access them from a GitHub Actions job run page (see just above), first click "Summary" for the run: + +![CI Summary location](../../assets/ci/summary_location.png) + +Then, scroll down to the bottom of the page to see the list of GitHub Actions artifacts. The GitHub Actions artifact that contains the uss_qualifier artifacts from all uss_qualifier test runs during the CI is the one labeled "monitoring-test-uss_qualifier-reports": + +![GitHub Actions artifacts](../../assets/ci/artifacts.png) + +Click on this artifact to download a zip file which can then be unzipped to reveal a uss_qualifier/output folder structure. Each subfolder in uss_qualifier/output contains the output of a particular test run of uss_qualifier -- for instance, the `f3548` folder contains the output from running the [`configurations.dev.f3548_self_contained` configuration](../../monitoring/uss_qualifier/configurations/dev/f3548_self_contained.yaml). The sequence view artifact is often particularly useful for debugging. + +### Automated testing logs + +The logs from all of the containers running during uss_qualifier automated tests (DSS instance, mock_uss instances, etc) can be found in the GitHub Actions artifact labeled "monitoring-test-uss_qualifier-logs"; see above for download instructions. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 28203bda8c..a4ca15b22b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -12,11 +12,13 @@ This repository has a very strict Python linter, as well as very strict expected When [a PR is created](https://github.com/interuss/tsc/blob/main/repo_contributions.md#create-draft-pr-in-interuss-repository), the [continuous integration (CI) tests for this repository](./.github/workflows/CI.md) will run, and the PR will generally not be reviewed until they pass (unless [committer help is requested](https://github.com/interuss/tsc/blob/main/repo_contributions.md#request-committer-help-via-comment-in-pr) to address the failure). See [the continuous integration test documentation](./.github/workflows/CI.md) for how to run these tests on your local system more quickly and efficiently to be confident your PR will pass the CI tests when created (or when updates are made). -### Failing "uss_qualifier tests" CI check +### Troubleshooting + +See [the continuous integration test documentation](./.github/workflows/CI.md) for how to troubleshoot failing CI for a PR. -If `make presubmit` succeeds on a developer's local machine, the GitHub CI actions should succeed as well. [A known issue](https://github.com/interuss/monitoring/issues/28) frequently causes the "uss_qualifier tests" check to fail. If the failed check indicates a query response code of 999 (this is the code InterUSS indicates when no response is received), this is very likely the problem. A committer can rerun the CI check and it is likely to succeed on the second try with no changes. +### Failing "uss_qualifier tests" CI check -If anyone can resolve [issue #28](https://github.com/interuss/monitoring/issues/28) which causes this problem, that help would be enormously appreciated by InterUSS. +If `make presubmit` succeeds on a developer's local machine, the GitHub CI actions should succeed as well. If `make presubmit` succeeds locally but the GitHub CI actions fail, that may indicate help from an InterUSS committer would be helpful. ## uss_qualifier test scenarios diff --git a/Makefile b/Makefile index 1b3a6f1c14..1e8c4debad 100644 --- a/Makefile +++ b/Makefile @@ -83,13 +83,13 @@ collect-local-logs: -sh -c "build/dev/run_locally.sh logs --timestamps" > logs/local_infra.log 2>&1 -docker logs mock_uss_scdsc_a > logs/mock_uss_scdsc_a.log 2>&1 -docker logs mock_uss_scdsc_b > logs/mock_uss_scdsc_b.log 2>&1 + -docker logs mock_uss_geoawareness > logs/mock_uss_geoawareness.log 2>&1 -docker logs mock_uss_ridsp > logs/mock_uss_ridsp.log 2>&1 - -docker logs mock_uss_ridsp_v22a > logs/mock_uss_ridsp_v22a.log 2>&1 -docker logs mock_uss_riddp > logs/mock_uss_riddp.log 2>&1 - -docker logs mock_uss_riddp_v22a > logs/mock_uss_riddp_v22a.log 2>&1 - -docker logs mock_uss_geoawareness > logs/mock_uss_geoawareness.log 2>&1 + -docker logs mock_uss_ridsp_v19 > logs/mock_uss_ridsp_v19.log 2>&1 + -docker logs mock_uss_riddp_v19 > logs/mock_uss_riddp_v19.log 2>&1 -docker logs mock_uss_tracer > logs/mock_uss_tracer.log 2>&1 - -docker logs mock_uss_tracer_v22a > logs/mock_uss_tracer_v22a.log 2>&1 + -docker logs mock_uss_scdsc_interaction_log > logs/mock_uss_scdsc_interaction_log.log 2>&1 .PHONY: stop-locally stop-locally: diff --git a/assets/ci/artifacts.png b/assets/ci/artifacts.png new file mode 100644 index 0000000000..9c51eee230 Binary files /dev/null and b/assets/ci/artifacts.png differ diff --git a/assets/ci/check_details.png b/assets/ci/check_details.png new file mode 100644 index 0000000000..f127f4dd0d Binary files /dev/null and b/assets/ci/check_details.png differ diff --git a/assets/ci/checks_repo_homepage.png b/assets/ci/checks_repo_homepage.png new file mode 100644 index 0000000000..735d0714a9 Binary files /dev/null and b/assets/ci/checks_repo_homepage.png differ diff --git a/assets/ci/summary_location.png b/assets/ci/summary_location.png new file mode 100644 index 0000000000..11535d0d9c Binary files /dev/null and b/assets/ci/summary_location.png differ diff --git a/monitoring/monitorlib/clients/flight_planning/client_v1.py b/monitoring/monitorlib/clients/flight_planning/client_v1.py index 2eb3451e5c..d47178941f 100644 --- a/monitoring/monitorlib/clients/flight_planning/client_v1.py +++ b/monitoring/monitorlib/clients/flight_planning/client_v1.py @@ -16,6 +16,7 @@ ) from monitoring.monitorlib.clients.flight_planning.planning import ( PlanningActivityResponse, + AdvisoryInclusion, ) from monitoring.monitorlib.clients.flight_planning.planning import ( PlanningActivityResult, @@ -93,7 +94,9 @@ def _inject( queries=[query], activity_result=resp.planning_result, flight_plan_status=resp.flight_plan_status, - includes_advisories=resp.includes_advisories, + includes_advisories=resp.includes_advisories + if "includes_advisories" in resp + else AdvisoryInclusion.Unknown, ) return response diff --git a/monitoring/monitorlib/errors.py b/monitoring/monitorlib/errors.py index dd5880f465..85121aaff0 100644 --- a/monitoring/monitorlib/errors.py +++ b/monitoring/monitorlib/errors.py @@ -4,3 +4,11 @@ def stacktrace_string(e: Exception) -> str: """Return a multi-line string containing a stacktrace for the specified exception.""" return "".join(traceback.format_exception(e)) + + +def current_stack_string(exclude_levels: int = 1) -> str: + """Return a multi-line string containing a trace of the current execution state.""" + stack = traceback.extract_stack() + if exclude_levels > 0: + stack = stack[0:-exclude_levels] + return "".join(traceback.format_list(stack)) diff --git a/monitoring/uss_qualifier/requirements/astm/f3548/v21.md b/monitoring/uss_qualifier/requirements/astm/f3548/v21.md index 917e37139e..a7aefa15ed 100644 --- a/monitoring/uss_qualifier/requirements/astm/f3548/v21.md +++ b/monitoring/uss_qualifier/requirements/astm/f3548/v21.md @@ -147,7 +147,7 @@ For information on these requirements, refer to [the ASTM standard F3548-21](htt ## Discovery and Synchronization Service -* DSS0005 +* DSS0005 * DSS0005,1 * DSS0005,2 * DSS0005,3 diff --git a/monitoring/uss_qualifier/resources/flight_planning/flight_planner.py b/monitoring/uss_qualifier/resources/flight_planning/flight_planner.py index 08ce4214ba..81d34bb85d 100644 --- a/monitoring/uss_qualifier/resources/flight_planning/flight_planner.py +++ b/monitoring/uss_qualifier/resources/flight_planning/flight_planner.py @@ -211,7 +211,14 @@ def request_flight( operational_intent_id="", ) - return response, resp.queries[0], flight_id, resp.includes_advisories + return ( + response, + resp.queries[0], + flight_id, + resp.includes_advisories + if "includes_advisories" in resp + else AdvisoryInclusion.Unknown, + ) def cleanup_flight( self, flight_id: str diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_sharing_operational_intent_but_with_invalid_interuss_data.md b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_sharing_operational_intent_but_with_invalid_interuss_data.md index 12f4ce1496..82dff852a4 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_sharing_operational_intent_but_with_invalid_interuss_data.md +++ b/monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/test_steps/validate_sharing_operational_intent_but_with_invalid_interuss_data.md @@ -4,7 +4,8 @@ This step verifies that a created flight is shared properly per ASTM F3548-21 by ## 🛑 DSS responses check -**[astm.f3548.v21.DSS0005](../../../../../requirements/astm/f3548/v21.md)** +If the DSS fails to properly respond to a valid search query for operational intent references in an area, +it is in violation of **[astm.f3548.v21.DSS0005,2](../../../../../requirements/astm/f3548/v21.md)**, and this check will fail. ## 🛑 Operational intent shared correctly check diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/flight_intent_validation/flight_intent_validation.md b/monitoring/uss_qualifier/scenarios/astm/utm/flight_intent_validation/flight_intent_validation.md index 1ab4a1f4cc..f606904927 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/flight_intent_validation/flight_intent_validation.md +++ b/monitoring/uss_qualifier/scenarios/astm/utm/flight_intent_validation/flight_intent_validation.md @@ -59,7 +59,9 @@ The flight intent should be successfully transition to Ended state by the flight ### Validate flight intent is non-discoverable test step #### DSS responses check -**[astm.f3548.v21.DSS0005](../../../../requirements/astm/f3548/v21.md)** + +If the DSS fails to reply to a query concerning operational intent references in a given area, +it is in violation of **[astm.f3548.v21.DSS0005,2](../../../../requirements/astm/f3548/v21.md)**, and this check will fail. #### Operational intent not shared check If the operational intent is still discoverable after it was transitioned to Ended, diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/off_nominal_planning/down_uss.md b/monitoring/uss_qualifier/scenarios/astm/utm/off_nominal_planning/down_uss.md index d8d2039b8c..06a7da38b2 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/off_nominal_planning/down_uss.md +++ b/monitoring/uss_qualifier/scenarios/astm/utm/off_nominal_planning/down_uss.md @@ -32,7 +32,7 @@ FlightPlannerResource that is under test and will manage Flight 1. ### dss DSSInstanceResource that provides access to a DSS instance where: - flight creation/sharing can be verified, -- the USS qualifier acting as a virtual USS can create operational intents, and +- the USS qualifier acting as a virtual USS can create operational intent references, and - the USS qualifier can act as an availability arbitrator. ## Setup test case @@ -44,11 +44,10 @@ Make a dummy request to the DSS in order to resolve the USS ID of the virtual US ### [Restore virtual USS availability test step](../set_uss_available.md) ### Clear operational intents created by virtual USS test step -Delete any leftover operational intents created at DSS by virtual USS. +Delete any leftover operational intent references created at DSS by virtual USS. #### Successful operational intents cleanup check -If the search for own operational intents or their deletion fail, this check fails per **[astm.f3548.v21.DSS0005](../../../../requirements/astm/f3548/v21.md)**. - +If the search for operational intent references or their deletion fail, this check fails per **[astm.f3548.v21.DSS0005,2](../../../../requirements/astm/f3548/v21.md)** or **[astm.f3548.v21.DSS0005,1](../../../../requirements/astm/f3548/v21.md)**, respectively. ## Plan Flight 1 in conflict with accepted operational intent managed by down USS test case This test case aims at testing requirement **[astm.f3548.v21.SCD0005](../../../../requirements/astm/f3548/v21.md)**. @@ -93,11 +92,14 @@ All flight intent data provided was complete and correct. It should have been pr to reject or accept Flight 1. If the USS indicates that the injection attempt failed, this check will fail per **[interuss.automated_testing.flight_planning.ExpectedBehavior](../../../../requirements/interuss/automated_testing/flight_planning.md)**. -### [Validate Flight 1 status test step](../validate_shared_operational_intent.md) +### Validate Flight 1 status test step This step validates that the response of the USS is consistent with the flight shared, i.e. either it was properly planned, or the USS rejected the planning. +#### [Validate accepted Flight 1 status](../validate_shared_operational_intent.md) If the planning was accepted, Flight 1 should have been shared. + +#### [Validate rejected Flight 1 status](../validate_not_shared_operational_intent.md) If the planning was rejected, Flight 1 should not have been shared, thus should not exist. ## Cleanup @@ -110,4 +112,4 @@ Delete flights injected at USS through the flight planning interface. ### Successful operational intents cleanup check Delete operational intents created at DSS by virtual USS. -If the search for own operational intents or their deletion fail, this check fails per **[astm.f3548.v21.DSS0005](../../../../requirements/astm/f3548/v21.md)**. +If the search for operational intent references or their deletion fail, this check fails per **[astm.f3548.v21.DSS0005,2](../../../../requirements/astm/f3548/v21.md)** or **[astm.f3548.v21.DSS0005,1](../../../../requirements/astm/f3548/v21.md)**, respectively. diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/off_nominal_planning/down_uss.py b/monitoring/uss_qualifier/scenarios/astm/utm/off_nominal_planning/down_uss.py index 43becbfecd..05486cefb4 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/off_nominal_planning/down_uss.py +++ b/monitoring/uss_qualifier/scenarios/astm/utm/off_nominal_planning/down_uss.py @@ -278,7 +278,7 @@ def _clear_op_intents(self): ) as check: if find_query.status_code != 200: check.record_failed( - summary=f"Failed to query operational intents from DSS in {self._intents_extent} for cleanup", + summary=f"Failed to query operational intent references from DSS in {self._intents_extent} for cleanup", severity=Severity.High, details=f"DSS responded code {find_query.status_code}; error message: {find_query.error_message}", query_timestamps=[find_query.request.timestamp], diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/off_nominal_planning/down_uss_equal_priority_not_permitted.md b/monitoring/uss_qualifier/scenarios/astm/utm/off_nominal_planning/down_uss_equal_priority_not_permitted.md index 70186a0a67..4bca58ad6f 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/off_nominal_planning/down_uss_equal_priority_not_permitted.md +++ b/monitoring/uss_qualifier/scenarios/astm/utm/off_nominal_planning/down_uss_equal_priority_not_permitted.md @@ -48,7 +48,7 @@ Make a dummy request to the DSS in order to resolve the USS ID of the virtual US Delete any leftover operational intents created at DSS by virtual USS. #### Successful operational intents cleanup check -If the search for own operational intents or their deletion fail, this check fails per **[astm.f3548.v21.DSS0005](../../../../requirements/astm/f3548/v21.md)**. +If the search for operational intent references or their deletion fail, this check fails per **[astm.f3548.v21.DSS0005,2](../../../../requirements/astm/f3548/v21.md)** or **[astm.f3548.v21.DSS0005,1](../../../../requirements/astm/f3548/v21.md)**, respectively. ## Plan Flight 2 in conflict with activated operational intent managed by down USS test case This test case aims at testing requirement **[astm.f3548.v21.SCD0010](../../../../requirements/astm/f3548/v21.md)**. @@ -176,4 +176,4 @@ Delete flights injected at USS through the flight planning interface. ### Successful operational intents cleanup check Delete operational intents created at DSS by virtual USS. -If the search for own operational intents or their deletion fail, this check fails per **[astm.f3548.v21.DSS0005](../../../../requirements/astm/f3548/v21.md)**. +If the search for operational intent references or their deletion fail, this check fails per **[astm.f3548.v21.DSS0005,2](../../../../requirements/astm/f3548/v21.md)** or **[astm.f3548.v21.DSS0005,1](../../../../requirements/astm/f3548/v21.md)**, respectively. diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/prep_planners.md b/monitoring/uss_qualifier/scenarios/astm/utm/prep_planners.md index 5ab857df78..e7c61aace3 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/prep_planners.md +++ b/monitoring/uss_qualifier/scenarios/astm/utm/prep_planners.md @@ -66,7 +66,9 @@ uss_qualifier verifies with the DSS that there are no operational intents remain #### 🛑 DSS responses check -**[astm.f3548.v21.DSS0005](../../../requirements/astm/f3548/v21.md)** +If the DSS fails to reply to a query concerning operational intent references in a given area, or fails to allow the deletion of +an operational intent from its own creator, it is in violation of **[astm.f3548.v21.DSS0005,1](../../../requirements/astm/f3548/v21.md)** +or **[astm.f3548.v21.DSS0005,2](../../../requirements/astm/f3548/v21.md)**, and this check will fail. #### 🛑 Area is clear check diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/test_steps.py b/monitoring/uss_qualifier/scenarios/astm/utm/test_steps.py index a760e0943b..bde5bb26da 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/test_steps.py +++ b/monitoring/uss_qualifier/scenarios/astm/utm/test_steps.py @@ -106,7 +106,7 @@ def _begin_step(self): len(oi_ids_delta) > 1 ): # TODO: could a USS cut up a submitted flight intent into several op intents? raise TestRunCannotContinueError( - f"unexpectedly got more than 1 new operational intent after planning request was created (IDs: {oi_ids_delta}): the test scenario might be malformed or some external requests might have interfered" + f"unexpectedly got more than 1 new operational intent reference after planning request was created (IDs: {oi_ids_delta}): the test scenario might be malformed or some external requests might have interfered" ) if len(oi_ids_delta) == 1: self._new_oi_ref = self._find_after_oi(oi_ids_delta.pop()) @@ -118,14 +118,14 @@ def _begin_step(self): with self._scenario.check("DSS responses", [self._dss.participant_id]) as check: if self._before_query.status_code != 200: check.record_failed( - summary="Failed to query DSS for operational intents before planning request", + summary="Failed to query DSS for operational intent references before planning request", severity=Severity.High, details=f"Received status code {self._before_query.status_code} from the DSS", query_timestamps=[self._before_query.request.timestamp], ) if self._after_query.status_code != 200: check.record_failed( - summary="Failed to query DSS for operational intents after planning request", + summary="Failed to query DSS for operational intent references after planning request", severity=Severity.High, details=f"Received status code {self._after_query.status_code} from the DSS", query_timestamps=[self._after_query.request.timestamp], diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/validate_not_shared_operational_intent.md b/monitoring/uss_qualifier/scenarios/astm/utm/validate_not_shared_operational_intent.md index af939a904c..fb3a89ef2e 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/validate_not_shared_operational_intent.md +++ b/monitoring/uss_qualifier/scenarios/astm/utm/validate_not_shared_operational_intent.md @@ -8,7 +8,8 @@ See `OpIntentValidator.expect_not_shared()` in [test_steps.py](test_steps.py). ## 🛑 DSS responses check -**[astm.f3548.v21.DSS0005](../../../requirements/astm/f3548/v21.md)** +If the DSS fails to reply to a query concerning operational intent references in a given area, +it is in violation of **[astm.f3548.v21.DSS0005,2](../../../requirements/astm/f3548/v21.md)**, and this check will fail. ## 🛑 Operational intent not shared check If there are new operational intent references in the area of the flight intent, this check will fail per diff --git a/monitoring/uss_qualifier/scenarios/astm/utm/validate_shared_operational_intent.md b/monitoring/uss_qualifier/scenarios/astm/utm/validate_shared_operational_intent.md index f7e77daf67..cf89e31a83 100644 --- a/monitoring/uss_qualifier/scenarios/astm/utm/validate_shared_operational_intent.md +++ b/monitoring/uss_qualifier/scenarios/astm/utm/validate_shared_operational_intent.md @@ -4,7 +4,8 @@ This step verifies that a created flight is shared properly per ASTM F3548-21 by ## 🛑 DSS responses check -**[astm.f3548.v21.DSS0005](../../../requirements/astm/f3548/v21.md)** +If the DSS fails to properly respond to a valid search query for operational intents in an area, +it is in violation of **[astm.f3548.v21.DSS0005,2](../../../requirements/astm/f3548/v21.md)**, and this check will fail. ## 🛑 Operational intent shared correctly check diff --git a/monitoring/uss_qualifier/scenarios/scenario.py b/monitoring/uss_qualifier/scenarios/scenario.py index ccdec94961..964ead7fdb 100644 --- a/monitoring/uss_qualifier/scenarios/scenario.py +++ b/monitoring/uss_qualifier/scenarios/scenario.py @@ -9,6 +9,7 @@ from monitoring import uss_qualifier as uss_qualifier_module from monitoring.monitorlib import fetch, inspection +from monitoring.monitorlib.errors import current_stack_string from monitoring.monitorlib.inspection import fullname from monitoring.uss_qualifier import scenarios as scenarios_module from monitoring.uss_qualifier.common_data_definitions import Severity @@ -363,6 +364,12 @@ def record_query(self, query: fetch.Query) -> None: self._expect_phase({ScenarioPhase.RunningTestStep, ScenarioPhase.CleaningUp}) if "queries" not in self._step_report: self._step_report.queries = [] + for existing_query in self._step_report.queries: + if query.request.timestamp == existing_query.request.timestamp: + logger.error( + f"The same query ({query.query_type} to {query.participant_id} at {query.request.timestamp}) was recorded multiple times. This is likely a bug in uss_qualifier at:\n{current_stack_string(2)}" + ) + return self._step_report.queries.append(query) participant = ( "UNKNOWN" diff --git a/monitoring/uss_qualifier/suites/astm/utm/f3548_21.md b/monitoring/uss_qualifier/suites/astm/utm/f3548_21.md index 69f6505dd9..6b63849983 100644 --- a/monitoring/uss_qualifier/suites/astm/utm/f3548_21.md +++ b/monitoring/uss_qualifier/suites/astm/utm/f3548_21.md @@ -31,20 +31,15 @@ Checked in - astm
.f3548
.v21 - DSS0005 - Implemented - ASTM F3548 flight planners preparation
Data Validation of GET operational intents by USS
Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted
Validation of operational intents - - + astm
.f3548
.v21 DSS0005,1 Implemented - ASTM F3548-21 UTM DSS Operational Intent Reference Access Control
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted + ASTM F3548 flight planners preparation
ASTM F3548-21 UTM DSS Operational Intent Reference Access Control
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted DSS0005,2 Implemented - ASTM F3548-21 UTM DSS Operational Intent Reference Access Control + ASTM F3548 flight planners preparation
ASTM F3548-21 UTM DSS Operational Intent Reference Access Control
Data Validation of GET operational intents by USS
Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted
Validation of operational intents DSS0100 diff --git a/monitoring/uss_qualifier/suites/faa/uft/message_signing.md b/monitoring/uss_qualifier/suites/faa/uft/message_signing.md index f1fc60c97e..3ff13eb844 100644 --- a/monitoring/uss_qualifier/suites/faa/uft/message_signing.md +++ b/monitoring/uss_qualifier/suites/faa/uft/message_signing.md @@ -18,20 +18,15 @@ Checked in - astm
.f3548
.v21 - DSS0005 - Implemented - ASTM F3548 flight planners preparation
Data Validation of GET operational intents by USS
Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted
Validation of operational intents - - + astm
.f3548
.v21 DSS0005,1 Implemented - ASTM F3548-21 UTM DSS Operational Intent Reference Access Control
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted + ASTM F3548 flight planners preparation
ASTM F3548-21 UTM DSS Operational Intent Reference Access Control
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted DSS0005,2 Implemented - ASTM F3548-21 UTM DSS Operational Intent Reference Access Control + ASTM F3548 flight planners preparation
ASTM F3548-21 UTM DSS Operational Intent Reference Access Control
Data Validation of GET operational intents by USS
Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted
Validation of operational intents DSS0100 diff --git a/monitoring/uss_qualifier/suites/uspace/flight_auth.md b/monitoring/uss_qualifier/suites/uspace/flight_auth.md index 21ea7f935c..e94aec6801 100644 --- a/monitoring/uss_qualifier/suites/uspace/flight_auth.md +++ b/monitoring/uss_qualifier/suites/uspace/flight_auth.md @@ -19,20 +19,15 @@ Checked in - astm
.f3548
.v21 - DSS0005 - Implemented - ASTM F3548 flight planners preparation
Data Validation of GET operational intents by USS
Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted
Validation of operational intents - - + astm
.f3548
.v21 DSS0005,1 Implemented - ASTM F3548-21 UTM DSS Operational Intent Reference Access Control
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted + ASTM F3548 flight planners preparation
ASTM F3548-21 UTM DSS Operational Intent Reference Access Control
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted DSS0005,2 Implemented - ASTM F3548-21 UTM DSS Operational Intent Reference Access Control + ASTM F3548 flight planners preparation
ASTM F3548-21 UTM DSS Operational Intent Reference Access Control
Data Validation of GET operational intents by USS
Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted
Validation of operational intents DSS0100 diff --git a/monitoring/uss_qualifier/suites/uspace/required_services.md b/monitoring/uss_qualifier/suites/uspace/required_services.md index c4a3ed480a..b1399f5b71 100644 --- a/monitoring/uss_qualifier/suites/uspace/required_services.md +++ b/monitoring/uss_qualifier/suites/uspace/required_services.md @@ -454,20 +454,15 @@ ASTM NetRID DSS: Concurrent Requests
ASTM NetRID DSS: ISA Expiry
ASTM NetRID DSS: ISA Subscription Interactions
ASTM NetRID DSS: Simple ISA
ASTM NetRID DSS: Submitted ISA Validations
ASTM NetRID DSS: Subscription Simple
ASTM NetRID DSS: Subscription Validation
ASTM NetRID DSS: Token Validation - astm
.f3548
.v21 - DSS0005 - Implemented - ASTM F3548 flight planners preparation
Data Validation of GET operational intents by USS
Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted
Validation of operational intents - - + astm
.f3548
.v21 DSS0005,1 Implemented - ASTM F3548-21 UTM DSS Operational Intent Reference Access Control
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted + ASTM F3548 flight planners preparation
ASTM F3548-21 UTM DSS Operational Intent Reference Access Control
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted DSS0005,2 Implemented - ASTM F3548-21 UTM DSS Operational Intent Reference Access Control + ASTM F3548 flight planners preparation
ASTM F3548-21 UTM DSS Operational Intent Reference Access Control
Data Validation of GET operational intents by USS
Nominal planning: conflict with higher priority
Nominal planning: not permitted conflict with equal priority
Off-Nominal planning: down USS
Off-Nominal planning: down USS with equal priority conflicts not permitted
Validation of operational intents DSS0100