Add empty scenarios for RID documentation

monitoring/uss_qualifier/action_generators/README.md

@@ -0,0 +1,5 @@
+# Action generators
+
+The bulk of uss_qualifier's automated testing logic is contained in [test scenarios](../scenarios/README.md).  A [test suite](../suites/README.md) is essentially a static "playlist" of test actions to perform (test scenarios, action generators, and other test suites), all of which ultimately resolve to test scenarios.  An action generator is essentially a dynamic "playlist" of test actions -- it can generate test actions that vary according to provided resource values, situations, or other conditions only necessarily known at runtime.
+
+For documentation purposes, all action generators must statically declare the test actions they may take.  However, whether each (or any) of these actions will actually be taken at runtime cannot be statically determined in general.

monitoring/uss_qualifier/scenarios/astm/dss/crdb_access.py

@@ -0,0 +1,9 @@
+from monitoring.uss_qualifier.scenarios.scenario import GenericTestScenario
+
+
+class CRDBAccess(GenericTestScenario):
+    def __init__(self):
+        super().__init__()
+
+    def run(self):
+        pass  # TODO: Implement

monitoring/uss_qualifier/scenarios/astm/netrid/common/operator_interactions.py

@@ -0,0 +1,9 @@
+from monitoring.uss_qualifier.scenarios.scenario import GenericTestScenario
+
+
+class OperatorInteractions(GenericTestScenario):
+    def __init__(self):
+        super().__init__()
+
+    def run(self):
+        pass  # TODO: Implement

monitoring/uss_qualifier/scenarios/astm/netrid/v19/__init__.py

@@ -2,3 +2,4 @@
 from .nominal_behavior import NominalBehavior
 from .misbehavior import Misbehavior
 from .aggregate_checks import AggregateChecks
+from .operator_interactions import OperatorInteractions

monitoring/uss_qualifier/scenarios/astm/netrid/v19/dss/__init__.py

@@ -1 +1,2 @@
 from .subscription_validation import SubscriptionValidation
+from .crdb_access import CRDBAccess

monitoring/uss_qualifier/scenarios/astm/netrid/v19/dss/crdb_access.md

@@ -0,0 +1,37 @@
+# ASTM NetRID DSS: Direct CRDB access test scenario
+
+## Overview
+
+Attempt to directly access the CockroachDB (CRDB) nodes intercommunicating to form the DSS Airspace Representation for the DSS instances under test, for the purpose of determining compliance to certain DSS interoperability requirements.
+
+Note that none of this scenario is implemented yet.
+
+## Resources
+
+### crdb_nodes
+
+Set of CockroachDB nodes constituting the DSS instances under test.
+
+TODO: Create this resource
+
+## Verify security interoperability test case
+
+### Attempt unauthorized access test step
+
+In this test step, uss_qualifier attempts to connect to each CRDB node in insecure mode.
+
+#### CRDB node in insecure mode check
+
+If connection to a CRDB node in insecure mode succeeds, the USS will have failed to authenticate clients per **[astm.f3411.v19.DSS0110](../../../../../requirements/astm/f3411/v19.md)**.
+
+TODO: Implement this step and check
+
+### Verify encryption test step
+
+This step verifies the use of TLS for every CRDB node specified.
+
+#### TLS in use check
+
+If a CRDB node does not have TLS in use, the test will have failed to verify the encryption requirement **[astm.f3411.v19.DSS0120](../../../../../requirements/astm/f3411/v19.md)**.
+
+TODO: Implement this step and check

monitoring/uss_qualifier/scenarios/astm/netrid/v19/dss/crdb_access.py

@@ -0,0 +1,8 @@
+from monitoring.uss_qualifier.scenarios.astm.dss.crdb_access import (
+    CRDBAccess as CommonCRDBAccess,
+)
+from monitoring.uss_qualifier.scenarios.scenario import TestScenario
+
+
+class CRDBAccess(TestScenario, CommonCRDBAccess):
+    pass

monitoring/uss_qualifier/scenarios/astm/netrid/v19/operator_interactions.md

@@ -0,0 +1,95 @@
+# ASTM NetRID: Operator interactions test scenario
+
+## Overview
+
+Set up situations in which operator interactions (notifications) are required, and verify that those notifications are observed for the USS under test.
+
+Note that none of this scenario is implemented yet.
+
+## Resources
+
+### service_provider
+
+A singular `NetRIDServiceProvider` to be tested via the injection of RID flight data.
+
+TODO: Create this resource
+
+### operator_notifications
+
+Means by which to ask "What user/operator notifications have been observed for user/operator X (from the USS under test) over time period Z".
+
+TODO: Create this resource
+
+### flights_data
+
+A [`FlightDataResource`](../../../../resources/netrid/flight_data.py) containing 1 flight.  This flight must:
+* (Phase 1): Start out nominal
+* (Phase 2): Then contain a pause (lack of telemetry) sufficient to trigger NET0040
+* (Phase 3): Then resume nominal telemetry
+* (Phase 4): Then contain insufficient data to trigger NET0030
+
+### orchestrated_dss
+
+A DSS instance that is equipped to fail on command, and will be used by the USS under test.
+
+TODO: Create this resource
+
+## Failed ISA test case
+
+### Verify no ISAs test step
+
+uss_qualifier checks the DSS to ensure that the Service Provider under test does not have any ISAs in the system.  If ISAs are present, the Service Provider is instructed to clear the area of active flights, after which uss_qualifier reverifies the absence of ISAs.
+
+### Disable DSS test step
+
+uss_qualifier commands the orchestrated DSS to fail when interacting with normal clients.
+
+### Enumerate pre-existing operator notifications test step
+
+uss_qualifier retrieves the current (pre-existing) set of operator notifications.
+
+### Inject flight test step
+
+uss_qualifier attempts to inject a flight into the Service Provider under test, knowing that the Service Provider will not be able to create an ISA.
+
+#### Flight failed check
+
+Since the DSS is known to fail when attempting to create an ISA, if the Service Provider successfully creates the flight, they will have not met **[astm.f3411.v19.NET0610](../../../../requirements/astm/f3411/v19.md)**.
+
+TODO: Implement
+
+### Enumerate operator notifications test step
+
+uss_qualifier retrieves the current (after failed flight) set of operator notifications.
+
+#### Operator notified of discoverability failure check
+
+The "after" set of operator notifications should contain at least one more entry than the "before" set of operator notifications.  If there was no new operator notification, the Service Provider will not have met **[astm.f3411.v19.NET0620](../../../../requirements/astm/f3411/v19.md)**.
+
+TODO: Implement
+
+## In-flight notifications test case
+
+### Inject flight test step
+
+uss_qualifier injects the flight into the Service Provider under test with the intention of observing the whole flight.
+
+### Enumerate pre-existing operator notifications test step
+
+uss_qualifier retrieves the current (pre-existing) set of operator notifications.
+
+### Poll Service Provider test step
+
+Throughout the duration of the flight, uss_qualifier periodically polls for operator notifications.
+
+#### Insufficient telemetry operator notification check
+
+If the Service Provider under test does not provide an operator notification in Phase 2 (regarding the telemetry feed stopping), it will not have complied with **[astm.f3411.v19.NET0040](../../../../requirements/astm/f3411/v19.md)**.
+
+TODO: Implement
+
+#### Missing data operator notification check
+
+If the Service Provider under test does not provide an operator notification in Phase 4 (regarding missing data fields in reported telemetry), it will not have complied with **[astm.f3411.v19.NET0030](../../../../requirements/astm/f3411/v19.md)**.
+
+TODO: Implement

monitoring/uss_qualifier/scenarios/astm/netrid/v19/operator_interactions.py

@@ -0,0 +1,8 @@
+from monitoring.uss_qualifier.scenarios.astm.netrid.common.operator_interactions import (
+    OperatorInteractions as CommonOperatorInteractions,
+)
+from monitoring.uss_qualifier.scenarios.scenario import TestScenario
+
+
+class OperatorInteractions(TestScenario, CommonOperatorInteractions):
+    pass

monitoring/uss_qualifier/scenarios/astm/netrid/v22a/__init__.py

@@ -2,3 +2,4 @@
 from .nominal_behavior import NominalBehavior
 from .misbehavior import Misbehavior
 from .aggregate_checks import AggregateChecks
+from .operator_interactions import OperatorInteractions

monitoring/uss_qualifier/scenarios/astm/netrid/v22a/dss/__init__.py

@@ -1,2 +1,3 @@
 from .isa_simple import ISASimple
 from .subscription_validation import SubscriptionValidation
+from .crdb_access import CRDBAccess

monitoring/uss_qualifier/scenarios/astm/netrid/v22a/dss/crdb_access.md

@@ -0,0 +1,37 @@
+# ASTM NetRID DSS: Direct CRDB access test scenario
+
+## Overview
+
+Attempt to directly access the CockroachDB (CRDB) nodes intercommunicating to form the DSS Airspace Representation for the DSS instances under test, for the purpose of determining compliance to certain DSS interoperability requirements.
+
+Note that none of this scenario is implemented yet.
+
+## Resources
+
+### crdb_nodes
+
+Set of CockroachDB nodes constituting the DSS instances under test.
+
+TODO: Create this resource
+
+## Verify security interoperability test case
+
+### Attempt unauthorized access test step
+
+In this test step, uss_qualifier attempts to connect to each CRDB node in insecure mode.
+
+#### CRDB node in insecure mode check
+
+If connection to a CRDB node in insecure mode succeeds, the USS will have failed to authenticate clients per **[astm.f3411.v22a.DSS0110](../../../../../requirements/astm/f3411/v22a.md)**.
+
+TODO: Implement this step and check
+
+### Verify encryption test step
+
+This step verifies the use of TLS for every CRDB node specified.
+
+#### TLS in use check
+
+If a CRDB node does not have TLS in use, the test will have failed to verify the encryption requirement **[astm.f3411.v22a.DSS0120](../../../../../requirements/astm/f3411/v22a.md)**.
+
+TODO: Implement this step and check

monitoring/uss_qualifier/scenarios/astm/netrid/v22a/dss/crdb_access.py

@@ -0,0 +1,8 @@
+from monitoring.uss_qualifier.scenarios.astm.dss.crdb_access import (
+    CRDBAccess as CommonCRDBAccess,
+)
+from monitoring.uss_qualifier.scenarios.scenario import TestScenario
+
+
+class CRDBAccess(TestScenario, CommonCRDBAccess):
+    pass

monitoring/uss_qualifier/scenarios/astm/netrid/v22a/operator_interactions.md

@@ -0,0 +1,95 @@
+# ASTM NetRID: Operator interactions test scenario
+
+## Overview
+
+Set up situations in which operator interactions (notifications) are required, and verify that those notifications are observed for the USS under test.
+
+Note that none of this scenario is implemented yet.
+
+## Resources
+
+### service_provider
+
+A singular `NetRIDServiceProvider` to be tested via the injection of RID flight data.
+
+TODO: Create this resource
+
+### operator_notifications
+
+Means by which to ask "What user/operator notifications have been observed for user/operator X (from the USS under test) over time period Z".
+
+TODO: Create this resource
+
+### flights_data
+
+A [`FlightDataResource`](../../../../resources/netrid/flight_data.py) containing 1 flight.  This flight must:
+* (Phase 1): Start out nominal
+* (Phase 2): Then contain a pause (lack of telemetry) sufficient to trigger NET0040
+* (Phase 3): Then resume nominal telemetry
+* (Phase 4): Then contain insufficient data to trigger NET0030
+
+### orchestrated_dss
+
+A DSS instance that is equipped to fail on command, and will be used by the USS under test.
+
+TODO: Create this resource
+
+## Failed ISA test case
+
+### Verify no ISAs test step
+
+uss_qualifier checks the DSS to ensure that the Service Provider under test does not have any ISAs in the system.  If ISAs are present, the Service Provider is instructed to clear the area of active flights, after which uss_qualifier reverifies the absence of ISAs.
+
+### Disable DSS test step
+
+uss_qualifier commands the orchestrated DSS to fail when interacting with normal clients.
+
+### Enumerate pre-existing operator notifications test step
+
+uss_qualifier retrieves the current (pre-existing) set of operator notifications.
+
+### Inject flight test step
+
+uss_qualifier attempts to inject a flight into the Service Provider under test, knowing that the Service Provider will not be able to create an ISA.
+
+#### Flight failed check
+
+Since the DSS is known to fail when attempting to create an ISA, if the Service Provider successfully creates the flight, they will have not met **[astm.f3411.v22a.NET0610](../../../../requirements/astm/f3411/v22a.md)**.
+
+TODO: Implement
+
+### Enumerate operator notifications test step
+
+uss_qualifier retrieves the current (after failed flight) set of operator notifications.
+
+#### Operator notified of discoverability failure check
+
+The "after" set of operator notifications should contain at least one more entry than the "before" set of operator notifications.  If there was no new operator notification, the Service Provider will not have met **[astm.f3411.v22a.NET0620](../../../../requirements/astm/f3411/v22a.md)**.
+
+TODO: Implement
+
+## In-flight notifications test case
+
+### Inject flight test step
+
+uss_qualifier injects the flight into the Service Provider under test with the intention of observing the whole flight.
+
+### Enumerate pre-existing operator notifications test step
+
+uss_qualifier retrieves the current (pre-existing) set of operator notifications.
+
+### Poll Service Provider test step
+
+Throughout the duration of the flight, uss_qualifier periodically polls for operator notifications.
+
+#### Insufficient telemetry operator notification check
+
+If the Service Provider under test does not provide an operator notification in Phase 2 (regarding the telemetry feed stopping), it will not have complied with **[astm.f3411.v22a.NET0040](../../../../requirements/astm/f3411/v22a.md)**.
+
+TODO: Implement
+
+#### Missing data operator notification check
+
+If the Service Provider under test does not provide an operator notification in Phase 4 (regarding missing data fields in reported telemetry), it will not have complied with **[astm.f3411.v22a.NET0030](../../../../requirements/astm/f3411/v22a.md)**.
+
+TODO: Implement

monitoring/uss_qualifier/scenarios/astm/netrid/v22a/operator_interactions.py

@@ -0,0 +1,8 @@
+from monitoring.uss_qualifier.scenarios.astm.netrid.common.operator_interactions import (
+    OperatorInteractions as CommonOperatorInteractions,
+)
+from monitoring.uss_qualifier.scenarios.scenario import TestScenario
+
+
+class OperatorInteractions(TestScenario, CommonOperatorInteractions):
+    pass

monitoring/uss_qualifier/suites/README.md

@@ -1,9 +1,37 @@
 # USS Qualifier test suites
 
-A test suite is a set of tests that establish compliance to the thing they're named after.  Example: Passing the "ASTM F3548-21" test suite should indicate the systems under test are compliant with ASTM F3548-21.
+A test suite is a set of tests that establish compliance to the thing they're named after; it may be thought of as a "playlist" of test scenarios.  Example: Passing the "ASTM F3548-21" test suite should indicate the systems under test are compliant with ASTM F3548-21.
 
-A test suite is composed of a list of {test suite|test scenario}; each element on the list is executed sequentially.
+A test suite is composed of a list of actions, each one being a [test scenario](../scenarios/README.md), [action generator](../action_generators/README.md), or another test suite.  Each action on the list is executed sequentially.
 
 A test suite is defined with a YAML file following the [`TestSuiteDefinition` schema](definitions.py).
 
+## Documentation
+
 Test suite documentation is generated automatically; use `make format` from the repository root to regenerate it.
+
+### Actions
+
+This section of a test suite documentation page summarizes the actions that will be taken when the test suite is run.
+
+### Checked requirements
+
+This section of a test suite documentation page summarizes the requirements the suite may be capable of checking.  A requirement is "checked" if there is [a declared test check](../scenarios/README.md#checks) with [documentation](../scenarios/README.md#test-checks) indicating that a failure of that check would imply a failure of the requirement.  A particular test run may not perform all potential checks.  For instance, if a test step is performed only in certain situations (e.g., a provided resource has certain characteristics) and skipped otherwise, then the checks associated with that test step will not be performed in that test run.  All checks that are performed will be recorded in the report as passed or failed.
+
+The columns of the table in this section are as follows:
+
+#### Package
+
+All [requirements defined in this monitoring repository](../requirements/README.md) are defined exactly once in a particular "package" (InterUSS organizational structure to keep track of requirements) which represents the canonical source of that requirement.  This column specifies and links to the package in which the requirement is defined in this monitoring repository.
+
+#### Requirement
+
+This column contains the name of the requirement in the [package](#package) namespace.  Since periods are not allowed in requirement names, when InterUSS finds it helpful to refer to a particular portion of a named requirement, the name and portion(s) are usually delimited with commas.  For instance, if a document specified REQ003 and that requirement had three distinct parts (a, b, and c), one requirement name might be `REQ003,a`.
+
+#### Status
+
+Test check documentation may include a `TODO:` note indicating that the check (and perhaps the containing test step) has not been fully implemented.  If there is no such note for any of the checks relevant to this requirement, this column will indicate "Implemented".  If all checks relevant to this requirement have this indication, this column will indicate "TODO".  If some checks have this indication and others do not, this column will indicate "Implemented + TODO".  A requirement is only listed in this table if the test suite may cause a check to happen that is associated with that requirement.
+
+#### Checked in
+
+This column includes a list of test scenarios containing checks associated with the corresponding requirement.  Only test scenarios that may be run as part of the test suite are included.