[uss_qualifier] Rename "badge" to "capability" #96

monitoring/uss_qualifier/README.md

@@ -33,10 +33,10 @@ At this point, uss_qualifier can be run again with a different configuration tar
 * [Test configurations](configurations/README.md)
 * [Test resources](resources/README.md)
 
-### Badges
+### Verifiable capabilities
 
-A test suite may define one or more "badges" and the criteria necessary for a participant to earn them.  Badges are intended to be used to indicate whether a participant has satisfied the requirements for an optional capability.
+A test suite may define one or more verifiable "capabilities" and the criteria necessary to verify them for a participant.  These verifiable capabilities are intended to be used to indicate whether a participant has satisfied the requirements for an optional capability.
 
-For instance, ASTM F3411-22a network remote identification includes the concept of an "intent-based network participant" who provides information about their overall flight intent, but not real-time telemetry.  A NetRID Service Provider is not required to support intent-based network participants, but if they do choose to support them, then they must follow a few requirements.  In this case, an "intent-based network participant support" badge may be defined to be granted only if uss_qualifier confirms compliance with the requirements regarding intent-based network participants.  If compliance to the intent-based network participants cannot be confirmed, the NetRID Service Provider under test may still be fully compliant with the standard; they just elected not to support this particular optional capability and would therefore be standard-compliant without earning the "intent-based network participant support" badge.
+For instance, ASTM F3411-22a network remote identification includes the concept of an "intent-based network participant" who provides information about their overall flight intent, but not real-time telemetry.  A NetRID Service Provider is not required to support intent-based network participants, but if they do choose to support them, then they must follow a few requirements.  In this case, an "intent-based network participant support" capability may be defined to be verified only if uss_qualifier confirms compliance with the requirements regarding intent-based network participants.  If compliance to the intent-based network participants cannot be confirmed, the NetRID Service Provider under test may still be fully compliant with the standard; they just elected not to support this particular optional capability and would therefore be standard-compliant without verifying the "intent-based network participant support" capability.
 
-As another example, ASTM F3411-22a does not require a NetRID Display Provider to provide Display Application clients with the serial number of an identified aircraft.  However, a particular jurisdiction may accept F3411-22a remote identification as a means of compliance only if Display Applications are capable of providing the viewer with the aircraft serial number.  In this case, the ASTM F3411-22a test suite could define a "serial number" badge indicating that a Display Provider provides the correct aircraft serial number to its Display Application clients, and then the test suite for the jurisdiction may define a "qualified in jurisdiction" badge which is not granted unless the ASTM F3411-22a test suite's "serial number" badge is granted (among other criteria).
+As another example, ASTM F3411-22a does not require a NetRID Display Provider to provide Display Application clients with the serial number of an identified aircraft.  However, a particular jurisdiction may accept F3411-22a remote identification as a means of compliance only if Display Applications are capable of providing the viewer with the aircraft serial number.  In this case, the ASTM F3411-22a test suite could define a "serial number" capability indicating that a Display Provider provides the correct aircraft serial number to its Display Application clients, and then the test suite for the jurisdiction may define a "qualified in jurisdiction" capability which is not verified unless the ASTM F3411-22a test suite's "serial number" capability is verified (among other criteria).

monitoring/uss_qualifier/reports/badges.py → monitoring/uss_qualifier/reports/capabilities.py

@@ -3,14 +3,14 @@
 import bc_jsonpath_ng.ext
 
 from monitoring.uss_qualifier.configurations.configuration import ParticipantID
-from monitoring.uss_qualifier.reports.badge_definitions import (
+from monitoring.uss_qualifier.reports.capability_definitions import (
     AllConditions,
     SpecificCondition,
     AnyCondition,
     NoFailedChecksCondition,
     RequirementsCheckedCondition,
-    BadgeGrantedCondition,
-    BadgeGrantCondition,
+    CapabilityVerifiedCondition,
+    CapabilityVerificationCondition,
 )
 from monitoring.uss_qualifier.reports.report import TestSuiteReport
 from monitoring.uss_qualifier.requirements.definitions import RequirementID
@@ -19,36 +19,36 @@
 )
 
 SpecificConditionType = TypeVar("SpecificConditionType", bound=SpecificCondition)
-_badge_condition_evaluators: Dict[
+_capability_condition_evaluators: Dict[
     Type, Callable[[SpecificConditionType, ParticipantID, TestSuiteReport], bool]
 ] = {}
 
 
-def badge_condition_evaluator(condition_type: Type):
-    """Decorator to label a function that evaluates a specific condition for granting a badge.
+def capability_condition_evaluator(condition_type: Type):
+    """Decorator to label a function that evaluates a specific condition for verifying a capability.
 
     Args:
-        condition_type: A Type that inherits from badge_definitions.SpecificCondition.
+        condition_type: A Type that inherits from capability_definitions.SpecificCondition.
     """
 
     def register_evaluator(func):
-        _badge_condition_evaluators[condition_type] = func
+        _capability_condition_evaluators[condition_type] = func
         return func
 
     return register_evaluator
 
 
 def condition_satisfied_for_test_suite(
-    grant_condition: BadgeGrantCondition,
+    grant_condition: CapabilityVerificationCondition,
     participant_id: ParticipantID,
     report: TestSuiteReport,
 ) -> bool:
-    """Determine if a condition for granting a badge is satisfied based on a Test Suite report.
+    """Determine if a condition for verifying a capability is satisfied based on a Test Suite report.
 
     Args:
-        grant_condition: Badge-granting condition to check.
-        participant_id: Participant for which the badge would be granted.
-        report: Test Suite report upon which the badge (and grant condition) are based.
+        grant_condition: Capability-verifying condition to check.
+        participant_id: Participant for which the capability would be verified.
+        report: Test Suite report upon which the capability (and verification condition) are based.
 
     Returns: True if the condition was satisfied, False if not.
     """
@@ -59,15 +59,15 @@ def condition_satisfied_for_test_suite(
     ]
     if not populated_fields:
         raise ValueError(
-            "No specific condition specified for grant_condition in BadgeGrantCondition"
+            "No specific condition specified for grant_condition in CapabilityVerificationCondition"
         )
     if len(populated_fields) > 1:
         raise ValueError(
-            "Multiple conditions specified for grant_condition in BadgeGrantCondition: "
+            "Multiple conditions specified for grant_condition in CapabilityVerificationCondition: "
             + ", ".join(populated_fields)
         )
     specific_condition = grant_condition[populated_fields[0]]
-    condition_evaluator = _badge_condition_evaluators.get(
+    condition_evaluator = _capability_condition_evaluators.get(
         type(specific_condition), None
     )
     if condition_evaluator is None:
@@ -77,7 +77,7 @@ def condition_satisfied_for_test_suite(
     return condition_evaluator(specific_condition, participant_id, report)
 
 
-@badge_condition_evaluator(AllConditions)
+@capability_condition_evaluator(AllConditions)
 def evaluate_all_conditions_condition(
     condition: AllConditions, participant_id: ParticipantID, report: TestSuiteReport
 ) -> bool:
@@ -87,7 +87,7 @@ def evaluate_all_conditions_condition(
     return True
 
 
-@badge_condition_evaluator(AnyCondition)
+@capability_condition_evaluator(AnyCondition)
 def evaluate_any_condition_condition(
     condition: AnyCondition, participant_id: ParticipantID, report: TestSuiteReport
 ) -> bool:
@@ -97,7 +97,7 @@ def evaluate_any_condition_condition(
     return False
 
 
-@badge_condition_evaluator(NoFailedChecksCondition)
+@capability_condition_evaluator(NoFailedChecksCondition)
 def evaluate_no_failed_checks_condition(
     condition: NoFailedChecksCondition,
     participant_id: ParticipantID,
@@ -108,7 +108,7 @@ def evaluate_no_failed_checks_condition(
     return True
 
 
-@badge_condition_evaluator(RequirementsCheckedCondition)
+@capability_condition_evaluator(RequirementsCheckedCondition)
 def evaluate_requirements_checked_conditions(
     condition: RequirementsCheckedCondition,
     participant_id: ParticipantID,
@@ -124,19 +124,21 @@ def evaluate_requirements_checked_conditions(
     return all(req_checked.values())
 
 
-@badge_condition_evaluator(BadgeGrantedCondition)
-def evaluate_badge_granted_condition(
-    condition: BadgeGrantedCondition,
+@capability_condition_evaluator(CapabilityVerifiedCondition)
+def evaluate_capability_verified_condition(
+    condition: CapabilityVerifiedCondition,
     participant_id: ParticipantID,
     report: TestSuiteReport,
 ) -> bool:
-    path = condition.badge_location if "badge_location" in condition else "$"
+    path = condition.capability_location if "capability_location" in condition else "$"
     matching_reports = bc_jsonpath_ng.ext.parse(path).find(report)
     result = False
     for matching_report in matching_reports:
         if isinstance(matching_report.value, TestSuiteReport):
-            badges = matching_report.value.badges_granted.get(participant_id, set())
-            if condition.badge_id in badges:
+            capabilities = matching_report.value.capabilities_verified.get(
+                participant_id, set()
+            )
+            if condition.capability_id in capabilities:
                 result = True
             else:
                 return False

monitoring/uss_qualifier/reports/capability_definitions.py

@@ -0,0 +1,93 @@
+from __future__ import annotations
+from typing import Optional, List
+
+from implicitdict import ImplicitDict
+from monitoring.uss_qualifier.requirements.definitions import RequirementCollection
+
+
+CapabilityID = str
+"""Identifier of a capability that uss_qualifier can verify."""
+
+JSONPathExpression = str
+"""JsonPath expression; see https://pypi.org/project/jsonpath-ng/"""
+
+
+class SpecificCondition(ImplicitDict):
+    pass
+
+
+class AllConditions(SpecificCondition):
+    """Condition will only be satisfied when all specified conditions are satisfied."""
+
+    conditions: List[CapabilityVerificationCondition]
+
+
+class AnyCondition(SpecificCondition):
+    """Condition will be satisfied when any of the specified conditions are satisfied."""
+
+    conditions: List[CapabilityVerificationCondition]
+
+
+class RequirementsCheckedCondition(SpecificCondition):
+    """Condition will only be satisfied if at least one successful check exists for all specified requirements."""
+
+    checked: RequirementCollection
+    """Each requirement contained within this collection must be covered by at least one successful check."""
+
+
+class NoFailedChecksCondition(SpecificCondition):
+    """Condition will only be satisfied if there are no applicable failed checks.
+
+    For a capability to be verified for a participant, only checks including the participant's ID will be considered."""
+
+    pass
+
+
+class CapabilityVerifiedCondition(SpecificCondition):
+    """Condition will be satisfied when the specified capability is verified."""
+
+    capability_id: CapabilityID
+    """Identifier of capability that must be verified for this condition to be satisfied."""
+
+    capability_location: Optional[JSONPathExpression]
+    """Location of report to inspect for the verification of the specified capability, relative to the report in which
+    the capability is defined.  Implicit default value is "$" (look for verified capability in the report in which the
+    dependant capability is defined).
+
+    If this location resolves to multiple TestSuiteReports, then the capability must be verified in all resolved reports
+    in order for this condition to be satisfied.  When this location resolves to artifacts that are not
+    TestSuiteReports, those artifacts will be ignored.
+
+    Note that capabilities are verified in the order they are defined.  So, if capability B defined in a particular
+    location depends on whether capability A in that same location is granted, capability A must be defined before
+    capability B is defined.  Also note that capability verifications are computed as test components are completed.
+    Since a parent test component (e.g., test suite) is not complete until all of its child components are complete, a
+    descendant test component's capability condition cannot depend on whether an ancestor's (e.g., parent's) capability
+    is verified.
+    """
+
+
+class CapabilityVerificationCondition(ImplicitDict):
+    """Specification of a single condition used to determine whether a capability should be verified.
+
+    Exactly one field must be specified."""
+
+    all_conditions: Optional[AllConditions]
+    any_conditions: Optional[AnyCondition]
+    no_failed_checks: Optional[NoFailedChecksCondition]
+    requirements_checked: Optional[RequirementsCheckedCondition]
+    capability_verified: Optional[CapabilityVerifiedCondition]
+
+
+class ParticipantCapabilityDefinition(ImplicitDict):
+    id: CapabilityID
+    """Identifier of this capability, unique at the level in which this capability is defined."""
+
+    name: str
+    """Human-readable name of the capability."""
+
+    description: str
+    """Human-readable description of the capability."""
+
+    verification_condition: CapabilityVerificationCondition
+    """Condition required in order to verify the capability."""

monitoring/uss_qualifier/reports/report.py

@@ -11,7 +11,7 @@
     ParticipantID,
 )
 from monitoring.uss_qualifier.fileio import FileReference
-from monitoring.uss_qualifier.reports.badge_definitions import BadgeID
+from monitoring.uss_qualifier.reports.capability_definitions import CapabilityID
 from monitoring.uss_qualifier.requirements.definitions import RequirementID
 
 
@@ -387,8 +387,8 @@ class TestSuiteReport(ImplicitDict):
     successful: bool = False
     """True iff test suite completed normally with no failed checks"""
 
-    badges_granted: Optional[Dict[ParticipantID, List[BadgeID]]]
-    """If badges are defined by this suite, the list of badges earned by each participant."""
+    capabilities_verified: Optional[Dict[ParticipantID, List[CapabilityID]]]
+    """If capabilities are defined by this suite, the list of capabilities verified for each participant."""
 
     def has_critical_problem(self) -> bool:
         return any(a.has_critical_problem() for a in self.actions)