-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add telemetry to flight information #16
base: main
Are you sure you want to change the base?
Conversation
f3eb63e
to
7788f1d
Compare
I agree with the bulk-injection approach, that does seem to be the most reasonable one to me. I think that synchronous injection would introduce too many potential issues in the process of the testing itself that would be difficult to discriminate from issues with an USS not acting as it should. There is one point that I feel might be problematic, which is the more fundamental difference between the flight plan itself and the (telemetry of the) flight.
If that is really an issue for some USSes, maybe that could be alleviated by implementing an optional and separate tool that would do that for them? For which the execution would be completely independent from the execution of the test scenario. |
Thanks Ben for adding telemetry. Its not clear which uas_state and usage_state would the telemetry be sent in the flight_plan? How would USS need to handle telemetry info sent in various UpsertFlightPlan requests for a single flight_plan_id? What would be a trigger for a USS to start the telemetry? What would the uas_state and the utm state be, when the telemetry is started? Could we have a sequence diagram for the test of a flight plan that has telemetry and goes from Planned to OkToFly and OffNominal state (NonConforming and Contingent due to position-report)? I agree with Mike's comment that some USSes may have issues implementing telemetry queuing and delivery system. So, it would be good if the test driver can provide some means of sending telemetry for such USSes. And I think test driver will hav e more control on the test sequences if the telemetry is sent synchronously by the test driver. |
flight_planning/v1/telemetry.yaml
Outdated
AircraftState: | ||
description: State of an aircraft for the purposes of simulating the execution of a flight plan. | ||
required: | ||
- timestamp |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason that ONLY the timestamp is required? With no other elements, do we really have any state information that is reportable? Only viewing in GitHub review editor... if I missed something in regards to required elements, just LMK.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This interface is essentially puppeteering a user of the USS's system. It seems likely that we may want to simulate loss of one or more telemetry components at some point, but it seems like we would always know when the injected telemetry should be injected into the system under test.
For instance, we don't want to make position
mandatory because then we would be precluded from testing a scenario where an aircraft lost position information but was still able to report operational status. Do you have suggestions on what other fields should be mandatory? Remember, mandatory here means that uss_qualifier must always inform the USS's virtual user being puppeteered of the value of that field in every test ever conducted.
states: | ||
description: >- | ||
The set of telemetry data that should be injected into the system (as if reported by the user or the user's system) at the appropriate times (and not before) for this flight. | ||
type: array |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we have a min/max elements? I think generally good to upper bound the array size when possible/reasonable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this is data we're going to have appear to originate from a user of the USS's system, it seems like we wouldn't want to set min/max because that would preclude testing how the USS would respond if the user provided values that were outside those reasonable bounds in their telemetry.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wouldn't have made the assumption that these are data that are just sent raw to the USS... the API between the client and USS may very well not take ANY array of position data, let alone one with a limit. I'm just thinking from a data security aspect... if we want more predicability in payloads or not. If it isn't a security or resource concern, then fine not to have the upper limit. But, again, I wouldn't assume the API between a client and a USS at all... the USS under test needs to deal with this array as needed.
type: string | ||
example: Airborne | ||
default: Undeclared | ||
AircraftPosition: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we re-use/reference the utm.yaml for VehicleTelemetry? https://github.com/astm-utm/Protocol/blob/cb7cf962d3a0c01b5ab12502f5f54789624977bf/utm.yaml#L1232
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, certainly not -- this interface is providing inputs to the USS-under-test's user interface. There is no reason those inputs would exactly match the format of ASTM F3548-21, and we need to be able to iterate on this interface independent of standards development. We actually already made the mistake of tying the original RID injection interface to F3411-19 data objects and we were forced to fix that mistake when we needed to test some things that were different in F3411-22a.
Additionally, some standard-defined data elements may not be feasible to obtain from the user (instead must be calculated or inferred from user-provided data) and in those cases we would not want to inject something that could not actually be provided to the true system via the normal user interface.
flight_planning/v1/telemetry.yaml
Outdated
format: float | ||
minimum: 0 | ||
example: 1 | ||
default: 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would suggest making the min, example, and default be more "floaty" for clarity. 0.0, 1.2, 0.0 for example.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure; sounds good.
Thanks for the initial pro/con breakdown of the options! I understand the reasoning in the bulk upload and why it is the easiest to get going from the uss_qualifier perspective. I like the option proposed by @mickmis to provide a separate tool to help USS's properly inject telemetry into their own systems, using the bulk data provided via this interface. The major drawback I think is the increased effort on service implementers to implement that functionality if they don't have it. The logical errors there seem more problematic to me than the synchronous approach's potential networking problems. So if we solved that for service implementers and provided a level of consistency for them in that implementation, it makes bulk upload a great choice. This separate tool could be a future feature and should not hold up this PR. |
@mickmis We're basically actuating ("puppeteering") user interactions with the system under test. Our synchronous requests now are something like: "The user does X. What is the content the user observes in the app in response to doing X?"
The state of the system may change in response to stimuli other than a user interaction with the system. Telemetry is the easy example: USSs are required to change F3548 state in response to telemetry under certain circumstances. Even if we consider telemetry to be a special type of user input, other stimuli could change the state of the system as well. F3548 takes great pains to avoid invalidating deconfliction accomplished strategically based on tactical situations that a USS/operator may not be able to respond effectively/correctly to -- in other words, a system not performing well after planning has very little effect on the safety benefit of strategic deconfliction that F3548 provides. This is not fundamental, however -- for instance, a jurisdiction may establish a requirement for an operator to resolve conflicts in particular ways within particular time limits (F3548 does not generally required the resolution of strategically-allowed conflicts). In this case, the system would change state at that deadline without any user input.
Merely instantaneous. Hopefully we can set up tests that verify what we need to verify by controlling the environment as needed to avoid race conditions and/or stimuli outside the test parameters, but unless we have done that setup, we can't necessarily declare a system under test as non-compliant since the behavior observed could be explained by compliant responses to stimuli outside the test parameters happening at the same time as the activity of interest.
No and maybe, respectively. A number of requirements relate to information presented to the operator/user and we do want some way of verifying these requirements. One way to determine what information is provided to the user could be to have a polling endpoint that just provides the state of the system as the user sees it, and commanding/puppeteering user actions would be a separate endpoint. In that paradigm, we would "plan a flight" by commanding the action (which would simply acknowledge that the action has been initiated) and then polling periodically until we determined that the action had resulted in the expected state change (or not). Instead, we happen to have chosen a system where the action is commanded, but that request does not return until the action has been completed, and then it is convenient to also return the state of the system (from the user perspective) at the same time. Our current system will not be sufficient to test requirements where users are required to get "push" notifications (e.g., SCD0090, SCD0095, GEN0400, GEN0405), but there are multiple ways to fix this. One way could be to have a "system state from user perspective" endpoint that we poll, and then detect the equivalent of push notifications by observing a change in that state. Another way could be to have a list of push notifications that were sent to the user, and we poll that list (I happen to have a predisposition toward the latter). But, I don't think we need either of these things to do many useful things with telemetry, especially (in the F3548 context) since we can poll the DSS and/or a mock_uss for HTTP interactions that it observed from other USSs.
Makes sense to me, and I think that optional-helper-tool approach would be more flexible since USSs could choose to use it or implement something better themselves. Sounds like @nasajoey agrees, assuming the separate tool.
@punamverma We're just describing what the virtual user is doing, so your question is equivalent to "it's not clear whether a user might send telemetry while indicating a particular UAS state and/or a particular airspace/flight plan usage state". I would expect that telemetry would nominally be sent whenever airspace is in use, but there are many off-nominal situations to consider and potentially produce during testing. What do you find unclear?
I've added documentation that essentially treats telemetry as a RESTful sub-resource.
Obtaining telemetry in a FlightPlan. And remember, this is the virtual user's UAS that is reporting telemetry. Hopefully the core USS system is doing nothing with telemetry apart from receiving it -- instead, a separate adapter system is receiving this message from uss_qualifier and doing whatever is necessary to translate it into a USS-native telemetry report that can be sent to the core USS system at the appropriate time. We're talking about the red box on slide 7.
The uas_state and usage_state would be whatever the virtual user indicated them as. The F3548 op intent state would be whatever the USS set it as. Perhaps I'm not clear on what "telemetry is started" means; we're telling the virtual UAS that it's going to report these telemetry points at the specified times and expecting that the virtual UAS will do that. There is no "starting" and "stopping" per se from uss_qualifier's perspective.
This is one of many possible diagrams involving the elements you mentioned. Coloring is designed to align with the slide 7 diagram I mentioned earlier. |
This PR is intended as a concrete design proposal and comments are welcome, including to the effect of taking an entirely different approach.
Motivation
The main purpose of the flight_planning interface is to allow uss_qualifier to "puppeteer" a user of a USS's system to attempt to perform various flight planning activities. To effectively puppeteer a user of a USS's system, InterUSS must provide all the information that would be provided by a real user for that flight. One piece of information we are currently missing is the telemetry that would be generated for the flight. This could be important for some USS implementations if, for instance, they decided to disallow activation until the aircraft associated with the flight reported telemetry inside the flight volumes. Furthermore, a number of ASTM F3548-21 requirements relate to telemetry (specifically: ACM0005, CMSA0100-CMSA0115, LOG0050), so we will likely need to specify it to test those requirements effectively.
Bulk-injection approach (this PR)
This PR approximately follows the approach used by the existing RID injection API by adding one additional field to
FlightPlan
(the main body of the request to perform a flight planning activity) potentially containing telemetry for that flight. This field would only be populated by InterUSS when appropriate -- for instance, planning activities that never intend to activate the flight would likely not specify telemetry. The USS receiving a planning request with telemetry is expected to queue that telemetry to be delivered to the system under test (by the USS's automated testing adapter), clear that queue if the flight is canceled/closed, and update that queue if the flight is updated with different telemetry.The data structures used in this PR are modeled after the data structures in the RID injection API and attempt to capture equivalent information, but are generally made more flexible to accommodate more general telemetry needs that don't necessarily match RID telemetry.
One additional feature in this PR (adopted from the RID injection API) is having a response field where the USS can indicate the information that was actually used. This is necessary in the RID injection API as it is expected that USSs will need to change certain features of the injected data (e.g., serial number), and providing USSs the ability to do this in the flight_planning API will provide them more flexibility as well. For backwards compatibility, a USS can omit this field to indicate that the all information was used as provided.
Advantages of the bulk-injection approach in this PR include:
Disadvantages of this approach include:
Synchronous individual telemetry injection alternative
One alternative to the approach in this PR is not to add any data to the
FlightPlan
structure, and instead create a new operation named something likePUT /telemetry/{telemetry_id}
that would be implemented by a USS under test. uss_qualifier would be responsible for calling this endpoint each time a single telemetry report should be injected into the system under test. The request body of this operation would correspond with theAircraftState
data structure in this PR.Advantages of this approach include:
Disadvantages of this approach include:
Self-generated telemetry alternative
Another alternative approach leverages that simulation capabilities of some USSs. Some USSs have sophisticated real-time aircraft behavior simulations available that could be incorporated into these tests if, instead of uss_qualifier injecting telemetry into USSs, USSs could report the telemetry that was actually generated by their simulations.
For this approach, there would need to be some mechanism for USSs to effectively push telemetry reports to uss_qualifier. Exposing and endpoint to receive these reports would be an enormous architectural change, so this is not a recommended approach. Push notifications may be able to use a similar approach as browser-based web apps, or uss_qualifier could continually poll for pseudo-push-based updates.
Advantages of this approach include:
Disadvantages of this approach include: