[uss_qualifier/scenarios/utm] Fix #178

[mickmis]

This should fix #178 by allowing NotSupported result from the USS when we are attempting to modify an activated flight intent with a pre-existing conflict in the nominal planning scenario.

Additionally:

• add requirement FlightCoveredByOperationalIntent and associated check when validating the sharing of the op intent
• add ability to override severity in submit_flight_intent

[BenjaminPelletier]

Note 1

I think it will be important to update the content of Modify activated flight with pre-existing conflict test case -- specifically, first, Modify activated flight 1 in conflict with activated flight 2 test step. Currently, it seems to say the modification must be accepted and doesn't seem to clarify the changes to the implementation in this PR (not supported being ok). Also, the following step indicates definitively "The first flight should have been modified", but it seems like that's not true if the USS indicates NotSupported in the previous step. I think we'll want to clearly explain what is expected from the modification attempt:

• Accepted: this is ok (does not violate any F3548 requirements) since the conflict previously existed (per SCD0030)
• NotSupported: this is ok since some USSs may want to be more conservative and there is no positive F3548 requirement to allow an operator to update a flight in conflict
• Rejected (or deprecated ConflictWithFlight): I think this is tricky. There are (at least) a few ways this result could be obtained:
  1. The USS missed the pre-existing-conflict exemption in SCD0030 and has therefore incorrectly rejected the update.
  2. The USS has purposefully chosen to be more conservative than required by F3548-21, mandating that its operators include volumes for a backup plan in the case of conflicts -- in this case, they protect against accidental unintended state changes by disallowing modifications of an operational intent in conflict, instead requiring the operator to follow their backup plan already included in the original operational intent.
  3. The USS doesn't support updates directly and instead achieves updates by deleting the operational intent and creating a new one whenever the user wants to change their flight...
     1. ...and the USS deleted the existing operational intent before creating a new one and now the aircraft in the air is stuck with no operational intent
     2. ...and the USS recognizes that a new (replacement) operational intent would not be possible to create in this scenario, so it disallows the change without deleting the existing operational intent

I think (i) above is probably not valid, but (ii) seems valid and compliant and I'm not sure how we could differentiate between (i) and (ii). (iii)a makes me very uneasy and I think is against the spirit of F3548 (4.3.1: "An operational intent is a volume-based representation of a UAS flight used to define the airspace and time bounds intended to contain the flight." -- if the op intent is removed when the aircraft is known or reasonably expected to be outside it, the op intent can't be intended to contain the flight). But, (iii)b seems reasonable given the limitations of the USS implementation. So, I'm actually not sure we can fail any checks based on the USS's response to our request that it modify the flight in conflict. I think it perhaps is fair to fail the (iii)a situation, which we can detect if 1) the USS indicated Reject and 2) the operational intent disappeared (in fact, any disappearance of the operational intent at this point might indicate the same kind of problem, but that seems arguable).

Note 2

Given the above and the fact that we might encounter NotSupported, it seems like we don't have a known state going into Attempt to modify activated flight in conflict test case. Flight 1 could be in the original conflicting state, or the new conflicting state, or perhaps missing (depending on whether we check against (iii)a). We'd want to explain how the check is still valid for all those circumstances if we allow all of them.

[mickmis]

Thanks a lot for the details. Indeed I went a bit fast on the fix and forgot to update the scenario .md, I only updated the test step .md. I have a few clarifying questions before going ahead:

• ii. Not sure I understand this case.

  • Do you mean that when the operator submits its intent, it has to have additional volume(s) as backup in case of conflict? Doesn't that just shift/delay the issue, i.e. what if there the conflict case also happens with the backup volume(s)?
  • And actually why does this case would result in a Rejected rather than a NotSupported? I.e. isn't this case equivalent to (or rather, encompassing) not allowing an operator to update a flight in conflict?

• iii.a.

  • In that case, shouldn't it be the responsibility of the USS to create a new operational intent before deleting the old one? I feel like this would follow the spirit of the standard. Although I'm not sure if in that case the requirements for the equal priority conflicts (permitted or not) would apply or not. Or am I missing something?
  • Additionally, shouldn't this be transparent to the user / test harness? I.e. through this interface we "just" receive a new operational intent ID, use it as the op intent ID for this flight, and ignore whatever went on behind the scene for this to happen?

[BenjaminPelletier]

ii. Not sure I understand this case.

I was just trying to come up with situations in which a USS would indicate Rejected when trying to change a flight in pre-existing conflict -- this example was just a somewhat-contrived implementation where the USS wouldn't allow a user to make that change because they set things up so that change would be unnecessary (by requiring operators to include a backup plan when planning). The purpose of this thought exercise was to determine whether Rejected would necessarily be due to some problem or mistake the USS made. I think my tentative conclusion is that we can't necessarily conclude that the USS is doing something wrong/violating some requirement if it indicates Rejected here.

In that case, shouldn't it be the responsibility of the USS to create a new operational intent before deleting the old one?

That (creating an operational intent before deleting the old one) would be the approach that leads to iii.b., and I agree that's certainly what the spirit of the standard intends. If the USS has the responsibility to not leave a flight without an operational intent, that responsibility would come from a requirement that all flights are encompassed in operational intents (in other words: F3548 is being used) and the requirement to use F3548 would come from the regulatory authority rather than F3548 because F3548 can't mandate that USSs use F3548. So, I think we need to have an automated testing requirement that substitutes for that regulatory requirement if we want to claim that failing to cover a flight with an operational intent, or failing to maintain an operational intent for a flight, is not allowed. The requirement would probably go something like: "For InterUSS to effectively test the requirements of ASTM F3548-21, a USS under test must act as if there is a regulatory requirement requiring all flights it manages to provide operational intents according to ASTM F3548-21 at all times for all flights it manages." I intend to clean up and elaborate on our F3548-related requirements soon like I recently did for our F3411-related requirements.

Additionally, shouldn't this be transparent to the user / test harness? I.e. through this interface we "just" receive a new operational intent ID, use it as the op intent ID for this flight, and ignore whatever went on behind the scene for this to happen?

In the nominal case for the current scd automated testing interface, yes -- if a USS achieved flight modification by deleting the existing operational intent and then creating a new operational intent, it would simply return the new op intent ID and everything should work fine. But iii.b. is an example of when it would impossible for the USS to update the operational intent with a pre-existing conflict. It's impossible in iii.b. because the USS's system only supports the delete-then-create approach, and F3548 prescribes there can be no creation in the iii.b. situation, even though direct modification would be allowed by F3548.

Regulators could mandate that USSs support modifications in pre-existing conflict situations such as these, in which case the USS that only supports delete-then-create would be in violation of that regulatory requirement. But, F3548 doesn't establish a positive requirement to support this ability -- it only prescribes NOT doing unsafe things. I would have to review the standard again to be sure, but I think a USS that simply refused to ever plan any flight would fully satisfy all F3548 requirements because F3548's goal is to achieve safety rather than ensuring each USS provides a certain level of utility. The thought was that USSs will compete to provide useful services to their customers/operators, so there is no need to mandate the things they can do, only the things they can't do (namely: those things that would result in an unacceptable decrease in safety). The way we'd fail a non-planning USS during testing is with the ExpectedBehavior automated testing requirement (to be refined, as noted above).