Residual subscriptions during F3548 testing

[punamverma]

Describe the bug
There are some tests that are not cleaning up their subscriptions, leading to unreliable run in - Data Validation of GET operational intents by USS - scenario.

To reproduce
Steps to reproduce the behavior:

1. Run uss_qualifier tests on CI
2. Check report.json under output/f3548
3. In the report, go to "Data Validation of GET operational intents by USS" test scenario> Successfully plan flight near an existing flight > Check for notification to tested_uss due to subscription in flight 2 area test step > queries > response > json > interactions
4. You will see there is POST notification made using 8 subscriptions, which is due to other tests that didn't clean their operations and hence subscriptions.

Difference from expected behavior
Should not get notifications due to existing subscriptions, while control_uss planning flights in "Data Validation of GET operational intents by USS" test scenario.

Possible solution
Investigate which tests are not cleaning up, and fix the cleanup.

Screenshots
Attaching a sample report.json file
report.json

System on which behavior was encountered
OS: [e.g. iOS, Windows, Linux; include version and/or other information if it may be relevant]

Codebase information
Currently seen in tests run while running tests in PR - #376

Additional context

[BenjaminPelletier]

Note that tests don't have subscriptions, but rather USSs participating in tests have subscriptions (and sometimes USSs not participating in tests have subscriptions). If mock_uss is not behaving correctly, it would be great to develop a uss_qualifier test scenario + check that will fail so that we can detect this kind of non-compliance in other USSs as well (and then fix mock_uss). If mock_uss is behaving correctly and the uss_qualifier scenarios are issuing instructions that cause USSs in general to be likely to leave subscriptions behind, then definitely let's update those scenarios to better clean up after themselves. If a requirement isn't being violated and changing the scenarios is difficult or ineffective, it could be that the expectation of no subscriptions is not a good expectation -- the presence of one or more subscriptions alone doesn't establish that something is wrong.

[punamverma]

@BenjaminPelletier I did some investigation into what’s causing the residual subscriptions in DSS. There are 22 residual subscriptions left after running the tests with f3548_self_contained configuration.

They are being left over by flight plans in conflict_equal_priority_not_permitted and down_uss_equal_priority_not_permitted test scenarios.

The reason why it is happening is that Uss_qualifier and Mock USS when planning a flight, use implicit subscription for the related operational intent. They also use implicit subscription, when they need to changes the state of the related operational intents to Activated, NonConforming and Contingent, which is an allowed feature in UTM API spec. Hence, every state change creates a new subscription for an operational intent submitted by Mock USS or Uss_qualifier.

What I observed was that when a new subscription gets associated with the operation on state change, the old implicit subscription also stays in DSS. DSS provides the old subscription ids along with the new subscription id in subscribers field of its response to other USSes who plan relevant operations in that area.

However, when the operation is finally deleted, the DSS deletes the operational intent and only the latest implicit
subscription related to it. The older implicit subscriptions of an operation still stay in the DSS, orphaned with no alive operational intent.

In order to solve this issue, it seems like that DSS should make sure to delete all the implicit subscriptions, and not just the latest implicit subscription, related to an operational intent that is deleted.

Though a question arises whether DSS should delete the implicit subscriptions in bulk when an operational intent is deleted from DSS, or DSS should delete an old implicit subscription when a new implicit subscription gets associated with an operational intent.

For a quick example of such subscription_ids, I have attached a sample report.json. In this report you can see -

Test Scenario Nominal planning: not permitted conflict with equal priority ->Test case Attempt to plan flight into conflict -> Test step: Plan Flight 2
operation id 5defcb1b-7d3e-490e-a972-5dca4b6b6151 subscription_id a057a69f-322d-468a-bed1-eb33b13709fd
...

->Test step: Activate Flight 2
operation id 5defcb1b-7d3e-490e-a972-5dca4b6b6151 subscription_id 49fd9850-101e-4b13-8dd8-db49d45eee40

You will notice here the subscription_id changes for operation id 5defcb1b-7d3e-490e-a972-5dca4b6b6151.
In the report you will see subscription_id a057a69f-322d-468a-bed1-eb33b13709fd is found in DSS responses including subscribers list, even after operation id 5defcb1b-7d3e-490e-a972-5dca4b6b6151 being deleted.

[BenjaminPelletier]

DSS0005(1) requires a DSS implementation to comply with the OpenAPI spec which says that implicit subscriptions created upon either creation or update of an operational intent reference should be deleted¹ upon deletion of the corresponding operational intent reference. It would be great to create a test scenario to check this behavior, or update existing scenario(s) to check this behavior.

In the InterUSS implementation, my expectation is: When an implicit subscription is created, it fully covers the operational intent reference. Therefore, if there was a pre-existing implicit subscription, it should be deleted when the new implicit subscription is created if the old implicit subscription wasn't serving any other purpose (no reason to have two subscriptions to serve the same notification purpose). Also, I would expect that if the InterUSS DSS would create a new implicit subscription that is effectively the same as an old implicit subscription, the old implicit subscription would instead be mutated and no new implicit subscription would be created.

¹Documentation says this should happen "generally", but I think it's fair to assume it should happen unless there is a specific reason for it not to happen. The only reason I would expect it not to happen is if that implicit subscription was then used explicitly to create or update another operational intent reference.

[BenjaminPelletier]

I think a good scenario to verify correct DSS management of implicit subscriptions would be:

Op intent	State	Relevant to	Does not intersect with
OI1	Any nominal	OI2, OI3	OI2, OI3
OI2	Accepted	OI1	OI1, OI3
OI3	Any nominal	OI1	OI1, OI2

t_now << t₁ < t₂ < t₃

1. Create OI3 in DSS from t₂ to t₃
2. Note any pre-existing subscriptions based on the subscribers in response to OI3 creation
3. Delete OI3 from DSS (and possibly confirm pre-existing subscriptions based on subscribers)
4. Create OI1 in DSS from t_now to t₃ using implicit subscription
5. Create OI2 in DSS from t₂ to t₃ without subscription, confirm OI1's implicit subscription is among subscribers
6. Update OI1 in DSS to be from t_now to t₁
7. Create OI3 in DSS from t₂ to t₃, confirm no subscriptions beyond pre-existing subscriptions in step 2 are among subscribers
8. Clean up

[Shastick]

@BenjaminPelletier I'm working on the scenario to check this and need to confirm some assumptions:

Taking the 8 steps you described:

1. Create OI3 in DSS from t2 to t3

I assume OI3 should not specify a subscription id, and an implicit subscription should be created.

2. Note any pre-existing subscriptions based on the subscribers in response to OI3 creation

3. Delete OI3 from DSS (and possibly confirm pre-existing subscriptions based on subscribers)

After deletion of OI3, I assume that the corresponding implicit subscription should be deleted (ie, querying it should result in a 404), as described in the OpenAPI spec

Note that this currently fails on the DSS cleanup for the simple case seems to work.

4. Create OI1 in DSS from tnow to t3 using implicit subscription

Does using implicit subscription here mean:

• "reuse implicit subscription from OI3" (which in my current understanding of things should fail, because the subscription was cleaned up)
• "create without specifying a subscription id, specify that an implicit subscription should be created if required, expect an implicit one to be created or attached"

5. Create OI2 in DSS from t2 to t3 without subscription, confirm OI1's implicit subscription is among subscribers

In addition to having OI1's subscription mentioned in subscribers, I also assume that the subscription that gets attached is the one of OI1 (assuming that the initial implicit subscription for OI3 has been removed)

6. Update OI1 in DSS to be from tnow to t1

Here, from my understanding, what should generally happen is that a new implicit subscription should be created and attached to OI1, as OI1 and OI2 are now entirely disjoint.

I am wondering if it would be acceptable for a DSS to not do anything with the implicit subscription, as it technically still covers both OI1 and OI2, although it also covers the interval from t_1 to t_2 where nothing exists? My hunch would be to not accept this, but I'd like a second opinion on that matter.

7. Create OI3 in DSS from t2 to t3, confirm no subscriptions beyond pre-existing subscriptions in step 2 are among subscribers

Here OI3 should be attached to the implicit subscription created in step 4 that now only covers OI2, assuming that step 6 caused the creation of a new implicit subscription.

[BenjaminPelletier]

I assume OI3 should not specify a subscription id, and an implicit subscription should be created.

It shouldn't matter (the scenario should work the same regardless of whether OI3 uses an explicit or implicit subscription), but yes, I would expect an implicit subscription should be easiest.

After deletion of OI3, I assume that the corresponding implicit subscription should be deleted (ie, querying it should result in a 404), as described in the OpenAPI spec

Yes (assuming OI3 used an implicit subscription)

Does using implicit subscription here mean:

• "reuse implicit subscription from OI3" (which in my current understanding of things should fail, because the subscription was cleaned up)
• "create without specifying a subscription id, specify that an implicit subscription should be created if required, expect an implicit one to be created or attached"

Providing a pre-existing subscription for an operational intent would be specifying an explicit subscription for that operational intent (even if that subscription was implicit for a different operational intent). Yes, the intention there was to allow the DSS to create and manage an implicit subscription rather than providing an explicit subscription ID.

In addition to having OI1's subscription mentioned in subscribers, I also assume that the subscription that gets attached is the one of OI1 (assuming that the initial implicit subscription for OI3 has been removed)

What do you mean by "gets attached"? (what gets attached to what?) OI2 is in the Accepted state and does not need a subscription (and the step suggestion specifies "without subscription")

Here, from my understanding, what should generally happen is that a new implicit subscription should be created and attached to OI1, as OI1 and OI2 are now entirely disjoint.

The DSS could create a new implicit subscription due to the update, but I would generally expect that the DSS creates and maintains a single implicit subscription per operational intent. So, I would expect the implicit subscription for OI1 would be mutated to change time range. OI2 has no subscription and therefore OI1 & OI2 being disjoint is irrelevant for OI1's subscription state.

I am wondering if it would be acceptable for a DSS to not do anything with the implicit subscription, as it technically still covers both OI1 and OI2, although it also covers the interval from t_1 to t_2 where nothing exists? My hunch would be to not accept this, but I'd like a second opinion on that matter.

Not sure I follow; OI2 has no subscription, implicit or explicit.

Here OI3 should be attached to the implicit subscription created in step 4 that now only covers OI2, assuming that step 6 caused the creation of a new implicit subscription.

I would generally expect implicit subscriptions not to be reused between operational intents by the DSS. There are difficulties when a USS uses an implicit subscription for one operational intent as an explicit subscription for a second operational intent and we should write tests for those cases, but I believe those are more edge cases and can therefore be prioritized after the core set of tests confirming basic implicit subscription behavior described here.

[Shastick]

Here is a possible cause for the residual subscriptions.

[Shastick]

This has been addressed on the DSS and is now also properly tested by the qualifier.

I expect the problem of residual subscriptions to be solve: if you witness this to be otherwise, please reopen.