[uss_qualifier] Clean up get op data validation scenario #419

BenjaminPelletier · 2023-12-18T04:34:42Z

This PR cleans up a few things in the Data Validation of GET operational intents by USS test scenario:

begin_test_step and end_test_step are removed from reusable test step fragments per Change reusable test steps to reusable test step fragments #380
control_uss is changed to mock_uss because we know that resource is a mock USS, so therefore there is no difference between control_uss and mock_uss, and one consistent name is easier to comprehend
The test step to check for notifications is moved to just before its outcome is needed
The purpose of the test step to check for notifications is clarified
Redundant explanation of the Validate flight2 GET interaction test step is removed to avoid the possibility of inconsistency (the same explanation remains in the reusable test step fragment)
Validation of flight intents is performed using ExpectedFlightIntents, similar to [uss_qualifier] Add flight intent validation and fix conflict same priority scenario #393
Test case function names are shortened for clarity
Severity overrides are removed where severity is specified in documentation per Specify check severity in test scenario documentation only #404

-- edit --

Identification of desired mock_uss interactions is improved

-- edit 2 --

Fix infinite loop preventing f3548_self_contained to complete #425

…ation

punamverma · 2023-12-18T19:01:48Z

monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.md

 The test driver attempts to plan flight 1 via the tested USS. It checks if any conflicts with flight 2
 which is of equal priority and came first.

 ### [Validate flight 1 sharing test step](../validate_shared_operational_intent.md)
 Validate flight 1 is planned.

+### Check for notification to tested_uss due to subscription in flight 2 area test step


After sharing flight1, there are other POST interactions that might happen. We will be able to catch the flight2 notification immediately after it's planned. Or else we will need to know the base_url for uss_endpoint of the tested_uss, which we don't have access to.

Good point -- currently, the test looks for any POST interaction which is needlessly broad since the notifications we're looking for would only be outgoing from mock_uss. I'll clean that up as well, and then this move should be fine since mock_uss does not make any outgoing POST notifications in the steps in between this step's old position and its new position.

This is now fixed -- instead of just searching according to POST generally (which is only one small characteristic of the interactions of interest), we now search much more specifically and can differentiate (for instance) between incoming and outgoing operational intent details notifications.

punamverma · 2023-12-18T19:02:55Z

monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.md

 The test driver attempts to plan the flight 1 via the tested_uss. It checks if any conflicts with flight 2
 which is of equal priority and came first.

 ### [Validate flight 1 not shared by tested_uss test step](../validate_not_shared_operational_intent.md)
 Validate flight 1 is not shared with DSS, as plan failed.

+### Check for notification to tested_uss due to subscription in flight 2 area test step


Same as above comment.

Fixed per above.

…ation

punamverma · 2023-12-19T04:33:47Z

monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.py

-                "Validate flight2 GET interaction, if no notification",
+
+        self.begin_test_step(
+            "Check for notification to tested_uss due to subscription in flight 2 area"


Not sure why we would want to check for notifications here, instead of close to when it was sent after planning flight 2. Then we have to go through less number of interactions to filter it out.
I am not saying this is wrong, but trying to understand the reason behind it. I understand its close to where its being used, is that the reason?

Checking later gives more time for the notification to happen, we're searching since flight_2_planning_time in any case, and the result isn't used until later (where this step is now). We're now filtering for the notification we actually want via method, URL, and op intent ID, so the likelihood of a false positive is very low regardless of the number of other interactions in the mean time.

We actually don't need to wait here, as the outgoing notifications from mock_uss are sent sequentially before sending Completed response to uss_qualifier. So, if we get a success plan from mock_uss, we know the notifications have already been sent.
Checking later is fine with me.

punamverma · 2023-12-19T04:37:06Z

...lifier/scenarios/astm/utm/data_exchange_validation/test_steps/query_mock_uss_interactions.md

+
+This step obtains interactions of interest from mock_uss.
+
+## 🛑 MockUSS interactions request check


Ok, we just copy the High Severity symbol into the doc. How does it get related to the code?

punamverma · 2023-12-19T04:46:57Z

The code looks good.
However, when I tried running the tests locally, somehow they are failing at Validation of operational intents scenario. I see CI ran successfully. Could be my local environment has issues.
But, I do see that from previous code that I had missed to assign None to Optional field in this line -

monitoring/monitoring/uss_qualifier/scenarios/astm/utm/test_steps.py

Line 503 in 93023ff

expected_invalid_fields: Optional[List[str]],

punamverma · 2023-12-19T04:49:48Z

...r/scenarios/astm/utm/data_exchange_validation/test_steps/expected_interactions_test_steps.py

@@ -1,54 +1,56 @@
 from __future__ import annotations

 import datetime
+import re
 from typing import List, Tuple, Optional


Missing import for Dict and Callable.

Added, thanks. My IDE is going crazy on this file specifically

punamverma · 2023-12-19T04:50:17Z

...r/scenarios/astm/utm/data_exchange_validation/test_steps/expected_interactions_test_steps.py

 from implicitdict import StringBasedDateTime
 from loguru import logger
+from uas_standards.astm.f3548.v21 import api
+from uas_standards.astm.f3548.v21.api import OperationID


Missing import for EntityID

Added, thanks

punamverma · 2023-12-19T05:08:43Z

I like the idea of removing begin_test_step and end_test_step from reusable code. Initially, it took me some time to understand the flow of the tests because of that. I thought it was a convention being used in the code. Taking it out is a good idea.
Liked renaming control_uss to mock_uss.
Also, refactoring the code with mock_uss_interactions method makes it better. Thanks.

barroco · 2023-12-19T16:29:48Z

monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.md

@@ -2,8 +2,8 @@

 ## Description
 This test checks that the USS being tested validates the operational intents received as response to its GET request from another USS.
-Control_uss which is a mock uss plans a nearby V-shaped operation, and provides the data that tested_uss GETs.
-tested_uss validates the GET response from control_uss and accordingly plan its operation.
+mock_uss which is a mock uss plans a nearby V-shaped operation, and provides the data that tested_uss GETs.


Suggested change

mock_uss which is a mock uss plans a nearby V-shaped operation, and provides the data that tested_uss GETs.

mock_uss plans a nearby V-shaped operation, and provides the data that tested_uss GETs.

barroco · 2023-12-19T16:32:12Z

monitoring/uss_qualifier/scenarios/astm/utm/data_exchange_validation/get_op_data_validation.md

-The control_uss, which is mock_uss is instructed to share invalid data with other USS, for negative test.
+### [mock_uss plans flight 2, sharing invalid operational intent data test step](../../../flight_planning/plan_flight_intent.md)
+Flight 2 should be successfully planned by the mock_uss.
+The mock_uss, which is mock_uss is instructed to share invalid data with other USS, for negative test.


Suggested change

The mock_uss, which is mock_uss is instructed to share invalid data with other USS, for negative test.

mock_uss is instructed to share invalid data with other USS, for negative test.

…ation

* Clean up get op data validation scenario * Clean up interaction search and severities * Add missing imports * Improve docs per comments, fix #425 d1d21a7

Clean up get op data validation scenario

6833bd1

BenjaminPelletier marked this pull request as ready for review December 18, 2023 05:05

BenjaminPelletier requested a review from punamverma December 18, 2023 05:05

BenjaminPelletier mentioned this pull request Dec 18, 2023

[uss_qualifier] Remove begin and end test step from prioritization_test_steps #420

Merged

BenjaminPelletier added 2 commits December 18, 2023 17:53

Merge remote-tracking branch 'interuss/main' into clean-up-data-valid…

341dd75

…ation

Merge remote-tracking branch 'interuss/main' into clean-up-data-valid…

5c6f5e2

…ation

punamverma reviewed Dec 18, 2023

View reviewed changes

BenjaminPelletier added 2 commits December 18, 2023 23:37

Clean up interaction search and severities

4f1aa98

Merge remote-tracking branch 'interuss/main' into clean-up-data-valid…

c1146ad

…ation

punamverma reviewed Dec 19, 2023

View reviewed changes

Add missing imports

dcefdaf

BenjaminPelletier requested a review from barroco December 19, 2023 06:27

punamverma approved these changes Dec 19, 2023

View reviewed changes

BenjaminPelletier mentioned this pull request Dec 19, 2023

infinite loop preventing f3548_self_contained to complete #425

Closed

barroco approved these changes Dec 19, 2023

View reviewed changes

BenjaminPelletier added 2 commits December 19, 2023 17:13

Improve docs per comments, fix interuss#425

ef7d73b

Merge remote-tracking branch 'interuss/main' into clean-up-data-valid…

96c8e75

…ation

BenjaminPelletier merged commit d1d21a7 into interuss:main Dec 19, 2023
9 checks passed

BenjaminPelletier deleted the clean-up-data-validation branch December 19, 2023 17:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[uss_qualifier] Clean up get op data validation scenario #419

[uss_qualifier] Clean up get op data validation scenario #419

BenjaminPelletier commented Dec 18, 2023 •

edited

Loading

punamverma Dec 18, 2023

BenjaminPelletier Dec 18, 2023

BenjaminPelletier Dec 19, 2023

punamverma Dec 18, 2023

BenjaminPelletier Dec 19, 2023

punamverma Dec 19, 2023

BenjaminPelletier Dec 19, 2023

punamverma Dec 19, 2023

punamverma Dec 19, 2023 •

edited

Loading

BenjaminPelletier Dec 19, 2023

punamverma commented Dec 19, 2023 •

edited

Loading

punamverma Dec 19, 2023

BenjaminPelletier Dec 19, 2023

punamverma Dec 19, 2023

BenjaminPelletier Dec 19, 2023

punamverma commented Dec 19, 2023 •

edited

Loading

barroco Dec 19, 2023

barroco Dec 19, 2023


		This step obtains interactions of interest from mock_uss.

		## 🛑 MockUSS interactions request check

	mock_uss which is a mock uss plans a nearby V-shaped operation, and provides the data that tested_uss GETs.
	mock_uss plans a nearby V-shaped operation, and provides the data that tested_uss GETs.

	The mock_uss, which is mock_uss is instructed to share invalid data with other USS, for negative test.
	mock_uss is instructed to share invalid data with other USS, for negative test.

[uss_qualifier] Clean up get op data validation scenario #419

[uss_qualifier] Clean up get op data validation scenario #419

Conversation

BenjaminPelletier commented Dec 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

punamverma Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

punamverma commented Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

punamverma commented Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenjaminPelletier commented Dec 18, 2023 •

edited

Loading

punamverma Dec 19, 2023 •

edited

Loading

punamverma commented Dec 19, 2023 •

edited

Loading

punamverma commented Dec 19, 2023 •

edited

Loading