[uss_qualifier] NET0260-a performance check for querying flights in an area from a SP #159

Shastick · 2023-08-11T10:58:16Z

Ensures that the multiple queries to a SP for requesting flights in an area match the required response times.

We have several queries done to that endpoint during the tests already, so this simply extends our aggregate checks.

Observers and service_providers defined in environment.yaml (and their respecive Python classes) now have a uss_base_urls array parameters that we use to map queries to USS'es. This is required as (to the best of my understanding) we need to be told which urls will be registered for an SP at the DSS.

Note this also includes a little utility for displaying information about a report.

Shastick · 2023-08-11T10:59:16Z

Note: not sure if we want to keep the small utility for exploring a report's content? It was helpful to me to learn a bit about how the classes are structured and expect it to help a bit more if we have more complex aggregated checks, but I can also remove it.

mickmis · 2023-08-15T10:46:17Z

Before a review, could you ensure the CI passes?
I see a failure in the log, which is unfortunately not caught by the CI, c.f. #162
Thanks!

Shastick · 2023-08-15T12:52:16Z

Before a review, could you ensure the CI passes? I see a failure in the log, which is unfortunately not caught by the CI, c.f. #162 Thanks!

Looks like the failure was unrelated. Rebased and PR still green and ready for review.

mickmis

The check does not seem to validate any query:

2023-08-15 12:43:57.628 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:263 - Note: uss1/flights -> skipped check: no relevant queries
2023-08-15 12:43:57.629 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:263 - Note: uss2/flights -> skipped check: no relevant queries

I assume that is not expected?

monitoring/uss_qualifier/scenarios/astm/netrid/common/aggregate_checks.py

monitoring/uss_qualifier/report_analyzer.py

Shastick

The check does not seem to validate any query

@mickmis was this for v22a or v19? IIRC I had this passing locally when tested against v22a only.

monitoring/uss_qualifier/scenarios/astm/netrid/common/aggregate_checks.py

monitoring/uss_qualifier/configurations/dev/library/environment.yaml

monitoring/uss_qualifier/resources/netrid/observers.py

monitoring/uss_qualifier/scenarios/astm/netrid/common/aggregate_checks.py

mickmis · 2023-08-18T12:37:55Z

monitoring/uss_qualifier/scenarios/astm/netrid/common/aggregate_checks.py

        self.end_test_scenario()

+    def _sp_flights_area_times_step(self):
+        pattern = re.compile(r"/flights\?view=")


I realize now an issue with the regex (which is also present in the _dp_display_data_times_step I wrote), which is that it is valid to have another parameter after the ? and before the view parameter. The regex should be updated accordingly to reflect this. (and if you don't mind fixing the other one at the same time that would be great)

A side question here: does it matter if there are additional parameters being passed? Note that for the queries I am interested in, there actually are more params: Ie http://host.docker.internal:8081/mock/ridsp/v2/uss/flights?view=37.17008162687164%2C-80.60282480412947%2C37.22330680061591%2C-80.5495996303852&recent_positions_duration=60, and there's no guarantee about their order to regex matching too precisely could cause troubles as well.

What might make sense is to give the full method path to avoid accidentally matching something else, but the goal being to find queries to a particular endpoint, I'm not sure there is a need to validate the parameters?

A side question here: does it matter if there are additional parameters being passed? Note that for the queries I am interested in, there actually are more params: Ie http://host.docker.internal:8081/mock/ridsp/v2/uss/flights?view=37.17008162687164%2C-80.60282480412947%2C37.22330680061591%2C-80.5495996303852&recent_positions_duration=60, and there's no guarantee about their order to regex matching too precisely could cause troubles as well.

For the requirement NET0260, no it does not matter.

What might make sense is to give the full method path to avoid accidentally matching something else, but the goal being to find queries to a particular endpoint, I'm not sure there is a need to validate the parameters?

Indeed you are right, no need to validate the parameters here for NET0260.

monitoring/uss_qualifier/configurations/dev/library/environment.yaml

monitoring/uss_qualifier/resources/netrid/observers.py

monitoring/uss_qualifier/scenarios/astm/netrid/common/aggregate_checks.py

mickmis · 2023-08-23T12:01:23Z

Additionally, looking at the CI logs something looks wrong. I think it's because of the uss_base_urls having only the hostname and not the rest of the path.

For v22a there is only 1 participant and only the /flights call being evaluated:

2023-08-22 05:28:45.802 | INFO     | monitoring.uss_qualifier.suites.suite:_run_test_scenario:122 - SUCCESS for "ASTM NetRID nominal behavior" scenario
2023-08-22 05:28:46.229 | INFO     | monitoring.uss_qualifier.suites.suite:_run_test_scenario:104 - Running "ASTM F3411-22a NetRID aggregate checks" scenario...
2023-08-22 05:28:46.229 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:263 - Note: participants -> {'http://host.docker.internal:8081': 'uss2'}
2023-08-22 05:28:46.229 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:263 - Note: nb_queries -> 276
2023-08-22 05:28:46.230 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:263 - Note: uss2/display_data -> skipped check: no relevant queries
2023-08-22 05:28:46.232 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:263 - Note: uss2/flights -> percentiles on 37 relevant queries: 95th: 0.034843299999999994; 99th: 0.060277460000000005
2023-08-22 05:28:46.232 | INFO     | monitoring.uss_qualifier.suites.suite:_run_test_scenario:122 - SUCCESS for "ASTM F3411-22a NetRID aggregate checks" scenario

For v19 nothing is evaluated:

2023-08-22 05:27:15.854 | INFO     | monitoring.uss_qualifier.suites.suite:_run_test_scenario:122 - SUCCESS for "ASTM NetRID nominal behavior" scenario
2023-08-22 05:27:16.286 | INFO     | monitoring.uss_qualifier.suites.suite:_run_test_scenario:104 - Running "ASTM F3411-19 NetRID aggregate checks" scenario...
2023-08-22 05:27:16.287 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:263 - Note: participants -> {'http://host.docker.internal:8071': 'uss2'}
2023-08-22 05:27:16.287 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:263 - Note: nb_queries -> 390
2023-08-22 05:27:16.289 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:263 - Note: uss2/display_data -> skipped check: no relevant queries
2023-08-22 05:27:16.290 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:263 - Note: uss2/flights -> skipped check: no relevant queries
2023-08-22 05:27:16.290 | INFO     | monitoring.uss_qualifier.suites.suite:_run_test_scenario:122 - SUCCESS for "ASTM F3411-19 NetRID aggregate checks" scenario

monitoring/uss_qualifier/scenarios/astm/netrid/common/aggregate_checks.py

Shastick · 2023-08-24T07:38:15Z

Additionally, looking at the CI logs something looks wrong. I think it's because of the uss_base_urls having only the hostname and not the rest of the path.

Indeed, something is off here.

Though we match URLS by prefix for mapping them to participants (via query.request.url.startswith(base_url)) so even if we only have the hostname it should still work.

edit: on master (see this run) v22a is only doing the aggregate checks for USS1. It's either expected or broken?

v22a aggreagate checks on master:

2023-08-22 22:45:50.054 | INFO     | monitoring.uss_qualifier.suites.suite:_run_test_scenario:104 - Running "ASTM F3411-22a NetRID aggregate checks" scenario...
2023-08-22 22:45:50.055 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: participants -> {'http://v22a.ridsp.uss1.localutm/ridsp/injection': 'uss1', 'http://v22a.riddp.uss1.localutm/riddp/observation': 'uss1'}
2023-08-22 22:45:50.055 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: nb_queries -> 290
2023-08-22 22:45:50.058 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: uss1/display_data -> percentiles on 46 relevant queries (24 different URLs, 35 initial queries, 11 subsequent queries): init 95th: 0.1469108; init 99th: 0.17310296; subsequent 95th: 0.0832726; subsequent 99th: 0.08596732000000001
2023-08-22 22:45:50.058 |

Shastick · 2023-08-25T15:01:10Z

Fixed a bug I introduced in aggregate_checks.py's constructor.

We now have, for v19:

2023-08-25 14:39:49.132 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: participants_by_base_url -> {'http://v19.ridsp.uss2.localutm/mock/ridsp': 'uss2', 'http://v19.ridsp.uss2.localutm/ridsp/injection': 'uss2', 'http://v19.riddp.uss3.localutm/riddp/observation': 'uss3'}
2023-08-25 14:39:49.133 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: uss2/display_data -> skipped check: no relevant queries
2023-08-25 14:39:49.135 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: uss3/display_data -> percentiles on 68 relevant queries (62 different URLs, 66 initial queries, 2 subsequent queries): init 95th: 0.11976845000000001; init 99th: 0.12929327; subsequent 95th: 0.0223759; subsequent 99th: 0.023277580000000003
2023-08-25 14:39:49.137 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: uss2/flights -> percentiles on 58 relevant queries: 95th: 0.05983205; 99th: 0.06097860000000001
2023-08-25 14:39:49.138 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: uss3/flights -> skipped check: no relevant queries

and for v22a:

2023-08-25 14:50:04.666 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: participants_by_base_url -> {'http://v22a.ridsp.uss1.localutm/mock/ridsp/v2': 'uss1', 'http://v22a.ridsp.uss1.localutm/ridsp/injection': 'uss1', 'http://v22a.riddp.uss1.localutm/riddp/observation': 'uss1'}
2023-08-25 14:50:04.668 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: uss1/display_data -> percentiles on 46 relevant queries (24 different URLs, 31 initial queries, 15 subsequent queries): init 95th: 0.09958079999999998; init 99th: 0.10547775999999999; subsequent 95th: 0.0798888; subsequent 99th: 0.08003536
2023-08-25 14:50:04.670 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: uss1/flights -> percentiles on 37 relevant queries: 95th: 0.029613199999999996; 99th: 0.03133384

environment.yaml only configures a uss1 participant so the discrepancy between v19 and v22a seems fine.

monitoring/uss_qualifier/resources/netrid/service_providers.py

monitoring/uss_qualifier/scenarios/astm/netrid/common/aggregate_checks.py

Shastick · 2023-08-28T13:26:41Z

Thanks for the feedback, @BenjaminPelletier, both suggestions seem doable to me, I'll get back to you if I need any clarification.

Shastick · 2023-08-31T14:45:44Z

Taking this to a draft state again until it's in a satisfactory state

…playing flights in a given area

Shastick · 2023-08-31T20:15:45Z

Ready for another pass of review. The current version is working, as far as I can trust the logs:

2023-08-31 20:04:52.192 | INFO     | monitoring.uss_qualifier.suites.suite:_run_test_scenario:104 - Running "ASTM F3411-22a NetRID aggregate checks" scenario...
2023-08-31 20:04:52.193 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: participants -> {'http://v22a.ridsp.uss1.localutm/ridsp/injection': 'uss1', 'http://v22a.riddp.uss1.localutm/riddp/observation': 'uss1'}
2023-08-31 20:04:52.193 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: nb_queries -> 290
2023-08-31 20:04:52.193 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: service_providers -> configured service providers: uss1 - http://v22a.ridsp.uss1.localutm/ridsp/injection
2023-08-31 20:04:52.193 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: observer -> configured observer: uss1 - http://v22a.riddp.uss1.localutm/riddp/observation
2023-08-31 20:04:52.196 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: uss1/display_data -> percentiles on 46 relevant queries (24 different URLs, 34 initial queries, 12 subsequent queries): init 95th: 0.07079325; init 99th: 0.07741384999999999; subsequent 95th: 0.0712499; subsequent 99th: 0.07249478000000001
2023-08-31 20:04:52.197 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:270 - Note: uss1/flights -> percentiles on 56 relevant queries: 95th: 0.027803249999999998; 99th: 0.02819827
2023-08-31 20:04:52.197 | INFO     | monitoring.uss_qualifier.suites.suite:_run_test_scenario:122 - SUCCESS for "ASTM F3411-22a NetRID aggregate checks" scenario

v19 also has output showing that queries are properly analysed.

monitoring/uss_qualifier/resources/netrid/service_providers.py

monitoring/uss_qualifier/scenarios/astm/netrid/common/aggregate_checks.py

BenjaminPelletier · 2023-08-31T22:08:13Z

monitoring/uss_qualifier/scenarios/astm/netrid/display_data_evaluator.py

@@ -234,6 +234,9 @@ def evaluate_system_instantaneously(
            )
            for q in sp_observation.queries:
                self._test_scenario.record_query(q)
+                # Keep track of who the query is being made to
+                # for later use in aggregate checks
+                q.server_id = self._dss.participant_id


sp_observation is a FetchedFlights which consists of a query to the DSS, queries to various USSs for flights, and queries to various USSs for flight details. The DSS participant ID only applies to the first of those, so we wouldn't want to apply it to all the USS queries as well.

At this point, we don't know which of sp_observation.uss_flight_queries belongs to which (if any) of the test participants -- it's conceivable that another USS not participating in the test may nonetheless be in the interoperability ecosystem, causing uss_qualifier and other USSs to interact with it during the test. We only deduce which interoperability ecosystem SPs must be which test participants here when we map the observed flights to injected flights. Since (at that point) we know which participant each injected flight was sent to, we can, at this point, associate observed flights with test participants.

Ah, indeed.

Stamping the query with the id now happens after the mapping was established.

Shastick · 2023-09-01T14:00:18Z

PR updated, log outputs for the checks are still looking as expected.

Shastick force-pushed the feature/net-0260-a branch from fa91295 to 86d160a Compare August 15, 2023 12:34

mickmis reviewed Aug 15, 2023

View reviewed changes

monitoring/uss_qualifier/scenarios/astm/netrid/common/aggregate_checks.py Outdated Show resolved Hide resolved

monitoring/uss_qualifier/report_analyzer.py Outdated Show resolved Hide resolved

Shastick marked this pull request as draft August 15, 2023 13:28

Shastick commented Aug 16, 2023

View reviewed changes

monitoring/uss_qualifier/scenarios/astm/netrid/common/aggregate_checks.py Outdated Show resolved Hide resolved

Shastick force-pushed the feature/net-0260-a branch 5 times, most recently from d729a19 to 6966e85 Compare August 16, 2023 18:34

Shastick marked this pull request as ready for review August 16, 2023 18:54

Shastick force-pushed the feature/net-0260-a branch 2 times, most recently from 59dce4d to 484bd82 Compare August 16, 2023 19:24

Shastick requested a review from mickmis August 16, 2023 19:27

Shastick force-pushed the feature/net-0260-a branch 2 times, most recently from 47d942f to 4ba121d Compare August 18, 2023 09:19

mickmis reviewed Aug 18, 2023

View reviewed changes

Shastick force-pushed the feature/net-0260-a branch 5 times, most recently from 6c3a882 to db9a0e8 Compare August 22, 2023 05:21

Shastick changed the title ~~[uss-qualifier] NET0260-a performance check for querying flights in an area from a SP~~ [uss_qualifier] NET0260-a performance check for querying flights in an area from a SP Aug 22, 2023

mickmis reviewed Aug 23, 2023

View reviewed changes

monitoring/uss_qualifier/scenarios/astm/netrid/common/aggregate_checks.py Outdated Show resolved Hide resolved

Shastick force-pushed the feature/net-0260-a branch from db9a0e8 to 535b8be Compare August 23, 2023 13:20

Shastick commented Aug 23, 2023

View reviewed changes

monitoring/uss_qualifier/scenarios/astm/netrid/common/aggregate_checks.py Outdated Show resolved Hide resolved

Shastick force-pushed the feature/net-0260-a branch 4 times, most recently from 8669c7c to d7f9362 Compare August 25, 2023 14:33

BenjaminPelletier reviewed Aug 27, 2023

View reviewed changes

monitoring/uss_qualifier/resources/netrid/service_providers.py Outdated Show resolved Hide resolved

monitoring/uss_qualifier/scenarios/astm/netrid/common/aggregate_checks.py Outdated Show resolved Hide resolved

Shastick marked this pull request as draft August 31, 2023 14:45

NET0260-a evaluate latency of calls to SP's /flights endpoint for dis…

a165f7b

…playing flights in a given area

Shastick force-pushed the feature/net-0260-a branch from d7f9362 to 41107d6 Compare August 31, 2023 15:07

Shastick mentioned this pull request Aug 31, 2023

[uss_qualifier] NET0460 evaluate display provider's flight data details endpoint performance #174

Merged

Shastick force-pushed the feature/net-0260-a branch from 41107d6 to 75290c1 Compare August 31, 2023 19:58

Shastick marked this pull request as ready for review August 31, 2023 20:13

Shastick requested a review from BenjaminPelletier August 31, 2023 20:13

BenjaminPelletier reviewed Aug 31, 2023

View reviewed changes

Shastick force-pushed the feature/net-0260-a branch from 75290c1 to 4cf5e85 Compare September 1, 2023 08:27

Shastick requested review from BenjaminPelletier September 1, 2023 08:29

Track participant id of queries and type of query

145d461

Shastick force-pushed the feature/net-0260-a branch from 4cf5e85 to 145d461 Compare September 1, 2023 08:39

BenjaminPelletier approved these changes Sep 1, 2023

View reviewed changes

BenjaminPelletier merged commit 09f7ba7 into interuss:main Sep 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[uss_qualifier] NET0260-a performance check for querying flights in an area from a SP #159

[uss_qualifier] NET0260-a performance check for querying flights in an area from a SP #159

Shastick commented Aug 11, 2023 •

edited

Loading

Shastick commented Aug 11, 2023

mickmis commented Aug 15, 2023

Shastick commented Aug 15, 2023

mickmis left a comment

Shastick left a comment

mickmis Aug 18, 2023

Shastick Aug 18, 2023

Shastick Aug 18, 2023 •

edited

Loading

mickmis Aug 23, 2023

mickmis commented Aug 23, 2023

Shastick commented Aug 24, 2023 •

edited

Loading

Shastick commented Aug 25, 2023

Shastick commented Aug 28, 2023

Shastick commented Aug 31, 2023

Shastick commented Aug 31, 2023

BenjaminPelletier Aug 31, 2023

Shastick Sep 1, 2023

Shastick commented Sep 1, 2023

[uss_qualifier] NET0260-a performance check for querying flights in an area from a SP #159

[uss_qualifier] NET0260-a performance check for querying flights in an area from a SP #159

Conversation

Shastick commented Aug 11, 2023 • edited Loading

Shastick commented Aug 11, 2023

mickmis commented Aug 15, 2023

Shastick commented Aug 15, 2023

mickmis left a comment

Choose a reason for hiding this comment

Shastick left a comment

Choose a reason for hiding this comment

mickmis Aug 18, 2023

Choose a reason for hiding this comment

Shastick Aug 18, 2023

Choose a reason for hiding this comment

Shastick Aug 18, 2023 • edited Loading

Choose a reason for hiding this comment

mickmis Aug 23, 2023

Choose a reason for hiding this comment

mickmis commented Aug 23, 2023

Shastick commented Aug 24, 2023 • edited Loading

v22a aggreagate checks on master:

Shastick commented Aug 25, 2023

Shastick commented Aug 28, 2023

Shastick commented Aug 31, 2023

Shastick commented Aug 31, 2023

BenjaminPelletier Aug 31, 2023

Choose a reason for hiding this comment

Shastick Sep 1, 2023

Choose a reason for hiding this comment

Shastick commented Sep 1, 2023

Shastick commented Aug 11, 2023 •

edited

Loading

Shastick Aug 18, 2023 •

edited

Loading

Shastick commented Aug 24, 2023 •

edited

Loading