[uss_qualifier] Run multiple uss_qualifier configurations #151

BenjaminPelletier · 2023-08-04T21:33:29Z

Currently, our CI relies primarily on running one giant "local_test" uss_qualifier configuration which uses many disparate parts of the uss_qualifier codebase. It is unnecessarily difficult to pipe all the resources to the right places, and is hard to isolate just the problem problem area when issues arise. This PR addresses this problem by changing uss_qualifier's run_locally.sh to run many configurations when the specific configuration desired is not specified. local_test is deleted entirely, replaced by the sum of smaller configuration files within configurations.dev. The intent is that we should be able to add small/targeted configuration files to test isolated parts of the uss_qualifier system which will allow us to iterate faster while still maintaining the same CI coverage we have today.

Since multiple configurations are being run, a separate report name is used for each one. This will incidentally allow easier identification of problems as the volume of content to search will be smaller (as long as it is clear in which test run the problem occurred).

Also, in this effort to make the configurations in configurations.dev more accessible, this PR adds a $content_schema key to indicate what schemas objects are intending to follow. This is not used anywhere programmatic and is intended to be a fancy comment to human readers.

A few random bugs occurring in off-nominal cases are fixed in common_dictionary_evaluator.py.

barroco

Thanks a lot for this, it is a great improvement.

barroco · 2023-08-07T19:47:29Z

monitoring/uss_qualifier/README.md

@@ -10,7 +10,7 @@ The `uss_qualifier` tool is a synchronous executable built into the `interuss/mo

 The primary input accepted by uss_qualifier is the "configuration" specified with the `--config` option.  This option should be a [reference to a configuration file](configurations/README.md) that the user has constructed or been provided to test the desired system for the desired characteristics.  If testing a standard local system (DSS + dummy auth + mock USSs), the user can specify an alternate configuration reference as a single argument to `run_locally.sh` (the default configuration is `configurations.dev.local_test`).

-When building a custom configuration file, consider starting from [`configurations.dev.self_contained_f3548`](configurations/dev/self_contained_f3548.yaml), as it contains all information necessary to run the test without the usage of sometimes-configuring `$ref`s and `allOf`s.  See [configurations documentation](configurations/README.md) for more information.
+When building a custom configuration file, consider starting from [`configurations.dev.self_contained_f3548`](configurations/dev/f3548_self_contained.yaml), as it contains all information necessary to run the test without the usage of sometimes-configuring `$ref`s and `allOf`s.  See [configurations documentation](configurations/README.md) for more information.


Suggested change

When building a custom configuration file, consider starting from [`configurations.dev.self_contained_f3548`](configurations/dev/f3548_self_contained.yaml), as it contains all information necessary to run the test without the usage of sometimes-configuring `$ref`s and `allOf`s. See [configurations documentation](configurations/README.md) for more information.

When building a custom configuration file, consider starting from [`configurations.dev.f3548_self_contained`](configurations/dev/f3548_self_contained.yaml), as it contains all information necessary to run the test without the usage of sometimes-configuring `$ref`s and `allOf`s. See [configurations documentation](configurations/README.md) for more information.

Thanks, fixed

barroco · 2023-08-07T19:57:08Z

monitoring/uss_qualifier/scenarios/astm/netrid/common_dictionary_evaluator.py

@@ -200,5 +204,6 @@ def evaluate_operational_status(
                        )
        else:
            self._test_scenario.record_note(
-                f"Unsupported version {self._rid_version}: skipping Operational Status evaluation"
+                key="skip_reason",
+                message=f"Unsupported version {self._rid_version}: skipping Operational Status evaluation",


Thanks for cleaning it up. It was actually going to be fixed by #150.

It'll be a race to see where it goes in first :)

barroco · 2023-08-07T20:09:24Z

monitoring/uss_qualifier/run_locally.sh

+    "configurations.dev.uspace" \
+  )
+  # TODO: Add configurations.dev.netrid_v19
+  # TODO: Add configurations.dev.netrid_v22a


Just to capture the intention, may I kindly ask you to describe what are the reasons preventing to add those configurations ?

TL;DR: Tracer.

Tracer must (currently) be run in F3411-19 mode OR F3411-22a mode, and the observation area is statically defined on startup. I'm actually working on a PR right now that will make tracer observation areas dynamic so that we won't have this issue.

…nfigurations

barroco

After a quick test, it is actually breaking on my machine. Investigating the issues.

monitoring/uss_qualifier/run_locally.sh

barroco · 2023-08-07T20:23:00Z

After a quick test, it is actually breaking on my machine. Investigating the issues.

For reference:

uss_qualifier % ./run_locally.sh
[+] Building 0.8s (17/17) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                           0.0s
 => => transferring dockerfile: 37B                                                                                                                                                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                              0.0s
 => => transferring context: 34B                                                                                                                                                                                                                                                               0.0s
 => [internal] load metadata for docker.io/library/python:3.8-slim                                                                                                                                                                                                                             0.6s
 => [ 1/12] FROM docker.io/library/python:3.8-slim@sha256:36ebeb57d8f126c0dfd412b09bbccfb256b32faa34255be0c45da763a5ba61f0                                                                                                                                                                     0.0s
 => => resolve docker.io/library/python:3.8-slim@sha256:36ebeb57d8f126c0dfd412b09bbccfb256b32faa34255be0c45da763a5ba61f0                                                                                                                                                                       0.0s
 => [internal] load build context                                                                                                                                                                                                                                                              0.1s
 => => transferring context: 136.83kB                                                                                                                                                                                                                                                          0.1s
 => CACHED [ 2/12] RUN apt-get update && apt-get install -y openssl curl libgeos-dev gcc && apt-get install ca-certificates                                                                                                                                                                    0.0s
 => CACHED [ 3/12] RUN mkdir -p /app/monitoring                                                                                                                                                                                                                                                0.0s
 => CACHED [ 4/12] COPY ./requirements.txt /app/monitoring/requirements.txt                                                                                                                                                                                                                    0.0s
 => CACHED [ 5/12] RUN pip install -r /app/monitoring/requirements.txt                                                                                                                                                                                                                         0.0s
 => CACHED [ 6/12] RUN rm -rf __pycache__                                                                                                                                                                                                                                                      0.0s
 => CACHED [ 7/12] ADD ./interfaces /app/interfaces                                                                                                                                                                                                                                            0.0s
 => CACHED [ 8/12] ADD ./monitoring /app/monitoring                                                                                                                                                                                                                                            0.0s
 => CACHED [ 9/12] COPY ./monitoring/health_check.sh /app/health_check.sh                                                                                                                                                                                                                      0.0s
 => CACHED [10/12] RUN chmod 766 /app/health_check.sh                                                                                                                                                                                                                                          0.0s
 => CACHED [11/12] WORKDIR /app/monitoring                                                                                                                                                                                                                                                     0.0s
 => CACHED [12/12] RUN mkdir -p /app/uss-host-files                                                                                                                                                                                                                                            0.0s
 => exporting to image                                                                                                                                                                                                                                                                         0.0s
 => => exporting layers                                                                                                                                                                                                                                                                        0.0s
 => => writing image sha256:96b124a5ff3d31275c0787d641a3bfffa69cbe88bb26e879d1d6da7e083bd757                                                                                                                                                                                                   0.0s
 => => naming to docker.io/interuss/monitoring                                                                                                                                                                                                                                                 0.0s
Running configurations: configurations.dev.noop configurations.dev.geoawareness_cis configurations.dev.generate_rid_test_data configurations.dev.geospatial_comprehension configurations.dev.general_flight_auth configurations.dev.f3548 configurations.dev.f3548_self_contained configurations.dev.uspace
date: illegal time format
usage: date [-jnRu] [-I[date|hours|minutes|seconds]] [-f input_fmt]
            [-r filename|seconds] [-v[+|-]val[y|m|w|d|H|M|S]]
            [[[[mm]dd]HH]MM[[cc]yy][.SS] | new_date] [+output_fmt]

barroco

See the latest suggestion which should fix the date issue when running locally on macos 13.3.1.

barroco · 2023-08-07T20:33:14Z

monitoring/uss_qualifier/run_locally.sh

+    docker_args="-it"
+  fi
+
+  start_time=$(date +%Y-%m-%dT%H:%M:%S -d '1 second ago')


On my mac, it looks like this syntax has been deprecated. The correct one would be:

Suggested change

start_time=$(date +%Y-%m-%dT%H:%M:%S -d '1 second ago')

start_time=$(date -v-1S +%Y-%m-%dT%H:%M:%S)

Ug, apparently date is not cross-platform. The old version works in the CI, the new version does not: date: invalid option -- 'v'

Upon additional research, it looks like date always rounds down so I think I don't need to subtract that second; I've removed the 1-second offset.

Run multiple uss_qualifier configurations

9433472

BenjaminPelletier force-pushed the feature/multiple-configurations branch from 9bb6108 to 9433472 Compare August 4, 2023 21:37

BenjaminPelletier marked this pull request as ready for review August 4, 2023 22:06

BenjaminPelletier requested a review from barroco August 4, 2023 22:06

barroco approved these changes Aug 7, 2023

View reviewed changes

BenjaminPelletier added 2 commits August 7, 2023 20:17

Address comments

21c5a98

Merge remote-tracking branch 'interuss/main' into feature/multiple-co…

286b7b9

…nfigurations

barroco requested changes Aug 7, 2023

View reviewed changes

monitoring/uss_qualifier/run_locally.sh Show resolved Hide resolved

barroco approved these changes Aug 7, 2023

View reviewed changes

BenjaminPelletier added 2 commits August 7, 2023 20:36

Address comments

64436c5

Remove 1 second offset

f3cdc0e

BenjaminPelletier merged commit 77b126a into interuss:main Aug 7, 2023

BenjaminPelletier deleted the feature/multiple-configurations branch August 7, 2023 21:23

mickmis mentioned this pull request Aug 15, 2023

[uss_qualifier/run_locally.sh] Do not ignore return code (fix CI) #162

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[uss_qualifier] Run multiple uss_qualifier configurations #151

[uss_qualifier] Run multiple uss_qualifier configurations #151

BenjaminPelletier commented Aug 4, 2023

barroco left a comment

barroco Aug 7, 2023

BenjaminPelletier Aug 7, 2023

barroco Aug 7, 2023

BenjaminPelletier Aug 7, 2023

barroco Aug 7, 2023

BenjaminPelletier Aug 7, 2023

barroco left a comment

barroco commented Aug 7, 2023

barroco left a comment

barroco Aug 7, 2023 •

edited

Loading

BenjaminPelletier Aug 7, 2023

BenjaminPelletier Aug 7, 2023

BenjaminPelletier Aug 7, 2023

	When building a custom configuration file, consider starting from [`configurations.dev.self_contained_f3548`](configurations/dev/f3548_self_contained.yaml), as it contains all information necessary to run the test without the usage of sometimes-configuring `$ref`s and `allOf`s. See [configurations documentation](configurations/README.md) for more information.
	When building a custom configuration file, consider starting from [`configurations.dev.f3548_self_contained`](configurations/dev/f3548_self_contained.yaml), as it contains all information necessary to run the test without the usage of sometimes-configuring `$ref`s and `allOf`s. See [configurations documentation](configurations/README.md) for more information.

	start_time=$(date +%Y-%m-%dT%H:%M:%S -d '1 second ago')
	start_time=$(date -v-1S +%Y-%m-%dT%H:%M:%S)

[uss_qualifier] Run multiple uss_qualifier configurations #151

[uss_qualifier] Run multiple uss_qualifier configurations #151

Conversation

BenjaminPelletier commented Aug 4, 2023

barroco left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

barroco left a comment

Choose a reason for hiding this comment

barroco commented Aug 7, 2023

barroco left a comment

Choose a reason for hiding this comment

barroco Aug 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

barroco Aug 7, 2023 •

edited

Loading