[uss_qualifier] expand the authentication validation scenario to include constraint reference endpoints #690

Shastick · 2024-05-10T16:06:49Z

This is heavily inspired from the existing validation logic for operational intent references.

Some additional updates:

scopes for any endpoint group may be missing without the whole scenario being skipped
if all scopes are missing (ie, impossible to test any group) then a MissingResourceError is raised (which should cause the scenario to be skipped entirely)
added a validation step to the utm_implementation_us configuration to make the expected number of skipped scenarios explicit, which should allow us to detect if scenarios suddenly start being skipped when we don't expect it.

Note: this also moves the instantiations of the authentication validator classes just before each validator is run, as they depend on the current time.

BenjaminPelletier · 2024-08-05T18:33:22Z

monitoring/uss_qualifier/scenarios/astm/utm/dss/authentication/authentication_validation.md

+- `utm.constraint_management`
+- `utm.constraint_processing`


Making these required makes this scenario unusable in jurisdictions that don't use these features (which is all of them I'm aware of right now). Probably the most important CI configuration currently is utm_implementation_us as it parallels a test configuration in active use for actual regulatory compliance in the US, and this causes the entire scenario to be skipped in that configuration. We should probably add a validation criterion to that configuration (and probably others) that the count of skipped actions doesn't exceed the number of actions we expect to be skipped -- that would catch newly-skipped scenarios like this.

Each of these scopes should be optional (and then the validation for them not performed if not provided), or else split this scenario into strategic coordination (existing), constraint management, and constraint processing.

Ah yes, that's an oversight, thank you for pointing it out. I'll:

make the relevant scopes (utm.constraint_management, utm.constraint_processing) optional and make sure utm_implementation_us properly runs

possibly make the required scope for any endpoint group optional

add a validation criterion to utm_implementation_us

BenjaminPelletier · 2024-08-05T18:39:44Z

monitoring/uss_qualifier/scenarios/astm/utm/dss/authentication/authentication_validation.md

@@ -317,6 +318,146 @@ it is in violation of **[astm.f3548.v21.DSS0210,A2-7-2,7](../../../../../require
 If the DSS does not allow searching for operational intents when valid credentials are presented,
 it is in violation of **[astm.f3548.v21.DSS0005,1](../../../../../requirements/astm/f3548/v21.md)**.

+### Constraint reference endpoints authentication test step


Probably not the highest priority for adjustment, but It seems like the "steps" in this scenario should actually be "cases". A step is supposed to be a single logical action (do one thing), reflecting what a manual checkout test card would look like ("(step 1) Ok, now we're going to X -- did things Q and P happen? (step 2) Now we're going Y -- did things R and S happen?"). I would imagine a good test case would be "test authentication of all the constraint management endpoints", and then each testing of each constraint management endpoint would be a test step (or, testing of each constraint management endpoint in a particular way would be a test step).

This makes sense. it might be possible to easily do this as part of this PR, as I need to handle the optional utm.constraint_processing scopes, but I'm not entirely sure yet.

(Replacing the current steps with a test case having a single step would be easy in any case: I might keep splitting each test case into proper steps for later)

I created #742 to keep track of it in any case.

Given the current priorities I'll leave the splitting into separate test cases for later.

Shastick · 2024-08-07T08:01:02Z

Updated the PR: the various endpoint groups are now only optionally tested when the relevant scopes are available.

The scenario now runs as part of the utm_implementation_us configuration and skips the constraints references endpoints (I also added a validation step to the relevant configuration)

2024-08-07 07:23:07.532 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:316 - Note: missing_scope -> Missing scope testing disabled.
2024-08-07 07:23:07.532 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:316 - Note: scd -> Testing Strategic Coordination endpoints (Subscriptions and OIRs)
2024-08-07 07:23:07.532 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:316 - Note: scd_wrong_scope -> Incorrect scope testing enabled with scope interuss.flight_planning.direct_automated_test.
2024-08-07 07:23:07.763 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:316 - Note: constraints -> Skipping Constraint Reference endpoints
2024-08-07 07:23:07.763 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:316 - Note: availability -> Testing Availability endpoints
2024-08-07 07:23:07.763 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:316 - Note: availability_wrong_scope -> Incorrect scope testing enabled for availability endpoints with scope interuss.flight_planning.direct_automated_test.

Shastick · 2024-08-07T08:07:13Z

@BenjaminPelletier this PR ends up mixing two or three things now, some of them have been split:

[config] add a pass condition for skipped_actions to utm_implementation_us #748 for the validation step
[uss_qualifier] authentication validation: don't skip if a scope is missing, improve dependency on current time #749 for the optional scope/time issue

(Will reopen once they are merged and integrated)

Shastick · 2024-08-13T07:09:27Z

Ready for review: locally, using the f3548_self_contained configuration, all endpoints are tested:

2024-08-13 07:06:38.056 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:316 - Note: missing_scope -> Missing scope testing enabled.
2024-08-13 07:06:38.056 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:316 - Note: scd -> Testing Strategic Coordination endpoints (Subscriptions and OIRs)
2024-08-13 07:06:38.056 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:316 - Note: scd_wrong_scope -> Incorrect scope testing enabled with scope interuss.flight_planning.direct_automated_test.
2024-08-13 07:06:38.357 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:316 - Note: availability -> Testing Availability endpoints
2024-08-13 07:06:38.358 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:316 - Note: availability_wrong_scope -> Incorrect scope testing enabled for availability endpoints with scope interuss.flight_planning.direct_automated_test.
2024-08-13 07:06:38.402 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:316 - Note: constraints -> Testing Constraint Reference endpoints
2024-08-13 07:06:38.402 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:316 - Note: constraints_wrong_scope -> Incorrect scope testing enabled with scope interuss.flight_planning.direct_automated_test.

mickmis

Some minor comments, otherwise LGTM.

mickmis · 2024-08-23T07:31:31Z

monitoring/uss_qualifier/scenarios/astm/utm/dss/authentication/authentication_validation.md

+#### 🛑 Create constraint reference with valid credentials check
+
+If the DSS does not allow the creation of a constraint reference when valid credentials are presented,
+it is in violation of **[astm.f3548.v21.DSS0005,1](../../../../../requirements/astm/f3548/v21.md)**.


I believe this should be:

Suggested change

it is in violation of **[astm.f3548.v21.DSS0005,1](../../../../../requirements/astm/f3548/v21.md)**.

it is in violation of **[astm.f3548.v21.DSS0005,3](../../../../../requirements/astm/f3548/v21.md)**.

Good catch!

mickmis · 2024-08-23T07:32:29Z

monitoring/uss_qualifier/scenarios/astm/utm/dss/authentication/authentication_validation.md

+#### 🛑 Get constraint reference with valid credentials check
+
+If the DSS does not allow fetching a constraint reference when valid credentials are presented,
+it is in violation of **[astm.f3548.v21.DSS0005,1](../../../../../requirements/astm/f3548/v21.md)**.


Suggested change

it is in violation of **[astm.f3548.v21.DSS0005,1](../../../../../requirements/astm/f3548/v21.md)**.

it is in violation of **[astm.f3548.v21.DSS0005,3](../../../../../requirements/astm/f3548/v21.md)**.

Also: no checking of the get response format?

Good catch. Added to the validator.

monitoring/uss_qualifier/scenarios/astm/utm/dss/authentication/authentication_validation.md

mickmis · 2024-08-23T07:34:22Z

monitoring/uss_qualifier/scenarios/astm/utm/dss/authentication/authentication_validation.md

+#### 🛑 Delete constraint reference with valid credentials check
+
+If the DSS does not allow the deletion of a constraint reference when valid credentials are presented,
+it is in violation of **[astm.f3548.v21.DSS0005,1](../../../../../requirements/astm/f3548/v21.md)**.


Suggested change

it is in violation of **[astm.f3548.v21.DSS0005,1](../../../../../requirements/astm/f3548/v21.md)**.

it is in violation of **[astm.f3548.v21.DSS0005,3](../../../../../requirements/astm/f3548/v21.md)**.

Also, no checking of delete response format?

monitoring/uss_qualifier/scenarios/astm/utm/dss/authentication/authentication_validation.md

monitoring/uss_qualifier/scenarios/astm/utm/dss/authentication/cr_api_validator.py

mickmis

Could you fix the missed requirements? Otherwise LGTM thanks!

mickmis · 2024-08-27T11:50:54Z

monitoring/uss_qualifier/scenarios/astm/utm/dss/authentication/authentication_validation.md

+#### 🛑 Delete constraint reference with valid credentials check
+
+If the DSS does not allow the deletion of a constraint reference when valid credentials are presented,
+it is in violation of **[astm.f3548.v21.DSS0005,1](../../../../../requirements/astm/f3548/v21.md)**.


mickmis · 2024-08-27T11:51:10Z

monitoring/uss_qualifier/scenarios/astm/utm/dss/authentication/authentication_validation.md

+#### 🛑 Get constraint reference with valid credentials check
+
+If the DSS does not allow fetching a constraint reference when valid credentials are presented,
+it is in violation of **[astm.f3548.v21.DSS0005,1](../../../../../requirements/astm/f3548/v21.md)**.


mickmis · 2024-08-27T11:51:19Z

monitoring/uss_qualifier/scenarios/astm/utm/dss/authentication/authentication_validation.md

+#### 🛑 Create constraint reference with valid credentials check
+
+If the DSS does not allow the creation of a constraint reference when valid credentials are presented,
+it is in violation of **[astm.f3548.v21.DSS0005,1](../../../../../requirements/astm/f3548/v21.md)**.


Shastick · 2024-08-27T12:11:48Z

Could you fix the missed requirements? Otherwise LGTM thanks!

Thanks for catching them, they did not make it into the previous commit. Should be correct now.

…ude constraint reference endpoints (#690) * [uss_qualifier] constraint reference endpoint authentication checks * comments * correct spec number 4f4a63a

Shastick mentioned this pull request May 10, 2024

Implement tests for F3548 DSS requirements #434

Open

Shastick force-pushed the constraint-ref-auth branch from 9db0eca to a26fbdc Compare August 2, 2024 11:02

Shastick marked this pull request as ready for review August 2, 2024 12:29

Shastick requested a review from mickmis August 2, 2024 12:29

Shastick force-pushed the constraint-ref-auth branch from a26fbdc to 9bb0ab2 Compare August 5, 2024 07:28

BenjaminPelletier reviewed Aug 5, 2024

View reviewed changes

Shastick mentioned this pull request Aug 5, 2024

[scd] endpoint authentication test scenario should be split into several test cases instead of test steps #742

Open

mickmis removed their request for review August 6, 2024 09:41

Shastick marked this pull request as draft August 6, 2024 13:49

Shastick force-pushed the constraint-ref-auth branch 3 times, most recently from 0e0cf17 to ed0ff98 Compare August 7, 2024 07:57

Shastick force-pushed the constraint-ref-auth branch from ed0ff98 to db7385b Compare August 7, 2024 08:01

Shastick marked this pull request as ready for review August 7, 2024 08:01

Shastick requested a review from BenjaminPelletier August 7, 2024 08:02

Shastick force-pushed the constraint-ref-auth branch from db7385b to 427263f Compare August 7, 2024 08:06

Shastick force-pushed the constraint-ref-auth branch from 427263f to dfa2049 Compare August 7, 2024 08:31

Shastick mentioned this pull request Aug 9, 2024

[config] add a pass condition for skipped_actions to utm_implementation_us #748

Merged

Shastick marked this pull request as draft August 9, 2024 08:16

Shastick mentioned this pull request Aug 9, 2024

[uss_qualifier] authentication validation: don't skip if a scope is missing, improve dependency on current time #749

Merged

Shastick force-pushed the constraint-ref-auth branch from dfa2049 to ed3c9a9 Compare August 13, 2024 07:07

Shastick marked this pull request as ready for review August 13, 2024 07:08

Shastick force-pushed the constraint-ref-auth branch from ed3c9a9 to 857279e Compare August 20, 2024 14:28

[uss_qualifier] constraint reference endpoint authentication checks

b465580

Shastick force-pushed the constraint-ref-auth branch from 857279e to b465580 Compare August 21, 2024 05:01

mickmis reviewed Aug 23, 2024

View reviewed changes

comments

09b67b3

Shastick force-pushed the constraint-ref-auth branch from f435b47 to 09b67b3 Compare August 23, 2024 15:34

Shastick requested a review from mickmis August 23, 2024 15:35

mickmis approved these changes Aug 27, 2024

View reviewed changes

correct spec number

2272d2e

mickmis merged commit 4f4a63a into interuss:main Aug 27, 2024
20 checks passed

Shastick deleted the constraint-ref-auth branch August 27, 2024 15:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[uss_qualifier] expand the authentication validation scenario to include constraint reference endpoints #690

[uss_qualifier] expand the authentication validation scenario to include constraint reference endpoints #690

Shastick commented May 10, 2024 •

edited

Loading

BenjaminPelletier Aug 5, 2024

Shastick Aug 5, 2024 •

edited

Loading

BenjaminPelletier Aug 5, 2024

Shastick Aug 5, 2024 •

edited

Loading

Shastick Aug 7, 2024

Shastick commented Aug 7, 2024

Shastick commented Aug 7, 2024 •

edited

Loading

Shastick commented Aug 13, 2024

mickmis left a comment

mickmis Aug 23, 2024

Shastick Aug 23, 2024

mickmis Aug 27, 2024

mickmis Aug 23, 2024

mickmis Aug 23, 2024

Shastick Aug 23, 2024

mickmis Aug 27, 2024

mickmis Aug 23, 2024

mickmis Aug 23, 2024

mickmis Aug 27, 2024

mickmis left a comment

mickmis Aug 27, 2024

mickmis Aug 27, 2024

mickmis Aug 27, 2024

Shastick commented Aug 27, 2024

	it is in violation of [astm.f3548.v21.DSS0005,1](../../../../../requirements/astm/f3548/v21.md).
	it is in violation of [astm.f3548.v21.DSS0005,3](../../../../../requirements/astm/f3548/v21.md).

[uss_qualifier] expand the authentication validation scenario to include constraint reference endpoints #690

[uss_qualifier] expand the authentication validation scenario to include constraint reference endpoints #690

Conversation

Shastick commented May 10, 2024 • edited Loading

Choose a reason for hiding this comment

Shastick Aug 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Shastick Aug 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Shastick commented Aug 7, 2024

Shastick commented Aug 7, 2024 • edited Loading

Shastick commented Aug 13, 2024

mickmis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mickmis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Shastick commented Aug 27, 2024

Shastick commented May 10, 2024 •

edited

Loading

Shastick Aug 5, 2024 •

edited

Loading

Shastick Aug 5, 2024 •

edited

Loading

Shastick commented Aug 7, 2024 •

edited

Loading